信用南京网站,网站的ftp在哪里可以查到,电子技术网,直播营销策划方案范文目录
Elasticsearch产品介绍
Fluentd 工作原理
Kibana产品介绍
一、环境准备
前三个主机都要操作
1、主机初始化配置
2、部署docker环境
2、部署kubernetes集群
2.1、组件介绍
2.2、配置阿里云yum源
2.3、安装kubelet kubeadm kubectl
2.4、配置init-config.yaml
…目录
Elasticsearch产品介绍
Fluentd 工作原理
Kibana产品介绍
一、环境准备
前三个主机都要操作
1、主机初始化配置
2、部署docker环境
2、部署kubernetes集群
2.1、组件介绍
2.2、配置阿里云yum源
2.3、安装kubelet kubeadm kubectl
2.4、配置init-config.yaml
2.5、安装master节点
2.6、安装node节点
2.7、安装flannel
3、部署企业镜像仓库
3.1、部署Harbor仓库
3.2、导入EFK镜像
4、部署EFK业务环境
4.1、准备组件Yaml文件
4.2、部署Elasticsearch
4.3、部署kibana
4.4、部署Fluentd
4.5、验证容器日志收集 随着 Docker 容器及云原生相关技术的迅速发展国内外厂商开始逐步向云原生方向转型。其中以 Kubernetes 为代表性的云原生技术凭借强大的功能成为各大厂商的第一选择。由于 Kubernetes 在容器编排领域的强势领先使得越来越多的企业将业务迁至基于 DockerKubernetes 技术栈打造的容器管理平台所以在 Kubernetes 集群环境下如何打造高效、可靠的业务日志收集系统也成为企业必须面临的问题。本章将主要介绍基于Elasticsearch、Fluentd 和 KibanaEFK技术栈实现完整 Kubernetes 集群日志收集解决方案。
Elasticsearch产品介绍
Elasticsearch 是一个 Restful 风格的、开源的分布式搜索引擎具备搜索和数据分析功能它的底层是开源库 Apache Lucene。Elasticsearch 具有如下特点。
一个分布式的实时文档存储每个字段可以被索引与搜索一个分布式实时分析搜索引擎能支撑上百个服务节点的扩展并支持 PB 级别的结构化或者非结构化数据。
Fluentd 工作原理
Fluentd 是一个日志的收集、处理、转发系统。通过丰富的插件可以收集来自各种系统或应用的日志转化为用户指定的格式后转发到用户所指定的日志存储系统中。 Fluentd 通过一组给定的数据源抓取日志数据处理后转换成结构化的数据格式将它们转发给其他服务比如 Elasticsearch、对象存储等等。Fluentd 支持超过 300 个日志存储和分析服务所以对日志存储和分析服务的支持是非常灵活的。Fluentd 采用了插件式的架构具有高可扩展性及高可用性同时还实现了高可靠的信息转发。其主要运行步骤如下所示
1首先 Fluentd 从多个日志源获取数据。
2结构化并且标记这些数据。
3最后根据匹配的标签将数据发送到多个目标服务。
Kibana产品介绍
Kibana 是一个开源的可视化分析平台用于和 Elasticsearch 一起工作。可以通过Kibana 搜索、查看、交互存放在 Elasticsearch 索引中的数据。也可以轻松地执行高级数据分析并且以各种图表、表格和地图的形式可视化数据。Kibana 简单的、基于浏览器的界面便于对大量数据进行呈现能够快速创建和共享动态仪表板实时显示 Elasticsearch 查询的变化。
一、环境准备 操作系统 IP地址 主机名 组件 CentOS7.x 192.168.50.54 k8s-master kubeadm、kubelet、kubectl、docker-ce CentOS7.x 192.168.50.51 k8s-node01 kubeadm、kubelet、kubectl、docker-ce、elasticsearch、fluentd CentOS7.x 192.168.50.50 k8s-node02 kubeadm、kubelet、kubectl、docker-ce、kibana、fluentd CentOS7.x 192.168.50.54 harbor docker-ce、docker-compose、harbor
注意所有主机配置推荐CPU2C Memory4G、运行 Elasticsearch 的节点要有足够的内存不低于 4GB。若 Elasticsearch 容器退出请检查宿主机中的/var/log/message 日志观察是否因为系统 OOM 导致进程被杀掉。 项目拓扑 前三个主机都要操作
1、主机初始化配置
所有主机配置禁用防火墙和selinux [rootlocalhost ~]# setenforce 0 [rootlocalhost ~]# iptables -F [rootlocalhost ~]# systemctl stop firewalld [rootlocalhost ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [rootlocalhost ~]# systemctl stop NetworkManager [rootlocalhost ~]# systemctl disable NetworkManager Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service. Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service. Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service. [rootlocalhost ~]# sed -i /^SELINUX/s/enforcing/disabled/ /etc/selinux/config 配置主机名并绑定hosts不同主机名称不同 [rootlocalhost ~]# hostname k8s-master [rootlocalhost ~]# bash [rootk8s-master ~]# cat EOF /etc/hosts 192.168.50.53 k8s-master 192.168.50.51 k8s-node01 192.168.50.50 k8s-node02 EOF 主机配置初始化 [rootk8s-master ~]# yum -y install vim wget net-tools lrzsz [rootk8s-master ~]# swapoff -a [rootk8s-master ~]# sed -i /swap/s/^/#/ /etc/fstab [rootk8s-master ~]# cat EOF /etc/sysctl.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 EOF [rootk8s-master ~]# modprobe br_netfilter [rootk8s-master ~]# sysctl -p net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 2、部署docker环境
三台主机上分别部署 Docker 环境因为 Kubernetes 对容器的编排需要 Docker 的支持。 [rootk8s-master ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo [rootk8s-master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2 [rootk8s-master ~]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo [rootk8s-master ~]# yum clean all yum makecache fast rootk8s-master ~]# yum -y install docker-ce [rootk8s-master ~]# systemctl start docker [rootk8s-master ~]# systemctl enable docker Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. 镜像加速器所有主机配置 [rootk8s-master ~]# systemctl daemon-reload [rootk8s-master ~]# systemctl restart docker [rootk8s-master ~]# cat END /etc/docker/daemon.json { registry-mirrors:[ https://nyakyfun.mirror.aliyuncs.com ] } END [rootk8s-master ~]# systemctl daemon-reload [rootk8s-master ~]# systemctl restart docker 2、部署kubernetes集群
2.1、组件介绍
三个节点都需要安装下面三个组件
kubeadm安装工具使所有的组件都会以容器的方式运行kubectl客户端连接K8S API工具kubelet运行在node节点用来启动容器的工具
2.2、配置阿里云yum源
使用 YUM 方式安装 Kubernetes时推荐使用阿里的 YUM 源。 [rootk8s-master ~]# cat EOF /etc/yum.repos.d/kubernetes.repo [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled1 gpgcheck1 repo_gpgcheck1 gpgkeyhttps://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF 2.3、安装kubelet kubeadm kubectl
所有主机配置 [rootk8s-master ~]# yum install -y kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0 [rootk8s-master ~]# systemctl enable kubelet Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. kubelet 刚安装完成后通过 systemctl start kubelet 方式是无法启动的需要加入节点或初始化为 master 后才可启动成功。
Kubeadm 提供了很多配置项Kubeadm 配置在 Kubernetes 集群中是存储在ConfigMap 中的也可将这些配置写入配置文件方便管理复杂的配置项。Kubeadm 配内容是通过 kubeadm config 命令写入配置文件的。 在master节点安装master 定于为192.168.50.53通过如下指令创建默认的init-config.yaml文件
2.4、配置init-config.yaml
[rootk8s-master ~]# kubeadm config print init-defaults init-config.yaml
[rootk8s-master ~]# vim init-config.yaml 1 apiVersion: kubeadm.k8s.io/v1beta2 2 bootstrapTokens: 3 - groups: 4 - system:bootstrappers:kubeadm:default-node-token 5 token: abcdef.0123456789abcdef 6 ttl: 24h0m0s 7 usages: 8 - signing 9 - authentication 10 kind: InitConfiguration 11 localAPIEndpoint: 12 advertiseAddress: 192.168.50.53 //master节点IP地址 13 bindPort: 6443 14 nodeRegistration: 15 criSocket: /var/run/dockershim.sock 16 name: k8s-master 17 taints: 18 - effect: NoSchedule 19 key: node-role.kubernetes.io/master 20 --- 21 apiServer: 22 timeoutForControlPlane: 4m0s 23 apiVersion: kubeadm.k8s.io/v1beta2 24 certificatesDir: /etc/kubernetes/pki 25 clusterName: kubernetes 26 controllerManager: {} 27 dns: 28 type: CoreDNS 29 etcd: 30 local: 31 dataDir: /var/lib/etcd 32 imageRepository: registry.aliyuncs.com/google_containers //修改为国内地址 33 kind: ClusterConfiguration 34 kubernetesVersion: v1.20.0 35 networking: 36 dnsDomain: cluster.local 37 serviceSubnet: 10.96.0.0/12 38 podSubnet: 10.244.0.0/16 //新增加 Pod 网段 39 scheduler: {} 2.5、安装master节点
拉取所需镜像 [rootk8s-master ~]# kubeadm config images list --config init-config.yaml registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0 registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0 registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.0 registry.aliyuncs.com/google_containers/kube-proxy:v1.20.0 registry.aliyuncs.com/google_containers/pause:3.2 registry.aliyuncs.com/google_containers/etcd:3.4.13-0 registry.aliyuncs.com/google_containers/coredns:1.7.0 [rootk8s-master ~]# kubeadm config images pull --configinit-config.yaml [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.2 [config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.4.13-0 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.7.0 [rootk8s-master ~]# ls | while read line do docker load $line done archive/tar: invalid tar header archive/tar: invalid tar header [rootk8s-master ~]# kubeadm init --configinit-config.yaml 初始化安装K8S [rootk8s-master ~]# kubeadm config images list --config init-config.yaml registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0 registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0 registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.0 registry.aliyuncs.com/google_containers/kube-proxy:v1.20.0 registry.aliyuncs.com/google_containers/pause:3.2 registry.aliyuncs.com/google_containers/etcd:3.4.13-0 registry.aliyuncs.com/google_containers/coredns:1.7.0 [rootk8s-master ~]# kubeadm config images pull --configinit-config.yaml [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.20.0 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.2 [config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.4.13-0 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.7.0 [rootk8s-master ~]# ls | while read line do docker load $line done archive/tar: invalid tar header archive/tar: invalid tar header [rootk8s-master ~]# kubeadm init --configinit-config.yaml 根据最下面提示操作
kubectl 默认会在执行的用户家目录下面的.kube 目录下寻找config 文件。这里是将在初始化时[kubeconfig]步骤生成的admin.conf 拷贝到.kube/config Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run kubectl apply -f [podnetwork].yaml with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.50.53:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:1b3e2037ebc4179efc8b4b678fd5a7d50a7d1e3afa197364bf2792c45b72f29e [rootk8s-master ~]# mkdir -p $HOME/.kube [rootk8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [rootk8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config 2.6、安装node节点
根据master安装时的提示信息
[rootk8s-node01 ~]# kubeadm join 192.168.50.53:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:1b3e2037ebc4179efc8b4b678fd5a7d50a7d1e3afa197364bf2792c45b72f29e [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected cgroupfs as the Docker cgroup driver. The recommended driver is systemd. Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.5. Latest validated version: 19.03 [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with kubectl -n kube-system get cm kubeadm-config -o yaml [kubelet-start] Writing kubelet configuration to file /var/lib/kubelet/config.yaml [kubelet-start] Writing kubelet environment file with flags to file /var/lib/kubelet/kubeadm-flags.env [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run kubectl get nodes on the control-plane to see this node join the cluster. [rootk8s-node02 ~]# kubeadm join 192.168.50.53:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:1b3e2037ebc4179efc8b4b678fd5a7d50a7d1e3afa197364bf2792c45b72f29e [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected cgroupfs as the Docker cgroup driver. The recommended driver is systemd. Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.5. Latest validated version: 19.03 [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with kubectl -n kube-system get cm kubeadm-config -o yaml [kubelet-start] Writing kubelet configuration to file /var/lib/kubelet/config.yaml [kubelet-start] Writing kubelet environment file with flags to file /var/lib/kubelet/kubeadm-flags.env [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run kubectl get nodes on the control-plane to see this node join the cluster. 前面已经提到在初始化 k8s-master 时并没有网络相关配置所以无法跟 node 节点通信因此状态都是“NotReady”。但是通过 kubeadm join 加入的 node 节点已经在k8s-master 上可以看到。 [rootk8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master NotReady control-plane,master 2m49s v1.20.0 k8s-node01 NotReady none 42s v1.20.0 k8s-node02 NotReady none 39s v1.20.0 2.7、安装flannel
Master 节点NotReady 的原因就是因为没有使用任何的网络插件此时Node 和Master的连接还不正常。目前最流行的Kubernetes 网络插件有Flannel、Calico、Canal、Weave 这里选择使用flannel。 所有主机上传flannel_v0.12.0-amd64.tar [rootk8s-master ~]# docker load flannel_v0.12.0-amd64.tar 256a7af3acb1: Loading layer 5.844MB/5.844MB d572e5d9d39b: Loading layer 10.37MB/10.37MB 57c10be5852f: Loading layer 2.249MB/2.249MB 7412f8eefb77: Loading layer 35.26MB/35.26MB 05116c9ff7bf: Loading layer 5.12kB/5.12kB Loaded image: quay.io/coreos/flannel:v0.12.0-amd64 [rootk8s-master ~]# kubectl apply -f kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17, unavailable in v1.22; use rbac.authorization.k8s.io/v1 ClusterRole clusterrole.rbac.authorization.k8s.io/flannel created Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17, unavailable in v1.22; use rbac.authorization.k8s.io/v1 ClusterRoleBinding clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created [rootk8s-master ~]# tar xf cni-plugins-linux-amd64-v0.8.6.tgz [rootk8s-master ~]# cp flannel /opt/cni/bin/ [rootk8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane,master 6m11s v1.20.0 k8s-node01 Ready none 4m4s v1.20.0 k8s-node02 Ready none 4m1s v1.20.0 [rootk8s-master ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-7f89b7bc75-8gh6f 1/1 Running 0 6m44s coredns-7f89b7bc75-dtclj 1/1 Running 0 6m44s etcd-k8s-master 1/1 Running 0 6m53s kube-apiserver-k8s-master 1/1 Running 0 6m53s kube-controller-manager-k8s-master 1/1 Running 0 6m53s kube-flannel-ds-amd64-4tvdv 1/1 Running 0 2m53s kube-flannel-ds-amd64-k54sk 1/1 Running 0 2m53s kube-flannel-ds-amd64-tm7dp 1/1 Running 0 2m53s kube-proxy-hf2p2 1/1 Running 0 6m44s kube-proxy-pr5np 1/1 Running 0 4m55s kube-proxy-zqrmf 1/1 Running 0 4m52s kube-scheduler-k8s-master 1/1 Running 0 6m53s 已经是ready状态
3、部署企业镜像仓库
3.1、部署Harbor仓库
所有主机配置禁用防火墙和selinux [rootlocalhost ~]# setenforce 0 [rootlocalhost ~]# iptables -F [rootlocalhost ~]# systemctl stop firewalld [rootlocalhost ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [rootlocalhost ~]# setenforce 0 [rootlocalhost ~]# iptables -F [rootlocalhost ~]# systemctl stop firewalld [rootlocalhost ~]# systemctl disable firewalld [rootlocalhost ~]# systemctl stop NetworkManager [rootlocalhost ~]# systemctl disable NetworkManager Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service. Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service. Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service. [rootlocalhost ~]# sed -i /^SELINUX/s/enforcing/disabled/ /etc/selinux/config 配置主机名 [rootlocalhost ~]# hostname harbor [rootlocalhost ~]# bash 部署docker环境
Harbor 仓库需要 Docker 容器支持所以 Docker 环境是必不可少的。 [rootharbor ~]#wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo [rootharbor ~]# yum install -y yum-utils device-mapper-persistent-data lvm2 使用 YUM 方式安装 Docker 时推荐使用阿里的 YUM 源。 [rootharbor ~]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo [rootharbor ~]# yum clean all yum makecache fast [rootharbor ~]# yum -y install docker-ce [rootharbor ~]# systemctl start docker [rootharbor ~]# systemctl enable docker 镜像加速器所有主机配置 [rootk8s-master ~]# cat END /etc/docker/daemon.json { registry-mirrors:[ https://nyakyfun.mirror.aliyuncs.com ] } END [rootk8s-master ~]# systemctl daemon-reload [rootk8s-master ~]# systemctl restart docker 部署docker-compose [rootharbor ~]# mv docker-compose /usr/local/bin/ [rootharbor ~]# chmod x /usr/local/bin/docker-compose 部署harbor
Harbor 私有仓库程序采用 docker-compose 方式部署不同的功能和应用处于不同的容器这样带来了很好的兼容性可在众多支持 Docker 的系统上运行 Harbor。
[rootharbor ~]# tar xf harbor-offline-installer-v2.0.0.tgz -C /usr/local/
[rootharbor harbor]# vim /usr/local/harbor/harbor.yml
Harbor 的配置文件是/usr/local/harbor/harbor.yml 文件默认的 hostname 要修改为
Harbor 虚拟机节点的 IP 地址。 # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: 192.168.50.54 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config #https: # https port for harbor, default is 443 # port: 443 # The path of cert and key files for nginx # certificate: /your/certificate/path # private_key: /your/private/key/path 启动harbor
[rootharbor ~]# cd /usr/local/harbor/
只有harbor.yml.tmpl给他改名就行了
[rootharbor harbor]# mv harbor.yml.tmpl harbor.yml
[rootharbor harbor]# sh install.sh [Step 0]: checking if docker is installed ... Note: docker version: 24.0.5 [Step 1]: checking docker-compose is installed ... Note: docker-compose version: 1.21.1 [Step 2]: loading Harbor images ... Loaded image: goharbor/notary-signer-photon:v2.0.0 Loaded image: goharbor/clair-adapter-photon:v2.0.0 Loaded image: goharbor/chartmuseum-photon:v2.0.0 Loaded image: goharbor/harbor-log:v2.0.0 Loaded image: goharbor/harbor-registryctl:v2.0.0 Loaded image: goharbor/registry-photon:v2.0.0 Loaded image: goharbor/clair-photon:v2.0.0 Loaded image: goharbor/notary-server-photon:v2.0.0 Loaded image: goharbor/redis-photon:v2.0.0 Loaded image: goharbor/nginx-photon:v2.0.0 Loaded image: goharbor/harbor-core:v2.0.0 Loaded image: goharbor/harbor-db:v2.0.0 Loaded image: goharbor/harbor-jobservice:v2.0.0 Loaded image: goharbor/trivy-adapter-photon:v2.0.0 Loaded image: goharbor/prepare:v2.0.0 Loaded image: goharbor/harbor-portal:v2.0.0 [Step 3]: preparing environment ... [Step 4]: preparing harbor configs ... prepare base dir is set to /usr/local/harbor WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml Generated and saved secret to file: /data/secret/keys/secretkey Successfully called func: create_root_cert Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir [Step 5]: starting Harbor ... Creating network harbor_harbor with the default driver Creating harbor-log ... done Creating registry ... done Creating harbor-db ... done Creating harbor-portal ... done Creating redis ... done Creating registryctl ... done Creating harbor-core ... done Creating nginx ... done Creating harbor-jobservice ... done ✔ ----Harbor has been installed and started successfully.---- 查看都是up [rootharbor harbor]# docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------------------------- harbor-core /harbor/entrypoint.sh Up (healthy) harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcp harbor-jobservice /harbor/entrypoint.sh Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514-10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80-8080/tcp,:::80-8080/tc p redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /home/harbor/entrypoint.sh Up (healthy) 5000/tcp registryctl /home/harbor/start.sh Up (healthy) Harbor 启动完成后浏览器访问 http://192.168.50.54打开 Harbor Web 页面
用户名admin 密码Harbor12345 修改所有主机docker启动脚本
[rootk8s-master ~]# vim /usr/lib/systemd/system/docker.service
13 ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --insecure-registr 192.168.50.54
重启服务
[rootk8s-master ~]# systemctl daemon-reload
[rootk8s-master ~]# systemctl restart docker
3.2、导入EFK镜像 [rootharbor ~]# docker login -u admin -p Harbor12345 http://192.168.50.54 WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [rootharbor ~]# docker load elasticsearch-7.4.2.tar [rootharbor ~]# docker load fluentd-es.tar [rootharbor ~]# docker load kibana-7.4.2.tar [rootharbor ~]# docker load alpine-3.6.tar [rootharbor ~]# docker tag b1179d 192.168.50.54/efk/elasticsearch:7.4.2 [rootharbor ~]# docker tag 636f3d 192.168.50.54/efk/fluentd-es-root:v2.5.2 [rootharbor ~]# docker tag 230d3d 192.168.50.54/efk/kibana:7.4.2 [rootharbor ~]# docker tag 43773d 192.168.50.54/efk/alpine:3.6 [rootharbor ~]# docker push 192.168.50.54/efk/elasticsearch:7.4.2 [rootharbor ~]# docker push 192.168.50.54/efk/fluentd-es-root:v2.5.2 [rootharbor ~]# docker push 192.168.50.54/efk/kibana:7.4.2 [rootharbor ~]# docker push 192.168.50.54/efk/alpine:3.6 访问存在 4、部署EFK业务环境
4.1、准备组件Yaml文件 [rootk8s-master ~]# mkdir /opt/efk [rootk8s-master ~]# cd /opt/efk/ [rootk8s-master efk]# grep image: elasticsearch.yaml image: 192.168.50.54/efk/elasticsearch:7.4.2 image: 192.168.50.54/efk/alpine:3.6 image: 192.168.50.54/efk/alpine:3.6 [rootk8s-master efk]# grep -A1 nodeSelector elasticsearch.yaml nodeSelector: kubernetes.io/hostname: k8s-node01 [rootk8s-master efk]# vim kibana.yaml [rootk8s-master efk]# grep image: kibana.yaml image: 192.168.50.54/efk/kibana:7.4.2 [rootk8s-master efk]# grep -A1 nodeSelector kibana.yaml nodeSelector: kubernetes.io/hostname: k8s-node02 [rootk8s-master efk]# vim fluentd [rootk8s-master efk]# vim fluentd.yaml [rootk8s-master efk]# grep image: fluentd.yaml image: 192.168.50.54/efk/fluentd-es-root:v2.5.2 [rootk8s-master efk]# vim test-pod.yaml [rootk8s-master efk]# grep image: test-pod.yaml image: 192.168.50.54/efk/alpine:3.6 4.2、部署Elasticsearch
创建命名空间
创建名为 logging 的命名空间用于存放 EFK 相关的服务。在 k8s-master节点的/opt/efk 目录下。 [rootk8s-master efk]# kubectl create -f namespace.yaml namespace/logging created [rootk8s-master efk]# kubectl get namespaces | grep logging logging Active 5s 创建 es 数据存储目录
Elasticsearch 服务通常可以简写为 es。到 k8s-node01 节点创建数据目录/esdata。 [rootk8s-node01 ~]# mkdir /esdata 部署 es 容器
进入 k8s-master节点的/opt/efk 目录部署 es 容器执行如下操作。 [rootk8s-master efk]# kubectl create -f elasticsearch.yaml statefulset.apps/elasticsearch-logging created service/elasticsearch created [rootk8s-master efk]# kubectl -n logging get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES elasticsearch-logging-0 0/1 PodInitializing 0 7s 10.244.2.4 k8s-node01 none none 等待片刻即可查看到 es 的 Pod已经部署到 k8s-node01 节点状态变为 running。 [rootk8s-master efk]# kubectl -n logging get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES elasticsearch-logging-0 0/1 PodInitializing 0 14s 10.244.2.4 k8s-node01 none none [rootk8s-master efk]# kubectl -n logging get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES elasticsearch-logging-0 1/1 Running 0 41s 10.244.2.4 k8s-node01 none none [rootk8s-master efk]# kubectl -n logging get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch ClusterIP 10.104.11.226 none 9200/TCP 69s 通过 curl 命令访问服务验证 es 是否部署成功。 [rootk8s-master efk]# curl 10.104.11.226:9200 { name : elasticsearch-logging-0, cluster_name : docker-cluster, cluster_uuid : tJHah36AQ8u8uHcBWRg2uw, version : { number : 7.4.2, build_flavor : default, build_type : docker, build_hash : 2f90bbf7b93631e52bafb59b3b049cb44ec25e96, build_date : 2019-10-28T20:40:44.881551Z, build_snapshot : false, lucene_version : 8.2.0, minimum_wire_compatibility_version : 6.8.0, minimum_index_compatibility_version : 6.0.0-beta1 }, tagline : You Know, for Search } 4.3、部署kibana
进入 k8s-master 的/opt/efk 目录执行如下命令。 [rootk8s-master efk]# kubectl create -f kibana.yaml service/kibana created deployment.apps/kibana created 查看 Pod 的状态。 [rootk8s-master efk]# kubectl -n logging get pods NAME READY STATUS RESTARTS AGE elasticsearch-logging-0 1/1 Running 0 109s kibana-86cf8fb94c-9r6jt 0/1 ContainerCreating 0 6s [rootk8s-master efk]# kubectl -n logging get pods NAME READY STATUS RESTARTS AGEelasticsearch-logging-0 1/1 Running 0 4m34s kibana-86cf8fb94c-9r6jt 1/1 Running 0 2m51s 查看对应的 Service得到 NodePort 值为 32564此端口为随机端口不同环境会不一致请以实际结果为准。 [rootk8s-master efk]# kubectl -n logging get svc |grep kibana kibana NodePort 10.111.216.111 none 5601:32564/TCP 3m16s 通过访问 192.168.50.53:32564 进入到 kibana 的访问界面观察是否可以正常打开其中 32564端口需要替换成实际的端口号。若能正常访问说明 Kibana 连接 es 已经正常。 4.4、部署Fluentd
给集群节点打标签
为了自由控制需要采集集群中节点上业务容器的服务日志。因此需要给 k8s-node01和 k8s-node02 节点打上 fluentdtrue 的标签 label。 [rootk8s-master efk]# kubectl label node k8s-node01 fluentdtrue node/k8s-node01 labeled [rootk8s-master efk]# kubectl label node k8s-node02 fluentdtrue node/k8s-node02 labeled k8s-node01 和 k8s-node02 已经打上了 fluentdtrue 的 label那么 Fluentd 服务就会启动到这两个节点也就意味着运行在这两个节点的 Pod 日志会被收集起来。 启动 Fluentd 服务
在 k8s-master节点的/opt/efk 目录启动 Fluentd 服务 [rootk8s-master efk]# kubectl create -f fluentd-es-config-main.yaml configmap/fluentd-es-config-main created [rootk8s-master efk]# kubectl create -f fluentd-configmap.yaml configmap/fluentd-config created [rootk8s-master efk]# kubectl create -f fluentd.yaml serviceaccount/fluentd-es created clusterrole.rbac.authorization.k8s.io/fluentd-es created clusterrolebinding.rbac.authorization.k8s.io/fluentd-es created daemonset.apps/fluentd-es-v2.5.2 created 查看 Pod 是否已经在 k8s-node01 和 k8s-node02 节点启动成功。 NAME READY STATUS RESTARTS AGE elasticsearch-logging-0 1/1 Running 0 9m2s fluentd-es-v2.5.2-7qxjm 1/1 Running 0 17s fluentd-es-v2.5.2-l7rtk 1/1 Running 0 17s kibana-86cf8fb94c-9r6jt 1/1 Running 0 7m19s 4.5、验证容器日志收集
创建测试容器
进入 k8s-master的/opt/efk 目录执行如下命令。 [rootk8s-master efk]# kubectl create -f test-pod.yaml pod/counter created [rootk8s-master efk]# kubectl get pods NAME READY STATUS RESTARTS AGE counter 1/1 Running 0 9s 4.6、配置 Kibana 索引创建完成后可以发现已经生成了多个索引域稍等片刻再次点击左上角的
discover 图标进入日志检索页面。 然后通过索引键去过滤比如根据Kubernetes.host、Kubernetes.container_name、 kubernetes.container_image_id等去做过滤。
通过其他元数据也可以过滤日志数据比如单击任何日志条目以查看其他元数据如容
器名称、Kubernetes 节点、命名空间等。
到这里在 Kubernetes 集群上已经成功部署了 EFK。
