杭州建设厅网站,什么是网站开发,鞍山吧台,wordpress 公众平台文章目录创建 虚拟机模板虚拟机名称配置静态地址配置代理yum 配置配置主机名安装 git安装 docker安装 ansible配置内核参数安装 k8s定制安装新增节点配置主机名配置代理配置互信更新 inventory报错kubespray v2.21.0 部署 kubernetes v1.24.0 集群 【1】在 Rocky linux 8.7 使用…
文章目录创建 虚拟机模板虚拟机名称配置静态地址配置代理yum 配置配置主机名安装 git安装 docker安装 ansible配置内核参数安装 k8s定制安装新增节点配置主机名配置代理配置互信更新 inventory报错kubespray v2.21.0 部署 kubernetes v1.24.0 集群 【1】在 Rocky linux 8.7 使用 Kubespray v2.21.0 离线部署 kubernetes v1.24.0 集群
上一篇专门为了练习部署跑通。这篇为了学习定制安装部署以及新增节点删除节点升级节点动态申请pv监控等一些部署测试
创建 虚拟机模板
下载 rocky iso 8.7 iso上传 rocky iso 8.7 iso 至 vcenter并安装vcenter 如何安装虚拟机请参考这篇文章
需求
系统 Rocky Linux 8.7CPU: 4MEM: 8GDISK1: 60GDISK2: 200G
虚拟机名称
192.168.50.20-rocky-8.7-up-bastion01192.168.50.21-rocky-8.7-up-kube-controller01192.168.50.41-rocky-8.7-up-kube-node01192.168.50.42-rocky-8.7-up-kube-node02
配置静态地址
如何初始化 rocky linux 8.7 细节请参考这篇文章
$ cat /etc/sysconfig/network-scripts/ifcfg-ens192
TYPEEthernet
PROXY_METHODnone
BROWSER_ONLYno
BOOTPROTOnone
DEFROUTEyes
IPV4_FAILURE_FATALno
IPV6INITyes
IPV6_AUTOCONFyes
IPV6_DEFROUTEyes
IPV6_FAILURE_FATALno
IPV6_ADDR_GEN_MODEeui64
NAMEens192
UUIDda6c78ff-c1f0-4c05-8f0e-08848ab0a3e5
DEVICEens192
ONBOOTyes
IPADDR192.168.50.42
PREFIX20
GATEWAY192.168.48.1
DNS1192.168.48.1
DNS28.8.8.8
IPV6_PRIVACYno重启生效
nmctl con reload; nmctl networking off ;nmctl networking on配置代理
registry.k8s.io 镜像下载需要梯子。
https://github.com/wanhebin/clash-for-linux.git
cat /root/.bashrc
# .bashrc# User specific aliases and functionsalias rmrm -i
alias cpcp -i
alias mvmv -i# Source global definitions
if [ -f /etc/bashrc ]; then. /etc/bashrc
fiproxy_urlhttp://192.168.48.92:7890
export no_proxy10.0.0.0/8,192.168.0.0/16,localhost,127.0.0.0/8,.svc,.svc.cluster-28,.coding.net,.tencentyun.com,.myqcloud.com
# proxy settings
enable_proxy() {export http_proxy${proxy_url}export https_proxy${proxy_url}git config --global http.proxy ${proxy_url}git config --global http.proxy ${proxy_url}
}disable_proxy() {unset http_proxyunset https_proxygit config --global --unset http.proxygit config --global --unset https.proxy
}#disable_proxy
enable_proxyyum 配置
linux yum 软件包管理
系统自带
$ ls /etc/yum.repos.d/
Rocky-AppStream.repo Rocky-Devel.repo Rocky-Media.repo Rocky-PowerTools.repo Rocky-Sources.repo
Rocky-BaseOS.repo Rocky-Extras.repo Rocky-NFV.repo Rocky-ResilientStorage.repo
Rocky-Debuginfo.repo Rocky-HighAvailability.repo Rocky-Plus.repo Rocky-RT.repoyum update
yum -y install vim socat wget bash-completion net-tools zip bzip2 bind-utils配置主机名 hostnamectl set-hostname kube-controller01hostnamectl set-hostname kube-node01hostnamectl set-hostname kube-node02安装 git
tar 安装指定 git 版本请看这里
yum -y install git安装 docker
How To Install and Use Docker on Rocky Linux 8
sudo dnf check-update
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf -y install docker-ce docker-ce-cli containerd.io
sudo systemctl start docker
sudo systemctl status docker
sudo systemctl enable docker
安装 ansible
sudo dnf update
sudo dnf -y install epel-release
sudo dnf -y install ansible配置 inventory.ini
$ cd k8s-install/kubespray-2.21.0/rocky9.1-calico-cluster
$ vim inventory/cluster-local/inventory.ini
# ## Configure ip variable to bind kubernetes services on a
# ## different ip than the default iface
# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value.
[all]
kube-controller01 ansible_host192.168.50.21
kube-node01 ansible_host192.168.50.41
kube-node02 ansible_host192.168.50.42# ## configure a bastion host if your nodes are not directly reachable
[bastion]
bastion01 ansible_host192.168.50.20 ansible_userroot[kube_control_plane]
kube-controller01[etcd]
kube-controller01[kube_node]
kube-node01
kube-node02[calico_rr][k8s_cluster:children]
kube_control_plane
kube_node
calico_rr测试节点连通性
ansible -i inventory/cluster-local/inventory.ini all -m ping配置内核参数
modprobe bridge
modprobe br_netfilter
cat EOF /etc/sysctl.conf
net.ipv4.ip_forward 1
net.bridge.bridge-nf-call-iptables1
net.bridge.bridge-nf-call-ip6tables 1
EOF
sysctl -p /etc/sysctl.conf安装 k8s
$ ./install-cluster.sh欢迎使用 Kubespray 工具部署 k8s容器 kubespray-v2.21.0 创建成功现在你可以开始安装 k8s:1. docker attach kubespray-v2.21.02. pip3 install jmespath3. ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml如果不执行命令python3.8 install jmespath 报错1 Ansible: “You need to install ‘jmespath’ prior to running json_query filter”, but it is installedhttps://github.com/kubernetes-sigs/kubespray/issues/9826该 bug 计划在 2.21.1版本修复 定制安装
$ cat inventory/cluster-local/group_vars/k8s_cluster/k8s-cluster.yml
...
kube_version: v1.23.16## Container runtime
## docker for docker, crio for cri-o and containerd for containerd.
## Default: containerd
#container_manager: containerd
container_manager: docker
kubespray 2.21.1kubernetes 1.23.16docker
[rootkube-controller01 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.k8s.io/kube-apiserver v1.23.16 2d5c6bb50aa7 2 months ago 130MB
registry.k8s.io/kube-controller-manager v1.23.16 99fbab52b1e5 2 months ago 120MB
registry.k8s.io/kube-proxy v1.23.16 28204678d22a 2 months ago 111MB
registry.k8s.io/kube-scheduler v1.23.16 73e02f61aa83 2 months ago 51.9MB
registry.k8s.io/metrics-server/metrics-server v0.6.2 25561daa6660 4 months ago 68.9MB
quay.io/calico/kube-controllers v3.24.5 38b76de417d5 5 months ago 71.4MB
quay.io/calico/cni v3.24.5 628dd7088041 5 months ago 198MB
quay.io/calico/pod2daemon-flexvol v3.24.5 2f8f95ac9ac4 5 months ago 14.5MB
quay.io/calico/node v3.24.5 54637cb36d4a 5 months ago 226MB
registry.k8s.io/pause 3.8 4873874c08ef 10 months ago 711kB
registry.k8s.io/coredns/coredns v1.9.3 5185b96f0bec 10 months ago 48.8MB
quay.io/metallb/speaker v0.12.1 579ce8a43ea8 14 months ago 70MB
registry.k8s.io/coredns/coredns v1.8.6 a4ca41631cc7 18 months ago 46.8MB
registry.k8s.io/dns/k8s-dns-node-cache 1.21.1 5bae806f8f12 19 months ago 104MB
registry.k8s.io/pause 3.6 6270bb605e12 19 months ago 683kB
registry.k8s.io/cpa/cluster-proportional-autoscaler-amd64 1.8.5 1e7da779960f 20 months ago 40.7MB[rootkube-node01 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.k8s.io/kube-apiserver v1.23.16 2d5c6bb50aa7 2 months ago 130MB
registry.k8s.io/kube-scheduler v1.23.16 73e02f61aa83 2 months ago 51.9MB
registry.k8s.io/kube-controller-manager v1.23.16 99fbab52b1e5 2 months ago 120MB
registry.k8s.io/kube-proxy v1.23.16 28204678d22a 2 months ago 111MB
nginx 1.23.2-alpine 19dd4d73108a 5 months ago 23.5MB
quay.io/calico/kube-controllers v3.24.5 38b76de417d5 5 months ago 71.4MB
quay.io/calico/cni v3.24.5 628dd7088041 5 months ago 198MB
quay.io/calico/pod2daemon-flexvol v3.24.5 2f8f95ac9ac4 5 months ago 14.5MB
quay.io/calico/node v3.24.5 54637cb36d4a 5 months ago 226MB
registry.k8s.io/pause 3.8 4873874c08ef 10 months ago 711kB
registry.k8s.io/coredns/coredns v1.9.3 5185b96f0bec 10 months ago 48.8MB
registry.k8s.io/coredns/coredns v1.8.6 a4ca41631cc7 18 months ago 46.8MB
registry.k8s.io/dns/k8s-dns-node-cache 1.21.1 5bae806f8f12 19 months ago 104MB$ kubectl get pod -n kube-system coredns-54bf8d85c7-6tz7x -oyaml
Cannot enforce NoNewPrivs: illegal version string v1kube-api-access-qf65p:Type: Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds: 3607ConfigMapName: kube-root-ca.crtConfigMapOptional: nilDownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/oslinux
Tolerations: node-role.kubernetes.io/control-plane:NoSchedulenode-role.kubernetes.io/master:NoSchedulenode.kubernetes.io/not-ready:NoExecute opExists for 300snode.kubernetes.io/unreachable:NoExecute opExists for 300s
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 15m default-scheduler Successfully assigned kube-system/coredns-54bf8d85c7-6tz7x to kube-node01Warning FailedMount 15m (x6 over 15m) kubelet MountVolume.SetUp failed for volume config-volume : object kube-system/coredns not registeredWarning FailedMount 63s (x15 over 15m) kubelet MountVolume.SetUp failed for volume config-volume : object kube-system/coredns not registered– Cannot enforce NoNewPrivs: illegal version string “v1” https://github.com/Mirantis/cri-dockerd/issues/167
更新 kubernetes_version1.24.10 部署成功
[rootkube-controller01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
kube-controller01 Ready control-plane,master 3h36m v1.24.10
kube-node01 Ready none 3h35m v1.24.10
kube-node02 Ready none 3h35m v1.24.10
[rootkube-controller01 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6fc44869bc-94hwv 1/1 Running 0 3h35m
kube-system calico-node-7stvj 1/1 Running 0 3h35m
kube-system calico-node-dzvls 1/1 Running 0 3h35m
kube-system calico-node-wgn7g 1/1 Running 0 3h35m
kube-system coredns-57f7f7b97d-8ts8q 1/1 Running 0 12m
kube-system coredns-57f7f7b97d-sbd7v 1/1 Running 0 13m
kube-system dns-autoscaler-78676459f6-rgdwr 1/1 Running 0 3h35m
kube-system kube-apiserver-kube-controller01 1/1 Running 1 (36m ago) 36m
kube-system kube-controller-manager-kube-controller01 1/1 Running 1 (36m ago) 36m
kube-system kube-proxy-g9z9m 1/1 Running 0 36m
kube-system kube-proxy-sctdr 1/1 Running 0 36m
kube-system kube-proxy-tp5kr 1/1 Running 0 36m
kube-system kube-scheduler-kube-controller01 1/1 Running 1 (36m ago) 36m
kube-system metrics-server-cc8bc6d9b-tjjn9 1/1 Running 0 12m
kube-system nginx-proxy-kube-node01 1/1 Running 0 36m
kube-system nginx-proxy-kube-node02 1/1 Running 0 36m
kube-system nodelocaldns-gst6d 1/1 Running 0 3h35m
kube-system nodelocaldns-hpg6f 1/1 Running 0 3h35m
kube-system nodelocaldns-tx5z4 1/1 Running 0 3h35m新增节点
kubespray添加删除etcd节点master节点.md
主机192.168.48.92
配置主机名
hostnamectl set-hostname kube-node03配置代理
proxy_urlhttp://192.168.48.92:7890
export no_proxy10.0.0.0/8,192.168.0.0/16,localhost,127.0.0.0/8,.svc,.svc.cluster-28,.coding.net,.tencentyun.com,.myqcloud.com
# proxy settings
enable_proxy() {export http_proxy${proxy_url}export https_proxy${proxy_url}git config --global http.proxy ${proxy_url}git config --global http.proxy ${proxy_url}
}disable_proxy() {unset http_proxyunset https_proxygit config --global --unset http.proxygit config --global --unset https.proxy
}#disable_proxy
enable_proxy配置互信
部署节点bastion01执行
ssh-copy-id root192.168.48.92更新 inventory
$ cd /root/k8s-install/kubespray-2.21.0/rocky9.1-calico-cluster
$ cat inventory/cluster-local/inventory.ini
# ## Configure ip variable to bind kubernetes services on a
# ## different ip than the default iface
# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value.
[all]
kube-controller01 ansible_host192.168.50.21
kube-node01 ansible_host192.168.50.41
kube-node02 ansible_host192.168.50.42
kube-node03 ansible_host192.168.48.92# ## configure a bastion host if your nodes are not directly reachable
[bastion]
bastion01 ansible_host192.168.50.20 ansible_userroot[kube_control_plane]
kube-controller01[etcd]
kube-controller01[kube_node]
kube-node01
kube-node02
kube-node03[calico_rr][k8s_cluster:children]
kube_control_plane
kube_node
calico_rr在容器内执行
docker exec -ti kubespray-v2.21.0 bash
ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa scale.yml -b -v 报错
TASK [kubernetes/kubeadm : Join to cluster] ****************************************************************************************************************************************
skipping: [kube-node01] {changed: false, skip_reason: Conditional result was False}
skipping: [kube-node02] {changed: false, skip_reason: Conditional result was False}
fatal: [kube-node03]: FAILED! {changed: false, cmd: [timeout, -k, 120s, 120s, /usr/local/bin/kubeadm, join, --config, /etc/kubernetes/kubeadm-client.conf, --ignore-preflight-errorsDirAvailable--etc-kubernetes-manifests, --skip-phases], delta: 0:01:00.082674, end: 2023-04-17 20:38:11.054440, msg: non-zero return code, rc: 1, start: 2023-04-17 20:37:10.971766, stderr: \t[WARNING FileExisting-tc]: tc not found in system path\nerror execution phase preflight: couldnt validate the identity of the API Server: Get \https://192.168.50.21:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout10s\: x509: certificate has expired or is not yet valid: current time 2023-04-17T20:38:0708:00 is before 2023-04-17T15:56:59Z\nTo see the stack trace of this error execute with --v5 or higher, stderr_lines: [\t[WARNING FileExisting-tc]: tc not found in system path, error execution phase preflight: couldnt validate the identity of the API Server: Get \https://192.168.50.21:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout10s\: x509: certificate has expired or is not yet valid: current time 2023-04-17T20:38:0708:00 is before 2023-04-17T15:56:59Z, To see the stack trace of this error execute with --v5 or higher], stdout: [preflight] Running pre-flight checks, stdout_lines: [[preflight] Running pre-flight checks]}TASK [kubernetes/kubeadm : Join to cluster with ignores] ***************************************************************************************************************************
fatal: [kube-node03]: FAILED! {changed: false, cmd: [timeout, -k, 120s, 120s, /usr/local/bin/kubeadm, join, --config, /etc/kubernetes/kubeadm-client.conf, --ignore-preflight-errorsall, --skip-phases], delta: 0:01:00.082909, end: 2023-04-17 20:39:11.322402, msg: non-zero return code, rc: 1, start: 2023-04-17 20:38:11.239493, stderr: \t[WARNING FileExisting-tc]: tc not found in system path\nerror execution phase preflight: couldnt validate the identity of the API Server: Get \https://192.168.50.21:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout10s\: x509: certificate has expired or is not yet valid: current time 2023-04-17T20:39:0708:00 is before 2023-04-17T15:56:59Z\nTo see the stack trace of this error execute with --v5 or higher, stderr_lines: [\t[WARNING FileExisting-tc]: tc not found in system path, error execution phase preflight: couldnt validate the identity of the API Server: Get \https://192.168.50.21:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout10s\: x509: certificate has expired or is not yet valid: current time 2023-04-17T20:39:0708:00 is before 2023-04-17T15:56:59Z, To see the stack trace of this error execute with --v5 or higher], stdout: [preflight] Running pre-flight checks, stdout_lines: [[preflight] Running pre-flight checks]}TASK [kubernetes/kubeadm : Display kubeadm join stderr if any] *********************************************************************************************************************
skipping: [kube-node01] {}
