Kubeadm安装K8s1.26集群

Kubeadm安装K8s1.26集群

基本介绍

k8s集群1.26.4在Centos7.9机器上基于Kubeadm安装，CRI使用containerd，CNI使用Calico，路由规则使用ipvs。具体步骤会有原理解释，会侧重企业级集群应用的考量。希望通过此文章帮助读者掌握Kubeadm安装k8s的基本原理。

环境介绍

机器名	ip	系统版本	系统内核	cpu架构
m1	10.11.81.152	CentOS Linux release 7.9	5.4.242-1.el7	x86
w1	10.11.81.153	CentOS Linux release 7.9	5.4.242-1.el7	x86
w2	10.11.81.154	CentOS Linux release 7.9	5.4.242-1.el7	x86
w3	10.11.81.155	CentOS Linux release 7.9	5.4.242-1.el7	x86

1.初始化基本环境

1.1确保每个机器之间ip互相能访问通

cat /etc/hosts
10.11.81.152  m1
10.11.81.153  w1
10.11.81.154  w2
10.11.81.155  w3

1.2 升级每台机器系统内核 参考文章
Centos7.9默认的系统内核为3.10，使用ipvs需要内核版本高于4.1

#安装ELRepo仓库的yum源
yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm

#更新yum源仓库
yum -y update

#查看可用的系统内核包
yum  --disablerepo="*"  --enablerepo="elrepo-kernel"  list  available
# 长期维护版本为lt，最新主线稳定版为ml

#选择的是长期维护版本kernel-lt，如需更新最新稳定版选择kernel-ml，建议选择长期维护版，目前为5.4版本，内核版本过高也不好
yum  --enablerepo=elrepo-kernel  install  -y  kernel-lt

#查看可用内核版本及启动顺序
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /boot/grub2/grub.cfg

0 : CentOS Linux (5.4.108-1.el7.elrepo.x86_64) 7 (Core)
1 : CentOS Linux (3.10.0-1160.11.1.el7.x86_64) 7 (Core)
2 : CentOS Linux (3.10.0-1160.el7.x86_64) 7 (Core)
3 : CentOS Linux (0-rescue-20210128140208453518997635111697) 7 (Core)

#安装辅助工具
yum install -y grub2-pc

#设置内核默认启动顺序
grub2-set-default 0

#编辑/etc/default/grub文件
vim /etc/default/grub

#设置GRUB_DEFAULT=0  #只需要修改这里即可
sed -i 's/GRUB_DEFAULT=saved/GRUB_DEFAULT=0/g' /etc/default/grub

#生成grub 配置文件
# 运行grub2-mkconfig命令来重新创建内核配置
grub2-mkconfig -o /boot/grub2/grub.cfg

#重启系统
reboot
# 或者
shutdown -r now

#重启完成后，查看内核版本是否正确
uname -r

1.3每台机器都设置转发IPv4并让iptables看到桥接流量

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

集群安装好后切换到ipvs

1.4每台机器设置时间同步

yum install chrony -y
systemctl start chronyd
systemctl enable chronyd
chronyc sources

1.5每台机器关闭防火墙
生产环境需要按需开放防火墙端口。当然生产环境K8s集群前还会有负载均衡的堡垒机，如果并不是直接面向外网也可以关闭K8s集群内机器的防火墙

systemctl stop firewalld
systemctl disable firewalld

1.6每台机器关闭swap，关闭swap主要是为了性能考虑

# 临时关闭
swapoff -a
# 永久关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab
# 可以通过这个命令查看swap是否关闭了
free

看到Swap区域都为0

1.7每台机器禁用SELinux

# 临时关闭
setenforce 0
# 永久禁用
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

2.安装CRI

自 Kubernetes 1.20 版本开始，官方不再支持使用 Docker 作为默认的容器运行时。不在支持的Docker的原因请查看 K8s宣布弃用Docker

2.1每台机器安装containerd

# 添加docker源
curl -L -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装containerd
yum install -y containerd.io
# 创建默认配置文件
containerd config default > /etc/containerd/config.toml
# 设置aliyun地址，不设置会连接不上, 如果无法下载镜像检查一下配置是否替换 cat /etc/containerd/config.toml |grep sandbox_image
# 将pause:3.6替换为pause:3.9，如果有关版本发生变化需要修改
sed -i "s#registry.k8s.io/pause:3.6#registry.aliyuncs.com/google_containers/pause:3.9#g" /etc/containerd/config.toml
# 设置驱动为systemd，Kubernetes官网推荐设置
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
# 设置dicker地址为aliyun镜像地址
sed -i '/\[plugins\."io\.containerd\.grpc\.v1\.cri"\.registry\.mirrors\]/a\      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]\n        endpoint = ["https://8aj710su.mirror.aliyuncs.com" ,"https://registry-1.docker.io"]' /etc/containerd/config.toml

# 重启服务
systemctl daemon-reload
systemctl enable --now containerd
systemctl restart containerd

# 查看是否安装成功
containerd -version

2.2每台机器安装crictl工具

# 安装crictl工具
yum install -y cri-tools
# 生成配置文件
crictl config runtime-endpoint
# 编辑配置文件
cat << EOF | tee /etc/crictl.yaml
runtime-endpoint: "unix:///run/containerd/containerd.sock"
image-endpoint: "unix:///run/containerd/containerd.sock"
timeout: 10
debug: false
pull-image-on-create: false
disable-pull-on-run: false
EOF

# 查看是否安装成功
crictl info
crictl version

3.安装kubelet kubeadm kubectl

3.1每台机器安装kubelet kubeadm kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl #添加可以确保不发生版本漂移
EOF

# 安装指定版本
sudo yum install -y kubelet-1.26.4 kubeadm-1.26.4 kubectl-1.26.4 --disableexcludes=kubernetes

#启动kubelet
systemctl enable kubelet

3.2 m1机器执行集群初始化

kubeadm init \
  --apiserver-advertise-address=10.11.81.152 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.26.4 \
  --service-cidr=172.16.0.0/16 \
  --pod-network-cidr=10.244.0.0/16 \
  --ignore-preflight-errors=all

3.3如果kubelet启动失败查看启动文件

cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
cat /var/lib/kubelet/kubeadm-flags.env

3.4初始化出错重置命令

kubeadm reset
rm -fr ~/.kube/  /etc/kubernetes/* var/lib/etcd/*

3.5出现类似输出则表示成功

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.11.81.152:6443 --token abcdef.1234567890abcdef \
    --discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

3.6将3个worker节点加入集群

kubeadm join 10.11.81.152:6443 --token abcdef.1234567890abcdef \
    --discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

#如果出现如下错误
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
#执行
sysctl -w net.ipv4.ip_forward=1

4.安装Calico网络插件

当前企业级环境使用的最多的CNI插件是Calico，其优点是转发效率好，容易查错
4.1 m1节点安装Calico插件 官方文档安装方式

# 创建一个yaml文件存放目录
mkdir /root/cni
cd /root/cni

# 下载operator & custom-resources
wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml
wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml

# 修改custom-resources.yaml匹配容器网络 --pod-network-cidr=10.244.0.0/16
vi custom-resources.yaml #cidr: 192.168.0.0/16 -> cidr: 10.244.0.0/16

# 器群启动calico网络
kubectl create -f tigera-operator.yaml
kubectl create -f custom-resources.yaml

4.2成功后如图

5.切换ipvs

5.1所有机器执行准备步骤

# 所有机器执行，为kube-proxy开启ipvs的前提需要加载以下的内核模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

# 执行加载命令，检查是否完成加载
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

# 所有机器ipset软件包
yum install -y ipset ipvsadm

5.2 m1机器修改kube-proxy configmap

#修改kube-proxy configmap为ipvs模式
kubectl edit configmap kube-proxy -n kube-system

...
    kind: KubeProxyConfiguration
    metricsBindAddress: ""
    mode: "ipvs"  # "" 改成 ipvs
    nodePortAddresses: null
    oomScoreAdj: null
...
 
configmap/kube-proxy edited

# 重启kube-proxy容器
kubectl  delete pods -n kube-system -l k8s-app=kube-proxy

# 成功验证
kubectl -n kube-system logs kube-proxy-qmdhp|grep ipvs

I0415 10:48:49.984477       1 server_others.go:259] Using ipvs Proxier.
 # 在 kube-proxy pod 看到如上信息后说明是启用ipvs的

6.后续可选操作

1.检验集群是否可用

# 使用 deployment 控制器部署镜像
kubectl create deployment nginx-demo --image=nginx --replicas=1
deployment.apps/nginx-demo created

# 创建service暴露方式为NodePort
kubectl expose deployment nginx-demo --port=80 --target-port=80 --type=NodePort
service/nginx-demo exposed

# 访问Nginx资源
kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   172.16.0.1      <none>        443/TCP        5h59m
nginx-demo   NodePort    172.16.17.244   <none>        80:30392/TCP   11s

curl http://9.46.255.229:30392
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html {
    
     color-scheme: light dark; }
body {
    
     width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

6.2 m1机器设置kubectl简写，实际操作中非常实用

cat >> ~/.bashrc << EOF 
alias kg='kubectl get'
alias k='kubectl'
alias kd='kubectl describe pods'
alias ke='kubectl explain'
alias ka='kubectl apply'
EOF

source ~/.bashrc