一、 环境准备
1.1软件版本
操作系统：centos7
容器引擎 Docker：Docker version 18.09.6
容器编排系统 Kubernetes：v1.14.0
集群数据存储系统 Etcd：3.3.12
集群网络插件 flanneld: 0.7.1
1.2服务器角色
| 角色 | ip | 安装组件 |
|---|---|---|
| k8s-master | 192.168.80.112 | etcd，kube-apiserver，kube-controller-manager，kube-scheduler |
| k8s-node1 | 192.168.80.130 | etcd，kubelet，kube-proxy，docker，flanneld |
| k8s-node2 | 192.168.80.146 | etcd，kubelet，kube-proxy，docker，flanneld |
1.3配置服务器
关闭防护墙
所有节点执行
systemctl stop firewalld.service && systemctl disable firewalld.service
关闭selinux
所有节点执行
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
关闭系统SWAP
所有节点执行
sed -i '/centos-swap/d' /etc/fstab
mount -a
加载ipvs模块
在k8s1.11版本之后，建议使用ipvs替代iptables来实现k8s中的service网络,在node所有节点上，加载ipvs模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
时间同步
所有节点执行
yum install -y ntp
systemctl enable ntpd
systemctl start ntpd
timedatectl set-timezone Asia/Shanghai
timedatectl set-ntp yes
ntpq -p
开启内核关于ip报文转发
cat /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
二、部署etcd集群
在三个节点都安装etcd组成一个etcd集群
2.1制作etcd的证书
安装证书生成工具cfssl
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/bin/cfssl
mv cfssljson_linux-amd64 /usr/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
创建生成证书的配置文件
mkdir /opt/etcd_ca
cd !$
##创建ca的证书配置文件
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
##创建ca的证书请求文件
cat > ca-csr.json <<EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
#创建etcd服务的证书请求文件，hosts[]中填写三个节点的ip地址
cat > server-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"192.168.80.112",
"192.168.80.130",
"192.168.80.146"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
生成证书
#生成Etcd根证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
#生成Etcd服务证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
生成后可以看到生成了4个pem文件
ca-key.pem #ca根证书的key文件
ca.pem #ca的根证书文件
server-key.pem #etcd的证书key文件
server.pem #ca签署过的etcd证书文件
2.2安装etcd集群
下载etcd软件
一下步骤在三个节点都执行
cd /usr/local/src/
wget https://github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz
mkdir /opt/etcd/{bin,cfg,ssl} -p
tar zxvf etcd-v3.3.12-linux-amd64.tar.gz
mv etcd-v3.3.12-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
创建etcd的配置文件
所有节点都执行，并根据每个节点的情况进行修改。
cat > /opt/etcd/cfg/etcd <<EOF
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.80.112:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.80.112:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.80.112:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.80.112:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.80.112:2380,etcd02=https://192.168.80.130:2380,etcd03=https://192.168.80.146:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
etcd配置文件注解
ETCD_NAME 当前节点在集群中的名称
ETCD_DATA_DIR 数据目录
ETCD_LISTEN_PEER_URLS 用于集群间通信地址
ETCD_LISTEN_CLIENT_URLS 供客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS 用于向集群通告地址
ETCD_ADVERTISE_CLIENT_URLS 用于向客户端通告地址
ETCD_INITIAL_CLUSTER 集群中所有节点的通告地址
ETCD_INITIAL_CLUSTER_TOKEN 集群Token，三个节点必须一致
ETCD_INITIAL_CLUSTER_STATE 加入集群的当前状态，new是新集群，existing表示加入已有集群
利用systemd系统管理etcd
每个节点都执行
vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd
#从EnvironmentFile文件中读取变量
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
#填写etcd服务签署过的证书文件
--key-file=/opt/etcd/ssl/server-key.pem \
#填写etcd服务的key文件
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
#填写ca根证书文件
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
分发证书文件
cp ~/etcd_ca/*.pem /opt/etcd/ssl/
scp ~/etcd_ca/*.pem [email protected]:/opt/etcd/ssl
scp ~/etcd_ca/*.pem [email protected]:/opt/etcd/ssl
启动etcd并设置自动启动
systemctl daemon-reload
systemctl start etcd && systemctl enable etcd && systemctl status etcd
注意：如果有问题，查看日志：tail /var/log/messages
验证集群
[root@192-168-80-112 etcd_ca]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.80.112:2379,https://192.168.80.130:2379,https://192.168.80.146:2379" cluster-health
member 3133c23ed28ebf5d is healthy: got healthy result from https://192.168.80.130:2379
member 8873af14c7e7ba6f is healthy: got healthy result from https://192.168.80.112:2379
member e34c3c9a1d48ab03 is healthy: got healthy result from https://192.168.80.146:2379
cluster is healthy
注意： cluster-health才是真正交给etcd执行的命令，前面只是完成连接etcd的认证。可以设置alias，以方便后面使用
alias etcdctl='/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.80.112:2379,https://192.168.80.130:2379,https://192.168.80.146:2379"'
[root@192-168-80-112 etcd_ca]# etcdctl cluster-health
member 3133c23ed28ebf5d is healthy: got healthy result from https://192.168.80.130:2379
member 8873af14c7e7ba6f is healthy: got healthy result from https://192.168.80.112:2379
member e34c3c9a1d48ab03 is healthy: got healthy result from https://192.168.80.146:2379
cluster is healthy
