一、安装前主题环境准备
1、docker安装
建议使用官网yum源安装，添加yum源之后，直接yum install docker即可
2、关闭所有节点的selinux
最好修改配置文件为disabled，而不是临时更改，避免以后重启引起不必要的麻烦
3、安装私有仓库环境Harbor
具体安装过程参考我的博客：https://blog.csdn.net/baidu_38432732/article/details/106430307
4、基本架构
| IP | 节点 | 备注 | |
| 192.168.0.221 | master | etcd复用此节点 | |
| 192.168.0.222 | node1 | etcd复用此节点 | |
| 192.168.0.223 | node2 | etcd复用此节点 |
二、安装预览
安装过程参考https://jimmysong.io/kubernetes-handbook/practice/install-kubernetes-on-centos.html，自己进行实践安装
1、创建 TLS 证书和秘钥
2、创建kubeconfig 文件
3、创建高可用etcd集群
4、部署master节点
5、安装flannel网络插件
6、部署node节点
7、安装kubedns插件
8、安装dashboard（后面教程已经更换为coredns）插件
9、安装heapster插件
10、安装EFK插件三、部署步骤
1、创建TLS证书和秘钥
1)安装CFSSL工具
复制全部粘贴的命令行执行，一步到位，操作节点master
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH2)创建CA
mkdir /root/ssl
cd /root/ssl
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json
# 根据config.json文件的格式创建如下的ca-config.json文件
# 过期时间设置成了 87600h
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
字段说明
ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
server auth：表示client可以用该 CA 对server提供的证书进行验证；
client auth：表示server可以用该CA对client提供的证书进行验证；3)创建CA证书签名请求
创建 CA 证书签名请求
创建 ca-csr.json 文件：
cat >ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}EOF目前为止4个文件了。
4)生成CA证书私钥
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2020/08/14 01:02:21 [INFO] generating a new CA key and certificate from CSR
2020/08/14 01:02:21 [INFO] generate received request
2020/08/14 01:02:21 [INFO] received CSR
2020/08/14 01:02:21 [INFO] generating key: rsa-2048
2020/08/14 01:02:21 [INFO] encoded CSR
2020/08/14 01:02:21 [INFO] signed certificate with serial number 270042469811423271743782413859714243682212941146
# ls ca*
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem目前为止7个文件了，ca开头的5个文件
5)创建kubernetes证书
hosts字段填写上所有你要用到的节点ip，创建 kubernetes 证书签名请求文件 kubernetes-csr.json：
# cat kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.0.221",
"192.168.0.222",
"192.168.0.223", "kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}6)生成kubernetes证书和私钥
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
2020/08/14 01:12:03 [INFO] generate received request
2020/08/14 01:12:03 [INFO] received CSR
2020/08/14 01:12:03 [INFO] generating key: rsa-2048
2020/08/14 01:12:03 [INFO] encoded CSR
2020/08/14 01:12:03 [INFO] signed certificate with serial number 612429958540327645676998182896534144132873819380
2020/08/14 01:12:03 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
# ls kubernetes*
kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem截止到目前11个文件了，kuber开头的4个
以上2步可以合并成一个步骤，少生成1个kubernetes-csr.json文件，直接在命令行中输入参数代理了文件输入。
echo '{"CN":"kubernetes","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname="127.0.0.1,192.168.0.221,192.168.0.222,192.168.0.223,kubernetes,kubernetes.default" - | cfssljson -bare kubernetes7)创建admin证书
vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}8)生成admin证书和私钥
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
2020/08/14 01:15:45 [INFO] generate received request
2020/08/14 01:15:45 [INFO] received CSR
2020/08/14 01:15:45 [INFO] generating key: rsa-2048
2020/08/14 01:15:45 [INFO] encoded CSR
2020/08/14 01:15:45 [INFO] signed certificate with serial number 39738051440333382124800615701424857189471887255
2020/08/14 01:15:45 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
# ls admin*
admin.csr admin-csr.json admin-key.pem admin.pem截止目前15个文件，admin开头的4个
9）创建kuber-proxy证书
vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}10)生成相关证书和私钥
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2020/08/14 01:17:25 [INFO] generate received request
2020/08/14 01:17:25 [INFO] received CSR
2020/08/14 01:17:25 [INFO] generating key: rsa-2048
2020/08/14 01:17:25 [INFO] encoded CSR
2020/08/14 01:17:25 [INFO] signed certificate with serial number 164915231628933918948800980424004596645806038266
2020/08/14 01:17:25 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
# ls kube-proxy*
kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem截止到目前19个文件，kube-proxy开头的4个
11)校验证书，举例校验kubernetes.pem证书，2个方法都可以，看输出内容可json定义是否一致。
# openssl x509 -noout -text -in kubernetes.pem
# cfssl-certinfo -cert kubernetes.pem# openssl x509 -noout -text -in kubernetes.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
6b:46:4f:23:58:a1:09:c6:3f:81:22:2c:a9:4d:b5:a7:ec:8d:94:f4
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = CN, ST = BeiJing, L = BeiJing, O = k8s, OU = System, CN = kubernetes
Validity
Not Before: Aug 13 17:07:00 2020 GMT
Not After : Aug 11 17:07:00 2030 GMT
Subject: C = CN, ST = BeiJing, L = BeiJing, O = k8s, OU = System, CN = kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:ed:6d:7a:af:c1:40:94:a0:d5:b5:23:44:3f:f4:
83:95:f0:9f:f9:4e:4c:85:a0:34:97:06:c9:1b:5d:
b8:03:f4:14:f9:9d:3c:fa:7c:a1:76:9b:d5:50:b3:
74:22:d1:f7:c8:40:93:35:46:15:a2:a0:fa:9e:24:
d0:ec:b2:b6:3e:1d:44:2a:c5:28:37:33:79:16:90:
d0:e4:a6:a9:ed:dc:61:0d:b1:a8:03:c2:b4:57:2b:
9f:36:3a:99:51:67:30:2e:20:9d:de:9b:0a:00:58:
66:6c:05:73:1f:6a:cd:5d:03:87:05:23:3c:5d:42:
68:75:7a:3a:8c:e5:62:6e:ab:02:15:a6:b5:58:5d:
31:fb:76:50:72:db:aa:76:03:19:32:5e:59:46:d1:
7b:81:fa:56:3a:8f:eb:44:95:34:8e:3a:cf:45:ed:
62:bb:e3:b5:3a:81:37:af:ae:9f:40:06:ea:14:b5:
f5:56:22:0f:b3:be:3a:ff:c7:fe:90:df:15:f8:72:
2e:26:ae:d0:f2:f9:d1:5f:53:c5:68:6f:46:97:8d:
96:bd:d3:5e:2c:e3:d1:17:03:3f:61:f2:d1:71:1a:
17:e0:08:30:fe:07:52:04:c8:9d:6f:74:49:8a:c3:
29:e7:60:91:b5:40:ca:e2:ec:3a:e7:60:67:75:0e:
a0:f3
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
64:8A:55:E8:B9:54:D7:7F:87:12:C1:86:51:16:AF:A8:53:39:CC:19
X509v3 Authority Key Identifier:
keyid:4A:7A:4A:78:4E:04:20:7B:FB:FD:E5:28:2B:AC:01:A1:74:0D:37:34
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.0.221, IP Address:192.168.0.222, IP Address:192.168.0.223
Signature Algorithm: sha256WithRSAEncryption
50:8c:6f:25:88:5b:10:92:9d:db:b1:27:cb:d6:5c:4a:1d:70:
9d:30:05:bc:a5:00:f8:c8:91:bf:b4:fa:66:2f:c7:13:25:79:
ad:1e:39:12:df:7a:a6:96:43:5a:2d:3a:2a:5b:38:cd:39:26:
04:cd:f3:ee:f0:ba:70:25:52:99:1c:1a:25:e6:41:83:47:11:
00:4b:6c:af:74:b7:ab:e4:af:47:f5:c4:9c:76:e7:d9:19:a9:
b0:82:9c:2b:c6:75:92:a7:6b:5a:f7:4d:02:9b:80:74:46:9c:
06:1a:f2:12:39:c0:0e:85:d2:69:0d:b7:72:7a:ad:aa:4f:69:
56:67:d2:83:b5:5a:ef:8c:e8:e9:1e:a1:93:f3:e9:30:5d:98:
c6:04:47:ee:c4:fb:8a:0a:0f:ad:7d:16:4f:10:ea:55:45:97:
b7:61:4c:ba:0d:38:0e:0a:cf:5c:4c:ec:34:db:c4:67:44:1f:
59:30:a5:82:3f:6e:30:98:aa:2f:fd:1f:a4:7c:46:76:b9:8b:
ad:7f:a1:e4:68:dd:48:a9:94:b8:0b:7e:c1:16:a3:52:d3:77:
8a:0e:4b:30:51:e0:4a:38:f3:07:e5:5d:40:84:33:1d:4b:7a:
63:ed:c0:61:ef:23:03:ad:74:3b:c9:d1:8d:c7:d2:a4:c8:e4:
69:95:6d:c6
# cfssl-certinfo -cert kubernetes.pem
{
"subject": {
"common_name": "kubernetes",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "BeiJing",
"province": "BeiJing",
"names": [
"CN",
"BeiJing",
"BeiJing",
"k8s",
"System",
"kubernetes"
]
},
"issuer": {
"common_name": "kubernetes",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "BeiJing",
"province": "BeiJing",
"names": [
"CN",
"BeiJing",
"BeiJing",
"k8s",
"System",
"kubernetes"
]
},
"serial_number": "612429958540327645676998182896534144132873819380",
"sans": [
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"127.0.0.1",
"192.168.0.221",
"192.168.0.222",
"192.168.0.223"
],
"not_before": "2020-08-13T17:07:00Z",
"not_after": "2030-08-11T17:07:00Z",
"sigalg": "SHA256WithRSA",
"authority_key_id": "4A:7A:4A:78:4E:4:20:7B:FB:FD:E5:28:2B:AC:1:A1:74:D:37:34",
"subject_key_id": "64:8A:55:E8:B9:54:D7:7F:87:12:C1:86:51:16:AF:A8:53:39:CC:19",
"pem": "-----BEGIN CERTIFICATE-----\nMIIEfzCCA2egAwIBAgIUa0ZPI1ihCcY/gSIsqU21p+yNlPQwDQYJKoZIhvcNAQEL\nBQAwZTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0Jl\naUppbmcxDDAKBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwpr\ndWJlcm5ldGVzMB4XDTIwMDgxMzE3MDcwMFoXDTMwMDgxMTE3MDcwMFowZTELMAkG\nA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0JlaUppbmcxDDAK\nBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwprdWJlcm5ldGVz\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7W16r8FAlKDVtSNEP/SD\nlfCf+U5MhaA0lwbJG124A/QU+Z08+nyhdpvVULN0ItH3yECTNUYVoqD6niTQ7LK2\nPh1EKsUoNzN5FpDQ5Kap7dxhDbGoA8K0VyufNjqZUWcwLiCd3psKAFhmbAVzH2rN\nXQOHBSM8XUJodXo6jOVibqsCFaa1WF0x+3ZQctuqdgMZMl5ZRtF7gfpWOo/rRJU0\njjrPRe1iu+O1OoE3r66fQAbqFLX1ViIPs746/8f+kN8V+HIuJq7Q8vnRX1PFaG9G\nl42WvdNeLOPRFwM/YfLRcRoX4Agw/gdSBMidb3RJisMp52CRtUDK4uw652BndQ6g\n8wIDAQABo4IBJTCCASEwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUF\nBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRkilXouVTXf4cS\nwYZRFq+oUznMGTAfBgNVHSMEGDAWgBRKekp4TgQge/v95SgrrAGhdA03NDCBoQYD\nVR0RBIGZMIGWggprdWJlcm5ldGVzghJrdWJlcm5ldGVzLmRlZmF1bHSCFmt1YmVy\nbmV0ZXMuZGVmYXVsdC5zdmOCHmt1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rl\ncoIka3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FshwR/AAABhwTA\nqADdhwTAqADehwTAqADfMA0GCSqGSIb3DQEBCwUAA4IBAQBQjG8liFsQkp3bsSfL\n1lxKHXCdMAW8pQD4yJG/tPpmL8cTJXmtHjkS33qmlkNaLToqWzjNOSYEzfPu8Lpw\nJVKZHBol5kGDRxEAS2yvdLer5K9H9cScdufZGamwgpwrxnWSp2ta900Cm4B0RpwG\nGvISOcAOhdJpDbdyeq2qT2lWZ9KDtVrvjOjpHqGT8+kwXZjGBEfuxPuKCg+tfRZP\nEOpVRZe3YUy6DTgOCs9cTOw028RnRB9ZMKWCP24wmKov/R+kfEZ2uYutf6HkaN1I\nqZS4C37BFqNS03eKDkswUeBKOPMH5V1AhDMdS3pj7cBh7yMDrXQ7ydGNx9KkyORp\nlW3G\n-----END CERTIFICATE-----\n"
}12)分发证书
将生成的证书cp到指定目录备用，除了master，2个node节点也需要拷贝到这个这些文件，为了方便copy文件，建议2个node节点针对master做免密码登录
# mkdir -p /etc/kubernetes/ssl
# cp *.pem /etc/kubernetes/ssl
# ls /etc/kubernetes/ssl/
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem kubernetes-key.pem kubernetes.pem[root@k8s_Master ssl]# scp *.pem 192.168.0.222:/etc/kubernetes/ssl
The authenticity of host '192.168.0.222 (192.168.0.222)' can't be established.
ECDSA key fingerprint is SHA256:gu9WNvUSMp+CjKu9Cm8vC6dQ6e6WWjjzRISSvsQ5dow.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.222' (ECDSA) to the list of known hosts.
[email protected]'s password:
admin-key.pem 100% 1675 1.1MB/s 00:00
admin.pem 100% 1399 1.6MB/s 00:00
ca-key.pem 100% 1675 2.4MB/s 00:00
ca.pem 100% 1359 2.3MB/s 00:00
kube-proxy-key.pem 100% 1679 2.0MB/s 00:00
kube-proxy.pem 100% 1403 2.0MB/s 00:00
kubernetes-key.pem 100% 1679 2.7MB/s 00:00
kubernetes.pem 100% 1619 1.5MB/s 00:00
[root@k8s_Master ssl]# scp *.pem 192.168.0.223:/etc/kubernetes/ssl
The authenticity of host '192.168.0.223 (192.168.0.223)' can't be established.
ECDSA key fingerprint is SHA256:gu9WNvUSMp+CjKu9Cm8vC6dQ6e6WWjjzRISSvsQ5dow.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.223' (ECDSA) to the list of known hosts.
[email protected]'s password:
admin-key.pem 100% 1675 1.0MB/s 00:00
admin.pem 100% 1399 1.9MB/s 00:00
ca-key.pem 100% 1675 2.6MB/s 00:00
ca.pem 100% 1359 2.2MB/s 00:00
kube-proxy-key.pem 100% 1679 2.7MB/s 00:00
kube-proxy.pem 100% 1403 1.9MB/s 00:00
kubernetes-key.pem 100% 1679 2.7MB/s 00:00
kubernetes.pem 100% 1619 2.4MB/s 00:00从上面的顺序可以看出pem文件的创建都是以一个json文件为输入进行创建的，json文件最后对我们并不重要，只需要把pem文件分别scp拷贝的所有node的/etc/kubernetes/ssl文件夹即可。