k8s安装环境

安装环境准备
192.168.1.19 master 安装软件 etcd、kube-apiserver、kubelet、kube-controller-manager,kube-scheduler、flannel、docker
192.168.1.20 node1 安装软件 etcd、kube-proxy、kubelet、flannel、docker
192.168.1.21 node2 安装软件 etcd、kube-proxy、kubelet、flannel、docker
Kubernetes集群功能描述
Master节点：
Master节点主要是四个模块组成：APIServer，Schedule，controller-manager，etcd

    APIServer: APIServer负责对外提供RESTful的kubernetes API的服务，它是系统管理指令的统一接口，任何对资源的增删该查都要交给APIServer处理后再交给etcd，如图，kubectl(kubernetes提供的客户端工具，该工具内部是对kubernetes API的调用）是直接和APIServer交互的。

    schedule: schedule负责调度Pod到合适的Node上，如果把scheduler看成一个黑匣子，那么它的输入是pod和由多个Node组成的列表，输出是Pod和一个Node的绑定。 kubernetes目前提供了调度算法，同样也保留了接口。用户根据自己的需求定义自己的调度算法。

    controller manager: 如果APIServer做的是前台的工作的话，那么controller manager就是负责后台的。每一个资源都对应一个控制器。而control manager就是负责管理这些控制器的，比如我们通过APIServer创建了一个Pod，当这个Pod创建成功后，APIServer的任务就算完成了。

    etcd：etcd是一个高可用的键值存储系统，kubernetes使用它来存储各个资源的状态，从而实现了Restful的API。

Node节点：
    每个Node节点主要由三个模块组成：kubelet，kube-proxy
    kube-proxy: 该模块实现了kubernetes中的服务发现和反向代理功能。kube-proxy支持TCP和UDP连接转发，默认基Round Robin算法将客户端流量转发到与service对应的一组后端pod。服务发现方面，kube-proxy使用etcd的watch机制监控集群中service和endpoint对象数据的动态变化，并且维护一个service到endpoint的映射关系，从而保证了后端pod的IP变化不会对访问者造成影响，另外，kube-proxy还支持session affinity。

    kublet：kublet是Master在每个Node节点上面的agent，是Node节点上面最重要的模块，它负责维护和管理该Node上的所有容器，但是如果容器不是通过kubernetes创建的，它并不会管理。本质上，它负责使Pod的运行状态与期望的状态一致。

Kubernetes 安装及配置

1.初始化环境
    设置关闭防火墙、SELINUX
    systemctl  stop  firewalld  &&  systemctl disable firewalld
    strnfore 0

    sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config

    vim /etc/selinux/config
    SELINUX=disabled

    关闭Swap
    swapoff -a && sysctl -w vm.swappiness=0
    vi /etc/fstab
    #UUID=      swap                    swap    defaults        0 0

    安装docker

删除已安装的docker
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine

yum install -y yum-utils device-mapper-persistent-data lvm2

配置阿里云docker yum 源

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

查看docker版本
yum list docker-ce --showduplicates

安装docker版本比如 docker 18.06.1
yum install docker-ce-18.06.1.ce

安装最新版
yum install docker-ce -y
yum install vim wget curl iptables iptables-service lrzsz yum install iptables-services -y

开启路由转发
echo "1" > /proc/sys/net/ipv4/ip_forward
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1

sysctl -p 重新加载
cat /proc/sys/net/ipv4/ip_forward

启动docker服务
systemctl enable docker && systemctl start docker

创建安装目录
mkdir /opt/k8s/etcd/{bin,cfg,ssl} -p
mkdir /opt/k8s/kubernetes/{bin,cfg,ssl} -p

安装、配置CFSSL
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

    创建证书
        etcd            ca.pem,  server.pem, server-key.pem
        kube-apiserver  ca.pem, server.pem, server-key.pem
        kubelet         ca.pem, ca-key.pem
        kube-proxy      ca.pem, kube-proxy.pem, kube-proxy-key.pem
        kubectl         ca.pem, admin.pem, admin-key.pem

#!/bin/bash
#创建ETCD证书
cat << EOF | tee ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
#鍒涘缓 ETCD CA 閰嶇疆鏂囦欢
cat << EOF | tee ca-csr.json
{
"CN": "etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "CA",
"ST": "San Francisco",
"O": "k8s",
"OU": "System"
}
]
}
EOF
#鍒涘缓 ETCD Server 璇佷功
cat << EOF | tee etcd-csr.json
{
"CN": "etcd",
"hosts": [
"192.168.1.19",
"192.168.1.20",
"192.168.1.21"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "beijing",
"ST": "beijing",
"OU": "System"
}
]
}
EOF

#鐢熸垚 ETCD CA 璇佷功鍜岀閽
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

#鍒涘缓 Kubernetes CA 璇佷功
cat << EOF | tee ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF

cat << EOF | tee ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "beijing",
"ST": "beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#鐢熸垚API_SERVER璇佷功
cat << EOF | tee server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.1.19",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "beijing",
"ST": "beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

#鍒涘缓 Kubernetes Proxy 璇佷功
cat << EOF | tee kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "beijing",
"ST": "beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

    ssh-key认证
        ssh-keygen  -t  rsa   一直回车就行
        ssh-copy-id  192.168.1.201  复制证书

    下载软件
    链接: https://pan.baidu.com/s/1lXjcXca_DBvvI_J_vtkvqQ 提取码: swdx

    解压软件
    for i in *.tar.gz; do tar zxvf $i; done 

    部署etcd
        cd etcd-v3.3.10-linux-amd64
        cp etcd  etcdctl  /opt/k8s/etcd/bin/

        etcd配置文件

cat /opt/k8s/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.211:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.211:2379"
#
##[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.211:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.211:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.211:2380,etcd02=https://192.168.1.212:2380,etcd03=https://192.168.1.213:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

创建 etcd systemd unit 文件

vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/k8s/etcd/cfg/etcd
ExecStart=/opt/k8s/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/k8s/etcd/ssl/etcd.pem \
--key-file=/opt/k8s/etcd/ssl/etcd-key.pem \
--peer-cert-file=/opt/k8s/etcd/ssl/etcd.pem \
--peer-key-file=/opt/k8s/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/opt/k8s/etcd/ssl/etcd-ca.pem \
--peer-trusted-ca-file=/opt/k8s/etcd/ssl/etcd-ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

cd /opt/ssl/
cp etcd*.pem /opt/k8s/etcd/ssl/

启动etcd服务
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

node
scp -r /opt/k8s/etcd/ root@node1:/opt/k8s/
scp /usr/lib/systemd/system/etcd.service root@node1:/usr/lib/systemd/system/

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.20:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.20:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.20:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.20:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.19:2380,etcd02=https://192.168.1.20:2380,etcd03=https://192.168.1
.21:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
~

验证etcd集群
./bin/etcdctl --ca-file=ssl/etcd-ca.pem --cert-file=ssl/etcd.pem --key-file=ssl/etcd-key.pem --endpoints="https://192.168.1.211:2379,https://192.168.1.212:2379,https://192.168.1.213:2379" cluster-health

member 76c80608d28a3281 is healthy: got healthy result from https://192.168.1.231:2379
member a2427d1f2581987b is healthy: got healthy result from https://192.168.1.246:2379
cluster is healthy

部署flannel网络

etcd写入集群pod网段信息
cd /opt/k8s/etcd/ssl && /opt/k8s/etcd/bin/etcdctl --ca-file=etcd-ca.pem --cert-file=etcd.pem --key-file=etcd-key.pem --endpoint="https://192.168.1.231:2379,https://192.168.1.246:2379" set /coreos.com/network/config '{ "Network": "172.16.0.0/16", "Backend": {"Type": "vxlan"}}' { "Network": "172.16.0.0/16", "Backend": {"Type": "vxlan"}}

下载安装包
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

解压安装
tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /opt/k8s/kubernetes/bin/

配置flannel网络
vim /opt/k8s/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.1.19:2379,https://192.168.1.20:2379,https://192.168.1.21:2379 -etcd-cafile=/opt/k8s/etcd/ssl/ca.pem -etcd-certfile=/opt/k8s/etcd/ssl/etcd.pem -etcd-keyfile=/opt/k8s/etcd/ssl/etcd-key.pem"

创建flannel启动文件

vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/k8s/kubernetes/cfg/flanneld
ExecStart=/opt/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件，后续 docker 启动时 使用这个文件中的环境变量配置 docker0 网桥；
flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 -iface 参数指定通信接口，如上面的 eth0 接口;
flanneld 运行时需要 root 权限；

配置docker启动指定子网段
/usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

启动flannel
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld

systemctl enable docker
systemctl restart docker

查看etcd中存的flannel网络信息

/opt/k8s/etcd/bin/etcdctl --ca-file=/opt/k8s/etcd/ssl/etcd-ca.pem --cert-file=/opt/k8s/etcd/ssl/etcd.pem --key-file=/opt/k8s/etcd/ssl/etcd-key.pem --endpoints="https://192.168.1.211:2379,https://192.168.1.212:2379,https://192.168.1.213:2379" ls /coreos.com/network/subnets

部署master节点
kubernetes 节点运行组件
kube-apiserver
kube-scheduler
kube-controller-manager
kube-scheduler和kube-controller-manager

下载、安装master节点
wget https://dl.k8s.io/v1.10.12/kubernetes-server-linux-amd64.tar.gz
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-scheduler kube-apiserver kube-controller-manager kubectl /opt/k8s/kubernetes/bin/

拷贝证书
cd /opt/ssl/ && cp *.pem /opt/k8s/kubernetes/ssl/

配置kubectl 命令全局使用
vim /etc/profile
PATH=/opt/k8s/kubernetes/bin:$PATH:$HOME/bin

source /etc/profile

创建 TLS Bootstrapping token
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
3860524c02c04f9a8065bee781aabf9c

vim /opt/k8s/kubernetes/cfg/token.csv
3860524c02c04f9a8065bee781aabf9c,kubelet-bootstrap,10001,"system:kubelet-bo
otstrap"

创建apiserver配置文件
vim /opt/k8s/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.1.211:2379,https://192.168.1.212:2379,https://192.168.1.213:2379 \
--bind-address=192.168.1.211 \
--insecure-port=8080 \
--secure-port=6443 \
--advertise-address=192.168.1.211 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/k8s/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/k8s/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/k8s/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/opt/k8s/etcd/ssl/etcd.pem \
--etcd-keyfile=/opt/k8s/etcd/ssl/etcd-key.pem"

创建启动文件
vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/k8s/kubernetes/cfg/kube-apiserver
ExecStart=/opt/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动服务
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver

查看服务状态
systemctl status kube-apiserver

ps -ef|grep kube-apiserver

部署kube-scheduler

创建kube-scheduler 配置文件
vim /opt/k8s/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect"

创建启动文件
vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/k8s/kubernetes/cfg/kube-scheduler
ExecStart=/opt/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动服务
systemctl daemon-reload
systemctl enable kube-scheduler.service
systemctl restart kube-scheduler.service

查看服务启动状态
systemctl status kube-scheduler

ps -ef |grep kube-scheduler

部署kube-controller-manager、

创建配置文件
vim /opt/k8s/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/k8s/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/k8s/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/k8s/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/k8s/kubernetes/ssl/ca-key.pem"

创建启动文件
vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/k8s/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]

启动服务
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager

查看服务
systemctl status kube-controller-manager
ps -ef |grep kube-controller-manager

kubernetes 命令添加到PATH
vim /etc/profile
PATH=/opt/k8s/kubernetes/bin:$PATH:$HOME/bin

source /etc/profile

查看master集群状态
kubectl get cs,nodes
NAME STATUS MESSAGE ERROR
componentstatus/controller-manager Healthy ok
componentstatus/scheduler Healthy ok
componentstatus/etcd-2 Healthy {"health":"true"}
componentstatus/etcd-1 Healthy {"health":"true"}
componentstatus/etcd-0 Healthy {"health":"true"}

部署node 节点

k8s work 节点运行组件: docker、kubelet、kube-proxy

kubelet 运行在每个worker节点上，接收kube-apiserver发送的请求，管理pod容器，执行交互式命令，如 exec、run、logs等

kubelet 启动时自动向kube-apiserver 注册节点信息，内置的cadvisor 统计和监控节点的资源使用情况
为了确保安全，开启接受https请求安全端口，对请求进行认证和授权，拒绝未授权的访问

将kubeklet 二进制文件拷贝node节点
cd /usr/local/src/kubernetes/server/bin/
cp kubelet kube-proxy /opt/k8s/kubernetes/bin/
scp kubelet kube-proxy node1:/opt/k8s/kubernetes/bin/
scp kubelet kube-proxy node2:/opt/k8s/kubernetes/bin/

创建 kubelet bootstrap kubeconfig 文件
cat cfg/environment.sh
BOOTSTRAP_TOKEN=bfa1728e004b4f7336829e1ec5690abd
KUBE_APISERVER="https://192.168.1.211:6443"

source cfg/environment.sh

设置集群参数

kubectl config set-cluster kubernetes \
--certificate-authority=../ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig

设置客户端认证参数

kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig

设置上下文参数

kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig

设置默认上下文

kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

创建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes \
--certificate-authority=../ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
--client-certificate=../ssl/kube-proxy.pem \
--client-key=../ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

将bootstrap kubeconfig kube-proxy.kubeconfig 文件拷贝到所有 nodes节点
cp bootstrap.kubeconfig kube-proxy.kubeconfig /k8s/kubernetes/cfg/
scp bootstrap.kubeconfig kube-proxy.kubeconfig 172.16.8.101:/k8s/kubernetes/cfg/
scp bootstrap.kubeconfig kube-proxy.kubeconfig 172.16.8.102:/k8s/kubernetes/cfg/

创建kubelet 参数配置文件拷贝到所有 nodes节点
创建 kubelet 参数配置模板文件：

vim /opt/k8s/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.1.211（每个节点的IP地址）
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true

创建kubelet配置文件
vim /opt/k8s/kubernetes/cfg/kubelet

KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.211 \
--kubeconfig=/opt/k8s/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/k8s/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/k8s/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/k8s/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

创建kubelet systemd unit 文件
vim /usr/lib/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/k8s/kubernetes/cfg/kubelet
ExecStart=/opt/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

将kubelet-bootstrap用户绑定到系统集群角色
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

启动服务
systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet

kubelet状态查看
systemctl status kubelet

ps -ef|grep kubelet

approve kubelet CSR 请求
可以手动或自动 approve CSR 请求。推荐使用自动的方式，因为从 v1.8 版本开始，可以自动轮转approve csr 后生成的证书。
手动 approve CSR 请求
查看 CSR 列表：
kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs 39m kubelet-bootstrap Pending
node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s 5m5s kubelet-bootstrap Pending

kubectl certificate approve node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs
certificatesigningrequest.certificates.k8s.io/node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs

kubectl certificate approve node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s
certificatesigningrequest.certificates.k8s.io/node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s approved
[
kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs 41m kubelet-bootstrap Approved,Issued
node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s 7m32s kubelet-bootstrap Approved,Issued

删除所有节点请求
kubectl delete csr --all

删除单个请求节点
kubectl delete csr node1 删除node1节点

删除node节点
kubectl delete nodes node1

删除所有node节点
kubectl delete nodes --all

查看集群状态
kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.1.211 Ready <none> 19h v1.13.0
192.168.1.212 Ready <none> 23m v1.13.0
192.168.1.213 Ready <none> 23m v1.13.0

部署 kube-proxy 组件
kube-proxy 运行在所有 node节点上，它监听 apiserver 中 service 和 Endpoint 的变化情况，创建路由规则来进行服务负载均衡。

创建 kube-proxy 配置文件
vim /opt/k8s/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.211 \
--cluster-cidr=10.0.0.0/24 \
--kubeconfig=/opt/k8s/kubernetes/cfg/kube-proxy.kubeconfig"

bindAddress: 监听地址；
clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件；
clusterCIDR: kube-proxy 根据 –cluster-cidr 判断集群内部和外部流量，指定 –cluster-cidr 或 –masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
mode: 使用 ipvs 模式；

创建kube-proxy systemd unit 文件
vim /usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/k8s/kubernetes/cfg/kube-proxy
ExecStart=/opt/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动服务
systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy

systemctl status kube-proxy

集群状态
打node 或者master 节点的标签
kubectl label node 172.16.8.100 node-role.kubernetes.io/master='master'
kubectl label node 172.16.8.101 node-role.kubernetes.io/node='node'
kubectl label node 172.16.8.102 node-role.kubernetes.io/node='node'

kubectl get node,cs

查看绑定用户
kubectl get clusterrolebindings