There are currently two Kubernetes clusters in hand, one is the k8s 1.3.7 version installed with kube-up.sh , and the other is the k8s 1.5.1 version installed with kubeadm . Since version 1.3.7 was installed before, and it is currently the environment that hosts our PaaS platform , I am relatively familiar with this version of Kubernetes installation environment, configuration operations, log viewing, cluster operations, etc. The 1.5.1 version of the K8s cluster installed by Kubeadm is quite different from the 1.3.7 version in many aspects such as component deployment, configuration, and logs. When you first get started, you will find that the 1.3.7 things you are familiar with are not in their original positions. It is estimated that many friends who use kubeadm to upgrade the cluster to version 1.5.1 will encounter such problems like me, so here I plan to conduct some small explorations of the Kubernetes cluster installed by Kubeadm, and make some changes. Listed for your reference.
1. Environment
The Kubernetes 1.5.1 cluster environment installed in the article " Installing Kubernetes with Kubeadm " is still used here. The bottom layer is Alibaba Cloud ECS, and the operating system is Ubuntu 16.04.1. The network uses the weave network .
The experimental cluster has only two Nodes: a master node and a minion node. However , because the master node is tainted , it participates in cluster scheduling and bears the load just like the minion node.
2. Podization of core components
Compared with the k8s cluster installed by Kubeadm and the kube-up.sh installation cluster, the biggest difference should be the podization of the core components of kubernetes, namely: kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, kube- Core components such as discovery and etcd all run in Pods in the cluster, which is quite CoreOS style. The only exception is the Kubelet, which is responsible for interacting with the local container engine on the node.
The core components of K8s Pod are placed in the kube-system namespace, which can be viewed through kubectl (on master node):
# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-iz25beglnhtz 1/1 Running 2 26d
kube-apiserver-iz25beglnhtz 1/1 Running 3 26d
kube-controller-manager-iz25beglnhtz 1/1 Running 2 26d
kube-scheduler-iz25beglnhtz 1/1 Running 4 26d
... ...
In addition, careful friends may find that the Pod names of these core components all end with the host name of the Node where they are located. For example, "iz25beglnhtz" in kube-apiserver-iz25beglnhtz is the host name of the Node where it is located.
However, even if these core components are running in a cluster in the form of a container, the network used by the components is still the host network of the Node where they are located, not the Pod Network :
# docker ps|grep apiserver
98ea64bbf6c8 gcr.io/google_containers/kube-apiserver-amd64:v1.5.1 "kube-apiserver --ins" 10 days ago Up 10 days k8s_kube-apiserver.6c2e367b_kube-apiserver-iz25beglnhtz_kube-system_033de1afc0844729cff5e100eb700a81_557d1fb2
4f87d22b8334 gcr.io/google_containers/pause-amd64:3.0 "/pause" 10 days ago Up 10 days k8s_POD.d8dbe16c_kube-apiserver-iz25beglnhtz_kube-system_033de1afc0844729cff5e100eb700a81_5931e490
# docker inspect 98ea64bbf6c8
... ...
"HostConfig": {
"NetworkMode": "container:4f87d22b833425082be55851d72268023d41b50649e46c738430d9dfd3abea11",
}
... ...
# docker inspect 4f87d22b833425082be55851d72268023d41b50649e46c738430d9dfd3abea11
... ...
"HostConfig": {
"NetworkMode": "host",
}
... ...
From the output of docker inspect above, we can see that the network mode adopted by the pause container in the kube-apiserver pod is the host network, and the kube-apiserver container based on the pause container network obviously inherits this network namespace . Therefore, from the outside, accessing components such as Kube-apiserver is the same as before: on the Master node, it can be accessed through localhost:8080; outside the Node, it can be accessed through the master_node_ip:6443 port.
3. Core component startup configuration adjustment
在kube-apiserver等核心组件还是以本地程序运行在物理机上的时代，修改kube-apiserver的启动参数，比如修改一下–service-node-port-range的范围、添加一个–basic-auth-file等，我们都可以通过直接修改/etc/default/kube-apiserver(以Ubuntu 14.04为例)文件的内容并重启kube-apiserver service(service restart kube-apiserver)的方式实现。其他核心组件：诸如：kube-controller-manager、kube-proxy和kube-scheduler均是如此。
但在kubeadm时代，这些配置文件不再存在，取而代之的是和用户Pod描述文件类似的manifest文件(都放置在/etc/kubernetes/manifests下面)：
/etc/kubernetes/manifests# ls
etcd.json kube-apiserver.json kube-controller-manager.json kube-scheduler.json
我们以为kube-apiserver增加一个启动参数：”–service-node-port-range=80-32767″ 为例：
打开并编辑/etc/kubernetes/manifests/kube-apiserver.json，在“command字段对应的值中添加”–service-node-port-range=80-32767″：
"containers": [
{
"name": "kube-apiserver",
"image": "gcr.io/google_containers/kube-apiserver-amd64:v1.5.1",
"command": [
"kube-apiserver",
"--insecure-bind-address=127.0.0.1",
"--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",
"--service-cluster-ip-range=10.96.0.0/12",
"--service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem",
"--client-ca-file=/etc/kubernetes/pki/ca.pem",
"--tls-cert-file=/etc/kubernetes/pki/apiserver.pem",
"--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
"--token-auth-file=/etc/kubernetes/pki/tokens.csv",
"--secure-port=6443",
"--allow-privileged",
"--advertise-address=10.47.217.91",
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
"--anonymous-auth=false",
"--etcd-servers=http://127.0.0.1:2379",
"--service-node-port-range=80-32767"
],
注意：不要忘记在–etcd-servers这一行后面添加一个逗号，否则kube-apiserver会退出。
修改后，你会发现kube-apiserver会被自动重启。这是kubelet的功劳。kubelet在启动时监听/etc/kubernetes/manifests目录下的文件变化并做适当处理：
# ps -ef|grep kubelet
root 1633 1 5 2016 ? 1-09:24:47 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local
kubelet自身是一个systemd的service，它的启动配置可以通过下面文件修改：
# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS
四、kubectl的配置
kube-up.sh安装的k8s集群会在每个Node上的~/.kube/下创建config文件，用于kubectl访问apiserver和操作集群使用。但在kubeadm模式下，~/.kube/下面的内容变成了：
~/.kube# ls
cache/ schema/
于是有了问题1：config哪里去了？
之所以在master node上我们的kubectl依旧可以工作，那是因为默认kubectl会访问localhost:8080来与kube-apiserver交互。如果kube-apiserver没有关闭–insecure-port，那么kubectl便可以正常与kube-apiserver交互，因为–insecure-port是没有任何校验机制的。
于是又了问题2：如果是其他node上的kubectl与kube-apiserver通信或者master node上的kubectl通过secure port与kube-apiserver通信，应该如何配置？
接下来，我们一并来回答上面两个问题。kubeadm在创建集群时，在master node的/etc/kubernetes下面创建了两个文件：
/etc/kubernetes# ls -l
total 32
-rw------- 1 root root 9188 Dec 28 17:32 admin.conf
-rw------- 1 root root 9188 Dec 28 17:32 kubelet.conf
... ...
这两个文件的内容是完全一样的，仅从文件名可以看出是谁在使用。比如kubelet.conf这个文件，我们就在kubelet程序的启动参数中看到过：–kubeconfig=/etc/kubernetes/kubelet.conf
# ps -ef|grep kubelet
root 1633 1 5 2016 ? 1-09:26:41 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local
打开这个文件，你会发现这就是一个kubeconfig文件，文件内容较长，我们通过kubectl config view来查看一下这个文件的结构：
# kubectl --kubeconfig /etc/kubernetes/kubelet.conf config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://{master node local ip}:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: admin
name: admin@kubernetes
- context:
cluster: kubernetes
user: kubelet
name: kubelet@kubernetes
current-context: admin@kubernetes
kind: Config
preferences: {}
users:
- name: admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
- name: kubelet
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
这和我们在《Kubernetes集群Dashboard插件安装》一文中介绍的kubeconfig文件内容并不二致。不同之处就是“REDACTED”这个字样的值，我们对应到kubelet.conf中，发现每个REDACTED字样对应的都是一段数据，这段数据是由对应的数字证书内容或密钥内容转换(base64)而来的，在访问apiserver时会用到。
我们在minion node上测试一下：
minion node：
# kubectl get pods
The connection to the server localhost:8080 was refused - did you specify the right host or port?
# kubectl --kubeconfig /etc/kubernetes/kubelet.conf get pods
NAME READY STATUS RESTARTS AGE
my-nginx-1948696469-359d6 1/1 Running 2 26d
my-nginx-1948696469-3g0n7 1/1 Running 3 26d
my-nginx-1948696469-xkzsh 1/1 Running 2 26d
my-ubuntu-2560993602-5q7q5 1/1 Running 2 26d
my-ubuntu-2560993602-lrrh0 1/1 Running 2 26d
kubeadm创建k8s集群时，会在master node上创建一些用于组件间访问的证书、密钥和token文件，上面的kubeconfig中的“REDACTED”所代表的内容就是从这些文件转化而来的：
/etc/kubernetes/pki# ls
apiserver-key.pem apiserver.pem apiserver-pub.pem ca-key.pem ca.pem ca-pub.pem sa-key.pem sa-pub.pem tokens.csv
- apiserver-key.pem：kube-apiserver的私钥文件
- apiserver.pem：kube-apiserver的公钥证书
- apiserver-pub.pem kube-apiserver的公钥文件
- ca-key.pem：CA的私钥文件
- ca.pem：CA的公钥证书
- ca-pub.pem ：CA的公钥文件
- sa-key.pem ：serviceaccount私钥文件
- sa-pub.pem ：serviceaccount的公钥文件
- tokens.csv：kube-apiserver用于校验的token文件
在k8s各核心组件的启动参数中会看到上面文件的身影，比如：
kube-apiserver --insecure-bind-address=127.0.0.1 --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota --service-cluster-ip-range=10.96.0.0/12 --service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem --client-ca-file=/etc/kubernetes/pki/ca.pem --tls-cert-file=/etc/kubernetes/pki/apiserver.pem --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --token-auth-file=/etc/kubernetes/pki/tokens.csv --secure-port=6443 --allow-privileged --advertise-address={master node local ip} --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --anonymous-auth=false --etcd-servers=http://127.0.0.1:2379 --service-node-port-range=80-32767
我们还可以在minion node上通过curl还手工测试一下通过安全通道访问master node上的kube-apiserver。在《Kubernetes集群的安全配置》一文中，我们提到过k8s的authentication（包括：客户端证书认证、basic auth、static token等）只要通过其中一个即可。当前kube-apiserver开启了客户端证书认证（–client-ca-file）和static token验证(–token-auth-file)，我们只要通过其中一个，就可以通过authentication，于是我们使用static token方式。static token file的内容格式：
token,user,uid,"group1,group2,group3"
对应到master node上的tokens.csv
# cat /etc/kubernetes/pki/tokens.csv
{token},{user},812ffe41-cce0-11e6-9bd3-00163e1001d7,system:kubelet-bootstrap
我们用这个token通过curl与apiserver交互：
# curl --cacert /etc/kubernetes/pki/ca.pem -H "Authorization: Bearer {token}" https://{master node local ip}:6443
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/apps",
"/apis/apps/v1beta1",
"/apis/authentication.k8s.io",
"/apis/authentication.k8s.io/v1beta1",
"/apis/authorization.k8s.io",
"/apis/authorization.k8s.io/v1beta1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/batch",
"/apis/batch/v1",
"/apis/batch/v2alpha1",
"/apis/certificates.k8s.io",
"/apis/certificates.k8s.io/v1alpha1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/apis/policy",
"/apis/policy/v1beta1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1alpha1",
"/apis/storage.k8s.io",
"/apis/storage.k8s.io/v1beta1",
"/healthz",
"/healthz/poststarthook/bootstrap-controller",
"/healthz/poststarthook/extensions/third-party-resources",
"/healthz/poststarthook/rbac/bootstrap-roles",
"/logs",
"/metrics",
"/swaggerapi/",
"/ui/",
"/version"
]
}http://tonybai.com/2017/01/24/explore-kubernetes-cluster-installed-by-kubeadm/