问题一：
kube-node节点的状态是notready，原因是kubelet启动时报错“no space left on device:”：
218的kube-node节点的状态是notready，分配和申请资源OK，但是服务启动有问题：启动失败
分析：218上kubelt节点启动7s后失败，原因是： no space left on device:journalctl -u kubelet
:140] Failed to watch directory "/sys/fs/cgroup/memory/system.slice/run-29422.scope": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422422.scope": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no space left on device
:140] Failed to watch directory "/sys/fs/cgroup/memory/system.slice": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no spaceify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no space left on device
t.go:1339] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-29422.scope: no space left on devicey/system.slice/run-29422.scope: no space left on device
ode=exited, status=255/n/a
total used free shared buff/cache available
Mem: 4001 1624 721 222 1655 1477
Swap: 4351 143 4208
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 55M 2.0G 3% /dev/shm
tmpfs 2.0G 220M 1.8G 11% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/centos-root 99G 17G 78G 18% /
/dev/mapper/centos-data 296G 3.8G 277G 2% /data
/dev/mapper/centos-home 44G 169M 42G 1% /home
/dev/sda1 477M 147M 301M 33% /boot
tmpfs 2.0G 12K 2.0G 1% /data/ips/app/work/node/kubelet/pods/6204604d-6b7b-11e8-958a-000c295deb47/volumes/kubernetes.io~secret/default-token-6xmh2
tmpfs 2.0G 12K 2.0G 1% /data/ips/app/work/node/kubelet/pods/63725613-6b7b-11e8-958a-000c295deb47/volumes/kubernetes.io~secret/heapster-token-5slgw
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=10.20.16.218
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp: Sat, 09 Jun 2018 08:18:16 +0800
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:05:33 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:05:33 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:05:33 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Mon, 11 Jun 2018 11:35:40 +0800 Sat, 09 Jun 2018 08:18:16 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Mon, 11 Jun 2018 11:35:40 +0800 Mon, 11 Jun 2018 11:06:03 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.20.16.218
Hostname: 10.20.16.218
Capacity:
cpu: 2
ephemeral-storage: 309504832Ki
hugepages-2Mi: 0
memory: 4097940Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 285239652699
hugepages-2Mi: 0
memory: 3995540Ki
pods: 110
System Info:
Machine ID: edb038c4975745dbaf2abf3c7fb54d32
System UUID: 564DFD41-7F6F-DC56-E6ED-43B11018119C
Boot ID: 2384972f-22a8-4979-9252-b7cebea31a43
Kernel Version: 4.4.47-1.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://1.12.6
Kubelet Version: v1.10.3-beta.0.10+18f4924fc90ffc
Kube-Proxy Version: v1.10.3-beta.0.10+18f4924fc90ffc
PodCIDR: 172.30.0.0/24
ExternalID: 10.20.16.218
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system docker-health-gjrh6 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system heapster-fzpp5 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
0 (0%) 0 (0%) 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
Normal NodeHasSufficientDisk 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 1h kubelet, 10.20.16.218 Node 10.20.16.218 status is now: NodeHasNoDiskPressure
Normal Starting 1h kubelet, 10.20.16.218 Starting kubelet.
【定位】"/sys/fs/cgroup/memory/system.slice/run-29422.scope""/sys/fs/cgroup/memory/system.slice/目录下run-*.scope"文件过多导致的。
解决方法：
NAME STATUS ROLES AGE VERSION
10.20.16.218 Ready <none> 2d v1.10.3-beta.0.10+18f4924fc90ffc
10.20.16.219 Ready <none> 1h v1.10.3-beta.0.10+18f4924fc90ffc