问题概述
查看kubelet或/var/log/messages日志一直包错，发现是孤儿pod，是由于其pod被删除后存储路径还保存在磁盘。
报错如下
[root@node5 ~]# journalctl -fu kubelet
-- Logs begin at Tue 2020-06-16 23:41:14 CST. --
Jun 19 17:25:12 node5 kubelet[4711]: E0619 17:25:12.038458 4711 kubelet_volumes.go:154] orphaned pod "27960f19-29f1-486a-9a9d-c6c9290d014a" found, but volume paths are still present on disk : There were a total of 2 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:14 node5 kubelet[4711]: E0619 17:25:14.071432 4711 kubelet_volumes.go:154] orphaned pod "27960f19-29f1-486a-9a9d-c6c9290d014a" found, but volume paths are still present on disk : There were a total of 2 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:16 node5 kubelet[4711]: E0619 17:25:16.037737 4711 kubelet_volumes.go:154] orphaned pod "27960f19-29f1-486a-9a9d-c6c9290d014a" found, but volume paths are still present on disk : There were a total of 2 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:18 node5 kubelet[4711]: E0619 17:25:18.070147 4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:20 node5 kubelet[4711]: E0619 17:25:20.036447 4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:22 node5 kubelet[4711]: E0619 17:25:22.069562 4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:24 node5 kubelet[4711]: E0619 17:25:24.065490 4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:26 node5 kubelet[4711]: E0619 17:25:26.073979 4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:28 node5 kubelet[4711]: E0619 17:25:28.037987 4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Jun 19 17:25:30 node5 kubelet[4711]: E0619 17:25:30.068136 4711 kubelet_volumes.go:154] orphaned pod "62e47eeb-8de7-4d4e-9e0f-28503d63be6a" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
解决办法
解决思路
- 根据/var/log/messages或kubelet报错，查找孤儿pod
- 通过mount命令查找孤儿pod的挂载
- 卸载孤儿pod的存储挂载
- 删除/var/lib/kubelet/pods下孤儿pod的存储路径
使用脚本处理，脚本内容如下
#!/bin/sh
orphanedPods=`cat /var/log/messages|grep 'orphaned pod'|awk -F '"' '{print $2}'|uniq`;
orphanedPodsNum=`echo $orphanedPods|awk -F ' ' '{print NF}'`;
echo -e "orphanedPods: $orphanedPodsNum \n$orphanedPods";
for i in $orphanedPods
do
echo "Deleting Orphaned pod id: $i";
rm -rf /var/lib/kubelet/pods/$i;
done
在报错的节点直接执行脚本即可
相关issue：
https://github.com/kubernetes/kubernetes/issues/60987
https://github.com/kubernetes/kubernetes/pull/68616