Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kubelet: W0109 19:04:13.327228 3021 pod_container_deletor.go:75] Container "b5882640988846582cadec740e7487413e78338f05bff827415d7d155ecb856b" not found in pod's containers
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kubelet: E0109 19:04:13.344960 3021 remote_runtime.go:96] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "kube-flannel-ds-amd64-7fcd4": Error response from daemon: Conflict. The container name "/k8s_POD_kube-flannel-ds-amd64-7fcd4_kube-system_bd115adb-0239-11e9-a9bf-000c292abd35_445150" is already in use by container "a72d5a200aa4dd07faeaf37770db6957b6c0f610f19b1e33f8af815fd806075c". You have to remove (or rename) that container to be able to reuse that name.
Jan 9 19:04:13 yanfa11 kubelet: E0109 19:04:13.345049 3021 kuberuntime_sandbox.go:65] CreatePodSandbox for pod "kube-flannel-ds-amd64-7fcd4_kube-system(bd115adb-0239-11e9-a9bf-000c292abd35)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "kube-flannel-ds-amd64-7fcd4": Error response from daemon: Conflict. The container name "/k8s_POD_kube-flannel-ds-amd64-7fcd4_kube-system_bd115adb-0239-11e9-a9bf-000c292abd35_445150" is already in use by container "a72d5a200aa4dd07faeaf37770db6957b6c0f610f19b1e33f8af815fd806075c". You have to remove (or rename) that container to be able to reuse that name.
Jan 9 19:04:13 yanfa11 kubelet: E0109 19:04:13.345079 3021 kuberuntime_manager.go:662] createPodSandbox for pod "kube-flannel-ds-amd64-7fcd4_kube-system(bd115adb-0239-11e9-a9bf-000c292abd35)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "kube-flannel-ds-amd64-7fcd4": Error response from daemon: Conflict. The container name "/k8s_POD_kube-flannel-ds-amd64-7fcd4_kube-system_bd115adb-0239-11e9-a9bf-000c292abd35_445150" is already in use by container "a72d5a200aa4dd07faeaf37770db6957b6c0f610f19b1e33f8af815fd806075c". You have to remove (or rename) that container to be able to reuse that name.
Jan 9 19:04:13 yanfa11 kubelet: E0109 19:04:13.345174 3021 pod_workers.go:190] Error syncing pod bd115adb-0239-11e9-a9bf-000c292abd35 ("kube-flannel-ds-amd64-7fcd4_kube-system(bd115adb-0239-11e9-a9bf-000c292abd35)"), skipping: failed to "CreatePodSandbox" for "kube-flannel-ds-amd64-7fcd4_kube-system(bd115adb-0239-11e9-a9bf-000c292abd35)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-flannel-ds-amd64-7fcd4_kube-system(bd115adb-0239-11e9-a9bf-000c292abd35)\" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod \"kube-flannel-ds-amd64-7fcd4\": Error response from daemon: Conflict. The container name \"/k8s_POD_kube-flannel-ds-amd64-7fcd4_kube-system_bd115adb-0239-11e9-a9bf-000c292abd35_445150\" is already in use by container \"a72d5a200aa4dd07faeaf37770db6957b6c0f610f19b1e33f8af815fd806075c\". You have to remove (or rename) that container to be able to reuse that name."
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
Jan 9 19:04:13 yanfa11 kernel: overlayfs: upper fs needs to support d_type. This is an invalid configuration.
最开始的现象是服务器报错Memory cgroup out ofmemory: Kill process XXX,导致服务器登录不进去。进一步排查,是docker容器启动失败,以至于hibernates不断创建pod,以至于内存超过了cgroup范围,操作系统频繁杀进程,hibernates就不断创建进程。
问题分析:
通过/var/log/messages日志信息,在网上找了大量资料后,是docker17.06挂载的时候,默认使用的Storage Driver为d-type,但是使用的xfs文件系统却不支持d-type。
执行指令:xfs_info /
ftype=0,显示我的文件系统不支持d-type。1表示支持 d_type,0表示不支持。
问题解决:
修改docker的Storage Driver:将d-type改为devicemapper
vi /etc/docker/daemon.json
{
"storage-driver":"devicemapper",
"registry-mirrors": [ "https://registry.docker-cn.com"],
"insecure-registries": [ "192.168.7.XXX:5000"]
}
docker info 可以查看Storage Driver信息
查看操作系统是否支持overlay:
lsmod |grep over
如果没有输出,表示不支持,可以通过下面的命令开启overlay
modprobe overlay