Article directory
本文使用的rancher版本是v2.7.5
1. Fault phenomenon
During the execution of the code in the red box below in the cluster environment that needs to be imported, due to the first operation error, the cluster has been unable to be imported normally. The namespace (cattle-system) that Rancher depends on is deleted, and the status is always Terminating.
As shown below:
2. Solution
After referring to my other blog, the problem is still not solved after the steps of the solution to the problem that the kubernetes namespace Terminating status cannot be deleted .
The following is the final solution to this problem, execute the following command:
kubectl patch namespace cattle-system -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system
kubectl delete namespace cattle-system --grace-period=0 --force
kubectl patch namespace cattle-global-data -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system
kubectl delete namespace cattle-global-data --grace-period=0 --force
kubectl patch namespace local -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system
for resource in `kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -o name -n local`; do kubectl patch $resource -p '{"metadata": {"finalizers": []}}' --type='merge' -n local; done
kubectl delete namespace local --grace-period=0 --force
You may encounter new problems during the implementation process. I have summarized them as follows:
Question one
(1) Problem description
Error from server (InternalError): Internal error occurred: failed calling webhook “rancher.cattle.io.namespaces.create-non-kubesystem”: failed to call webhook: Post “https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s”: service “rancher-webhook” not found；
(2) Solution
[root@k8s-master rancher]# kubectl get MutatingWebhookConfiguration
NAME WEBHOOKS AGE
cert-manager-webhook 1 5h50m
mutating-webhook-configuration 8 5h49m
rancher.cattle.io 5 120m
[root@k8s-master rancher]# kubectl delete MutatingWebhookConfiguration rancher.cattle.io
mutatingwebhookconfiguration.admissionregistration.k8s.io "rancher.cattle.io" deleted
[root@k8s-master rancher]# kubectl get ValidatingWebhookConfiguration
NAME WEBHOOKS AGE
cert-manager-webhook 1 5h51m
ingress-nginx-admission 1 6h6m
rancher.cattle.io 13 121m
validating-webhook-configuration 11 5h50m
[root@k8s-master rancher]# kubectl delete ValidatingWebhookConfiguration rancher.cattle.io
validatingwebhookconfiguration.admissionregistration.k8s.io "rancher.cattle.io" deleted
[root@k8s-master rancher]# kubectl create ns cattle-system
namespace/cattle-system created
View MutatingWebhookConfiguration and ValidatingWebhookConfiguration
Use delete to delete MutatingWebhookConfiguration and ValidatingWebhookConfiguration that affect the operation
Finally, re-create the namespace.
(3) Explanation
ValidatingWebhookConfiguration is an implementation form of the extensible admission controllers (Admission Controllers) mechanism in Kubernetes. It defines a set of Webhook rules for automatic verification and correction of newly created or modified Kubernetes resources. Specifically, when the Kubernetes API Server receives a new resource submission request, it will call the configured Validaing Admission Controller, select the corresponding Webhook according to different ValidatingWebhookConfiguration for verification and correction, and return the results to the API Server for response. In this way, some common verification and correction operations can be automatically completed before resources enter the Kubernetes cluster, such as enforcing best practices, compatibility checks, security audits, etc.
MutatingWebhookConfiguration is similar to ValidatingWebhookConfiguration. It is an implementation form of the extensible admission controller (Admission Controllers) mechanism in Kubernetes. It defines a set of Webhook rules for new resource submission requests when the Kubernetes API Server receives it. Resources are automatically modified to achieve the purpose of automatic resource management. Specifically, when the Kubernetes API Server receives the request, it will select the corresponding Webhook according to different MutatingWebhookConfiguration configurations to automatically modify the resources, and return the modified results to the API Server for response.
My problem is solved here, I hope it can help other coders.
Finished, call it a day