一、问题现象
执行kubectl delete ns paas-qa
删除命名空间paas-qa,命令一直处于deleted不返回,查看namespace状态显示Terminating,通过执行kubectl delete ns paas-qa --force --grace-period=0命令也无法清除。
二、解决办法
1、开启一个代理终端
root@qa-opreator:~# kubectl proxy
Starting to serve on 127.0.0.1:8001
2、新开启一个操作终端,将namespace的配置,输出json文件保存
root@qa-opreator:~# kubectl get ns paas-qa -o json > tmp.json
3、执行vim tmp.json
,删除其中spec.finalizers的内容
4、调用接口删除
root@qa-opreator:~# curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/paas-qa/finalize
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "paas-qa",
"selfLink": "/api/v1/namespaces/paas-qa/finalize",
"uid": "9269c2a1-b861-4d27-8e29-cf4f4732cfff",
"resourceVersion": "1804327",
"creationTimestamp": "2023-05-11T09:22:19Z",
"deletionTimestamp": "2023-05-15T10:25:39Z",
"managedFields": [
{
"manager": "kubectl-create",
"operation": "Update",
"apiVersion": "v1",
"time": "2023-05-11T09:22:19Z",
"fieldsType": "FieldsV1",
"fieldsV1": {"f:status":{"f:phase":{}}}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "v1",
"time": "2023-05-15T10:25:45Z",
"fieldsType": "FieldsV1",
"fieldsV1": {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}
}
]
},
"spec": {
},
"status": {
"phase": "Terminating",
"conditions": [
{
"type": "NamespaceDeletionDiscoveryFailure",
"status": "False",
"lastTransitionTime": "2023-05-15T10:25:45Z",
"reason": "ResourcesDiscovered",
"message": "All resources successfully discovered"
},
{
"type": "NamespaceDeletionGroupVersionParsingFailure",
"status": "False",
"lastTransitionTime": "2023-05-15T10:25:45Z",
"reason": "ParsedGroupVersions",
"message": "All legacy kube types successfully parsed"
},
{
"type": "NamespaceDeletionContentFailure",
"status": "False",
"lastTransitionTime": "2023-05-15T10:25:45Z",
"reason": "ContentDeleted",
"message": "All content successfully deleted, may be waiting on finalization"
},
{
"type": "NamespaceContentRemaining",
"status": "True",
"lastTransitionTime": "2023-05-15T10:25:45Z",
"reason": "SomeResourcesRemain",
"message": "Some resources are remaining: kafkaclusters.kafka.banzaicloud.io has 1 resource instances, kafkatopics.kafka.banzaicloud.io has 1 resource instances"
},
{
"type": "NamespaceFinalizersRemaining",
"status": "True",
"lastTransitionTime": "2023-05-15T10:25:45Z",
"reason": "SomeFinalizersRemain",
"message": "Some content in the namespace has finalizers remaining: finalizer.kafkaclusters.kafka.banzaicloud.io in 1 resource instances, finalizer.kafkatopics.kafka.banzaicloud.io in 1 resource instances, topics.kafkaclusters.kafka.banzaicloud.io in 1 resource instances, users.kafkaclusters.kafka.banzaicloud.io in 1 resource instances"
}
]
}
}
查看命名空间,已经正常删除。
附:K8S资源一直Terminating删除
某些原因使得K8S资源一直Terminating,使用kubectl delete显示XXX deleted,然后卡住删不掉,可用如下命令更新资源配置:
kubectl patch 资源类型/资源名称 -p '{"metadata":{"finalizers":[]}}' --type=merge
就是修改其metadata下的finalizers的值为空就可以了。
类似的资源类型,如CRD、pod、deployment、namespaces等,执行过程比前面代理的方式更简单。