Delete pod requests are missing requestObject field

[cswinter]

Current and expected behavior

I was seeing some strange behavior for pod deletions: #879
I went ahead and enabled audit logging to see how the requests generated by kube-rs and kubectl differ, and kube-rs request doesn't look quite right.

Running kubectl delete pod POD --grace-period=0 actually generates two requests:

kubectl request 1

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "RequestResponse",
    "auditID": "a1b89a81-0634-4866-bef9-b4fdc082924d",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/namespaces/clemens/pods/xprun.22559675820e4c00ba5dc7add3b29dfe.main-0",
    "verb": "delete",
    "user": {
        "username": "admin",
        "uid": "admin",
        "groups": [
            "system:masters",
            "system:authenticated"
        ]
    },
    "sourceIPs": [
        "<REDACTED>"
    ],
    "userAgent": "kubectl/v1.23.4 (linux/amd64) kubernetes/e6c093d",
    "objectRef": {
        "resource": "pods",
        "namespace": "clemens",
        "name": "xprun.22559675820e4c00ba5dc7add3b29dfe.main-0",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 200
    },
    "responseObject": "<OMMITED>",
    "requestReceivedTimestamp": "2022-04-18T00:27:08.589504Z",
    "stageTimestamp": "2022-04-18T00:27:08.604811Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}

kubectl request 2

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "RequestResponse",
    "auditID": "93894db4-cdf5-4327-91a5-76d41dcb4851",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/namespaces/clemens/pods/xprun.22559675820e4c00ba5dc7add3b29dfe.main-0",
    "verb": "delete",
    "user": {
        "username": "system:node:ubuntu",
        "uid": "kubelet-0",
        "groups": [
            "system:nodes",
            "system:authenticated"
        ]
    },
    "sourceIPs": [
        "127.0.0.1"
    ],
    "userAgent": "Go-http-client/2.0",
    "objectRef": {
        "resource": "pods",
        "namespace": "clemens",
        "name": "xprun.22559675820e4c00ba5dc7add3b29dfe.main-0",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 200
    },
    "requestObject": {
        "kind": "DeleteOptions",
        "apiVersion": "v1",
        "gracePeriodSeconds": 0,
        "preconditions": {
            "uid": "be0105bf-5251-4f59-9dc1-3b2e3961c973"
        }
    },
    "responseObject": "<OMMITTED>",
    "requestReceivedTimestamp": "2022-04-18T00:27:11.164468Z",
    "stageTimestamp": "2022-04-18T00:27:11.191063Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}

For the kube-rs request, the requestObject field is missing and the "gracePeriodSeconds" is not specified anywhere:

kube-rs request

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "RequestResponse",
    "auditID": "f46a99cb-b59d-4883-a06d-41a68eaecc70",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/namespaces/clemens/pods/xprun.22559675820e4c00ba5dc7add3b29dfe.main-0?",
    "verb": "delete",
    "user": {
        "username": "admin",
        "uid": "admin",
        "groups": [
            "system:masters",
            "system:authenticated"
        ]
    },
    "sourceIPs": [
        "<REDACTED>"
    ],
    "objectRef": {
        "resource": "pods",
        "namespace": "clemens",
        "name": "xprun.22559675820e4c00ba5dc7add3b29dfe.main-0",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 200
    },
    "responseObject": "<OMITTED>",
    "requestReceivedTimestamp": "2022-04-18T00:25:41.657579Z",
    "stageTimestamp": "2022-04-18T00:25:41.670201Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}

Possible solution

No response

Additional context

No response

Environment

Client Version: v1.23.4
Server Version: v1.23.5-2+c812603a312d2b

Configuration and features

No response

Affected crates

No response

Would you like to work on fixing this bug?

No response

[clux]

hm, not sure why you are getting two requests on the audit, but i'm not used to reading it from there.
at any rate, doing it locally with kubectl delete pod version-76f8f8fcf7-chw2k --grace-period=0 -v=9 shows one DELETE request (and some irrelevant get requests):

Request Body: {"gracePeriodSeconds":1,"propagationPolicy":"Background"}
curl -v -XDELETE  -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: kubectl/v1.23.5 (linux/amd64) kubernetes/c285e78" 'https://0.0.0.0:39301/api/v1/namespaces/default/pods/version-76f8f8fcf7-chw2k'

The API does not really block in any of these cases; none of the DeleteParams (grace period or propagation policy) can be used to control api blocking.

The part where kubectl waits for the container to be deleted is just its own magic. They frequently deviate from the API to provide a nicer UX.

If you want to wait for a pod to be deleted you should use the is_deleted condition
via something like await_condition(api, name, conditions::is_deleted(&deleted_obj_uid)).

[cswinter]

So I figured out the issue. Basically, using a gracePeriodSeconds of 0 behaves very weirdly. The pod is immediately deleted without entering a Terminating state, but the associated container keeps running for tens of seconds. I assume this is why kubectl turns a grace period of 0 into a grace period of 1. When using a gracePeriodSeconds of 1, the pod enters Terminating state as expected, and the pod object is not deleted until the container process has actually been killed.

[cswinter]

Curiously, with a gracePeriodSeconds of 1 the container process is killed very quickly, much faster than with a gracePeriodSeconds of 0.

[olix0r]

Note the docs for DeleteParams

https://github.com/kube-rs/kube-rs/blob/276719c0174fe4e3b2bd99f1353e316eac429e9e/kube-core/src/params.rs#L343-L349

I read this to mean that the resource is deleted from the API immediately (and does not wait for normal state transitions).

[clux]

Yeah, it's a bit confusing. This means the apiserver will send a force delete immediately when receiving the request. The API will return 200 regardless. Should add a clarification here.