Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

object doesn't get updated after fakeClient.Delete(object) #3059

Open
Nihileon opened this issue Jan 3, 2025 · 15 comments
Open

object doesn't get updated after fakeClient.Delete(object) #3059

Nihileon opened this issue Jan 3, 2025 · 15 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@Nihileon
Copy link

Nihileon commented Jan 3, 2025

fakeClient.Delete(obj) doesn't update deletionTimestamp of the obj passed to the function. The bug is in fakeClient.deleteObject, since it gets a new object and updates the new object without rewriting the original object.

Reference: https://github.com/kubernetes-sigs/controller-runtime/blob/v0.19.0/pkg/client/fake/client.go#L1085-L1095

@Nihileon Nihileon changed the title Object doesn't get updated after fakeClient.Delete object doesn't get updated after fakeClient.Delete(object) Jan 3, 2025
@alvaroaleman alvaroaleman added the kind/bug Categorizes issue or PR as related to a bug. label Jan 7, 2025
@sbueringer
Copy link
Member

I assume the regular client updates it? (I think I remember that the regular client also doesn't update it, but I'm not sure)

@alvaroaleman
Copy link
Member

Actually it looks like it doesn't:

Body(deleteOpts.AsDeleteOptions()).

So after someone deletes an object, their in-mem representation doesn't reflect that? That is different from all other methods and very confusing

@sbueringer
Copy link
Member

Yeah. I think I noticed this 1-2 months ago in a unit test. First I thought the fake client is the problem but then I saw that the real client has the same behavior :)

@troy0820
Copy link
Member

troy0820 commented Jan 8, 2025

I can take this one if you all don't mind.

/assign @troy0820

@sbueringer
Copy link
Member

I'm wondering if there are comparable cases in client-go and how they are handled there

@troy0820
Copy link
Member

/unassign @troy0820

I unfortunately do not have the time to commit to work through the PR for this. I am going to unassign myself for anyone who wants to take this.

@troy0820
Copy link
Member

Yeah. I think I noticed this 1-2 months ago in a unit test. First I thought the fake client is the problem but then I saw that the real client has the same behavior :)

Do we want to change the real client behavior and then map that to the fakeClient?

@sbueringer
Copy link
Member

sbueringer commented Jan 28, 2025

I'm wondering if there are comparable cases in client-go and how they are handled there

I think knowing ^^ would be good to inform our decision.

Do we get the object with deletionTimestamp back when we do the actual delete call?

@troy0820
Copy link
Member

troy0820 commented Jan 28, 2025

I'm wondering if there are comparable cases in client-go and how they are handled there

I think knowing ^^ would be good to inform our decision.

Do we get the object with deletionTimestamp back when we do the actual delete call?

I don't think we do unless I am reading this wrong on the dynamic client which uses the rest.Interface.

https://github.com/kubernetes/client-go/blob/f2030849e1d6d961fa150b9be9560da33e5d00df/dynamic/simple.go#L205-L219

func (c *dynamicResourceClient) Delete(ctx context.Context, name string, opts metav1.DeleteOptions, subresources ...string) error {
	if len(name) == 0 {
		return fmt.Errorf("name is required")
	}
	if err := validateNamespaceWithOptionalName(c.namespace, name); err != nil {
		return err
	}

	result := c.client.client.
		Delete().
		AbsPath(append(c.makeURLSegments(name), subresources...)...).
		Body(&opts).
		Do(ctx)
	return result.Error()
}

@sbueringer
Copy link
Member

Looks like it. Not sure if there either simply is nothing in the response body or if we just don't read it

@bigkevmcd
Copy link
Contributor

I think simply modifying typedClient.Delete()

-               Error()
+               Into(obj)

From my testing, If there's a finalizer on the resource then you will get a response including the timestamp, otherwise it gets deleted, and there's no deletion timestamp.

@troy0820
Copy link
Member

I think simply modifying typedClient.Delete()
From my testing, If there's a finalizer on the resource then you will get a response including the timestamp, otherwise it gets deleted, and there's no deletion timestamp.

I had this in a branch after I closed the first one but the behavior doesn't seem to mimic what the restClient would do at least to be used outside of fakeClient.

If we start assuming the response body would be there instead of a 204 "No content" I think that would be dangerous and not compatible with upstream?

If we need to simulate what the fake client would provide back as a timestamp to the object that gets persisted, that's a different story on how we can create that mechanism with what you describe.

@bigkevmcd
Copy link
Contributor

I had assumed that a No Content response would happen too.

But try...

$ kubectl create configmap test-configmap
$ kubectl delete -v9 configmap/my-config

You'll see that we get a response body

I0129 06:10:45.071084   71427 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","details":{"name":"my-config","kind":"configmaps","uid":"ca64cd2f-cbec-4123-8d5e-a63914e4d72c"}}

Recreating the configmap and editing it to have a finalizer causes the kubectl delete to hang, and editing the configmap in a separate terminal to remove the finalizer results in the kubectl delete completing with full logs.

$ kubectl delete -v9 configmap/my-config
I0129 06:14:25.566840   71551 loader.go:395] Config loaded from file:  /Users/kevin/.kube/config
I0129 06:14:25.571589   71551 request.go:1212] Request Body: {"propagationPolicy":"Background"}
I0129 06:14:25.571681   71551 round_trippers.go:466] curl -v -XDELETE  -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.30.5 (darwin/amd64) kubernetes/74e84a9" 'https://kubernetes.docker.internal:6443/api/v1/namespaces/default/configmaps/my-config'
I0129 06:14:25.573145   71551 round_trippers.go:495] HTTP Trace: DNS Lookup for kubernetes.docker.internal resolved to [{127.0.0.1 }]
I0129 06:14:25.573497   71551 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:6443 succeed
I0129 06:14:25.584263   71551 round_trippers.go:553] DELETE https://kubernetes.docker.internal:6443/api/v1/namespaces/default/configmaps/my-config 200 OK in 12 milliseconds
I0129 06:14:25.584291   71551 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 4 ms ServerProcessing 5 ms Duration 12 ms
I0129 06:14:25.584298   71551 round_trippers.go:577] Response Headers:
I0129 06:14:25.584305   71551 round_trippers.go:580]     Content-Type: application/json
I0129 06:14:25.584309   71551 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: a062054d-b8f6-4995-8c0e-d62de5fa33a5
I0129 06:14:25.584312   71551 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: b60e697d-f60e-4c74-831c-2751b65ff672
I0129 06:14:25.584315   71551 round_trippers.go:580]     Content-Length: 317
I0129 06:14:25.584318   71551 round_trippers.go:580]     Date: Wed, 29 Jan 2025 06:14:25 GMT
I0129 06:14:25.584320   71551 round_trippers.go:580]     Audit-Id: 982435e0-8bb2-4266-a4a5-f5ae05340fe8
I0129 06:14:25.584323   71551 round_trippers.go:580]     Cache-Control: no-cache, private
I0129 06:14:25.584372   71551 request.go:1212] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-config","namespace":"default","uid":"6a3ffd9a-74fd-4c3d-b3d7-595d23107a8e","resourceVersion":"50004","creationTimestamp":"2025-01-29T06:12:30Z","deletionTimestamp":"2025-01-29T06:14:25Z","deletionGracePeriodSeconds":0,"finalizers":["bigkevmcd.com/test"]}}
configmap "my-config" deleted

These indicate that we get the deletion timestamp back.

Running this code https://gist.github.com/bigkevmcd/2be4e0835f6a865cf7a01c1af28a3680 also shows this.

        configMap := &corev1.ConfigMap{
		ObjectMeta: metav1.ObjectMeta{
			Namespace: "default",
			Name:      "my-config",
		},
	}
	if err := k8sClient.Create(context.Background(), configMap); err != nil {
		log.Fatal(err)
	}
	log.Printf("Deletion timestamp = %v\n", configMap.ObjectMeta.DeletionTimestamp)

	controllerutil.AddFinalizer(configMap, "example.com/test")
	if err := k8sClient.Update(context.Background(), configMap); err != nil {
		log.Fatal(err)
	}
	log.Printf("Deletion timestamp = %v\n", configMap.ObjectMeta.DeletionTimestamp)

	if err := k8sClient.Delete(context.Background(), configMap); err != nil {
		log.Fatal(err)
	}
	log.Printf("Deletion timestamp = %v\n", configMap.ObjectMeta.DeletionTimestamp)

	if err := k8sClient.Get(context.Background(), types.NamespacedName{Namespace: "default", Name: "my-config"}, configMap); err != nil {
		log.Fatal(err)
	}
	log.Printf("Deletion timestamp = %v\n", configMap.ObjectMeta.DeletionTimestamp)

	controllerutil.RemoveFinalizer(configMap, "example.com/test")
	if err := k8sClient.Update(context.Background(), configMap); err != nil {
		log.Fatal(err)
	}
	log.Printf("Deletion timestamp = %v\n", configMap.ObjectMeta.DeletionTimestamp)

	if err := k8sClient.Get(context.Background(), types.NamespacedName{Namespace: "default", Name: "my-config"}, configMap); err != nil {
		log.Fatal(err)
	}
	log.Printf("Deletion timestamp = %v\n", configMap.ObjectMeta.DeletionTimestamp)

Outputs...

2025/01/29 06:23:50 Deletion timestamp = <nil>
2025/01/29 06:23:50 Deletion timestamp = <nil>
2025/01/29 06:23:51 Deletion timestamp = 2025-01-29 06:23:50 +0000 GMT
2025/01/29 06:23:51 Deletion timestamp = 2025-01-29 06:23:50 +0000 GMT
2025/01/29 06:23:51 Deletion timestamp = 2025-01-29 06:23:50 +0000 GMT

With a modified version of the code with the change above proposed.

I think this points to a bug in the real-client in that it doesn't update the state on Delete, and the fake should presumably follow the behaviour of the real client?

@bigkevmcd
Copy link
Contributor

I've fixed the first part of this (the real client behaviour, not the fake).

#3098

I'll see what can be done for the fake client for the same behaviour.

@bigkevmcd
Copy link
Contributor

I had a look at the fake Client, and both now work in the same way.

With the real client, the DeletionTimestamp is only populated when the server responds with a resource, which seems to only be when the resource exists after deletion i.e. a finalizer prevents removal.

Switching out the real client for the fake Client in the code above yields the same output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants