Seeing slow updates to client-side field indexers

[jonathan-innis]

While scale testing kubernetes-sigs/karpenter, we created thousands of pending pods at one time. We have a cache field indexer on pending pods that we use to get all of the pending pods on the cluster before we start executing our scheduling loop. We noticed (under high load) that the pending pods were deployed to the cluster but Karpenter was not attempting to schedule these pending pods/did not retrieve these pending pods back when pulling them from the field indexer.

In particular, this seemed like the field indexer may be single threaded or may be bottlenecking under high load. What was stranger is that Karpenter has other threads that it runs in its containers where it publishes metrics on pods. These threads (which weren't using the field indexers) observed the pending pods on the cluster much quicker than the thread that was using the field indexers.

Do we know if there is anything that could cause the field indexers to be slow when placed under high load (are they single threaded) and are there ways that we could improve this performance so that we can rely on these indexers? Or should we look to remove them if we care deeply about the performance of retrieving these pods?

[alvaroaleman]

It is a bit hidden, but the relevant code is not maintained in controller-runtime but imported, specifically its the AddIndexers here: https://pkg.go.dev/k8s.io/client-go/tools/cache#SharedIndexInformer

Refs:

1. controller-runtime/pkg/cache/internal/informers.go

   Line 60 in 990f2ed

   newInformer := cache.NewSharedIndexInformer

2. controller-runtime/pkg/cache/informer_cache.go

   Line 259 in 990f2ed

   return informer.AddIndexers(cache.Indexers{internal.FieldIndexName(field): indexFunc})

I suggest to create an issue in k/k and link it here