High spikes of ingester inflight query requests #7107
Description
Hi,
My team was looking into setting the max_inflight_query_requests limit to help protect our ingesters from OOMKills due to heavy query load. However, we were reviewing our data to figure out what a good value would be and noticed massive spikes in ingester inflight query requests whenever blocks were shipped:
We have a ship interval of 2h, and confirmed that these spikes occur during the same times our ingesters are shipping blocks. We also looked at metrics for the ingress we have in front of our Cortex deployment and do not believe the spikes are related to user activity, this seems to be triggered by some internal Cortex component.
Can you all share what component might be causing this high number of requests? We are wondering if either the store-gateways or compactors might be responsible? If an internal component is making a high number of query requests, we are concerned about setting this limit too low and causing issues.
Thanks!