Cortex Ingestor PV running out of disk space in a weeks time #5566
Description
We have a cortex setup which supports 8 tenants and ~12 Mil active series of metrics. We have 30 ingestors, each with100gb PV (disk). Longterm metrics are stored in Azure Blob Storage.
We are observing Ingestor PV size is increasing gradually over 2 days and hitting maximum PV capacity soon. We have configured Ingestor to hold blocks for upto 12 hrs. Considering ingestor blocks are pushed to Azure Storage every 2 hours and removed from ingestor every 12 hours (after block expiry), we couldn't understand why ingestor PV size is increasing gradually, even though there is no change in Active series or number of tenants.
Appreciate some help understanding this behavior and guidance on how to plan Ingestor disk capacity.
