Somebody set the Mimir querier to 5.6 GiB of memory a long time ago, "to be safe". Eight replicas later, the cluster reserves 44.8 GiB for a component that has never used more than 80 MiB. This is a real case. A gap this large is unusual, but it makes each step easy to see. Here is how the assistant notices this and decides the right number. It never lets the guess of an AI model (LLM) go below what it measured.
The metrics scan runs daily. For each metrics tenant, it asks for the CPU and memory working set (the memory a container really uses). It also asks for the configured requests, OOM kills and restarts. It does this for every container in every namespace, over a 30-day window.
The assistant does not collect any data itself. It reads the Prometheus or Mimir data you already have, read-only.
A pre-filter picks the interesting ones. It always compares usage with the request (the amount the container asks the cluster to reserve):
mimir-querier requests 5.6 GiB and uses about 64 MiB:
roughly 1% of what it reserves.
An average hides bursts, so the sizing never rests on it alone. For each container the assistant keeps the average, the 95th percentile (p95) and the peak working set. It takes the highest value across all 8 replicas that ran in the window.
Here the three numbers sit close together. The average is 64 MiB, p95 is 76 MiB and the peak is 80 MiB.
Volatility is peak divided by p95. At 1.25 or below, the workload is flat. It has no real spikes. A big gap between request and limit is then just reserved memory that nobody uses.
Above 1.25 it is spiky. A second number then decides how much headroom to reserve. This number is namespace concurrency. It shows if the pods around it peak at the same time or take turns.
The querier: 80 / 76 = 1.06, flat.
The assistant sends the measured numbers, the classification and the current manifest to the AI model. It also sends the sizing rules for each case. The AI model returns a small structured proposal. The proposal lists which values to change, and why.
The AI model may answer "keep as proposed". The goal is not to tune for the sake of tuning.
AI models sometimes ignore their own rules. So the code enforces the hard limits again after the AI model answers:
Here the AI model suggested a 64 MiB request. That is below the 80 MiB peak, so the code raised it.
The assistant writes the change back into the file in place. Comments, key order and every other value stay exactly as they were. It rounds values up to standard steps (64, 96, 128 MiB…), never down.
For a HelmRelease (a Flux object that installs a Helm chart), the edit goes into
spec.values. The chart reads its values from there. The edit does not go into rendered output.
Every MR that touches resources carries the measurements it used for sizing. These are the classification, volatility, namespace concurrency, average, p95 and peak. They also include how many replicas the assistant observed and which safety clamps fired. You can check the arithmetic without opening Grafana.
# working set, per container avg by (namespace, pod, container) (container_memory_working_set_bytes{container!=""}) # CPU, per container rate(container_cpu_usage_seconds_total{container!=""}[5m]) # what each container asked for kube_pod_container_resource_requests{resource="memory"} # OOM kills that survive the restart kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} → 1,284 containers · 61 namespaces · 30 days
volatility = peak / p95 = 80 / 76 = 1.06 # ≤ 1.25 → flat concurrency = 14.2% # namespace pods rarely peak together
{
"requests": { "memory": "64Mi" },
"limits": { "memory": "128Mi", "cpu": "2800m" },
"rationale": "Flat workload (1.06). Request to typical use,
limit about 1.5x peak. Existing CPU limit kept."
}
values: querier: replicas: 8 # sized for query fan-out, see runbook resources: requests: cpu: 500m - memory: 5.6Gi + memory: 96Mi limits: - cpu: 2800m - memory: 5.6Gi + memory: 128Mi
The numbers come from an example run, but the shape is real. Every MR that touches resources carries this evidence block and these clamp notes today.
gitops-ai/rightsize-mimir-querier-94a2 into main · 1 file · +2 −3The querier component of the mimir HelmRelease in monitoring requests 5.6 GiB of memory per replica. In the last 30 days it used at most 80 MiB, across all 8 replicas. This MR brings the request to 96 MiB and the limit to 128 MiB. It also removes the CPU limit.
The workload is flat (peak / p95 = 1.06), so request and limit stay close together. The request covers the observed peak. The limit leaves about 1.5× peak of headroom. The assistant rounds values up to the nearest standard step.
About 44 GiB of reserved memory across 8 replicas. The scheduler can now give it to other pods.
| Signal | Value | Source |
|---|---|---|
| Classification | flat | peak / p95 ≤ 1.25 |
| Volatility (peak / p95) | 1.06 | working set, 30d |
| Namespace concurrency | 14.2% | joint vs summed peaks in monitoring |
| Memory avg / p95 / peak | 64Mi / 76Mi / 80Mi | max across replicas |
| Replicas observed | 8 | 30-day window |
| OOM kills / restarts (24h) | 0 / 0 | kube-state-metrics |
| Current request / limit | 5.6Gi / 5.6Gi, CPU limit 2800m | HelmRelease values |
| Applied request / limit | 96Mi / 128Mi, no CPU limit | after clamps |
values:
querier:
replicas: 8 # sized for query fan-out, see runbook
resources:
requests:
cpu: 500m
- memory: 5.6Gi
+ memory: 96Mi
limits:
- cpu: 2800m
- memory: 5.6Gi
+ memory: 128Mi
mimir HelmRelease. Querier pods restart one at a time.container_memory_working_set_bytes, container querier).OOMKilled terminations on the querier over the next week.Revert this MR. The next nightly scan measures again either way. If a workload starts to hit its limit, the "over 85%" rule finds it.
Checked the cited peak against the tenant's 30-day data. Checked the chart's values path for querier.resources. The request covers the peak. The limit is within bounds. LGTM.
The AI model proposes a number. Measured data and fixed rules decide if it ships.