Silencing an alert is easy. The harder part is to find out why it fired, then change the configuration so it does not fire again. Teams often postpone that part. The assistant picks up each new alert and investigates it with read-only tools. Before it writes anything, it asks one question: could a change in git have prevented this?
Every 15 minutes the assistant collects alerts from PagerDuty and Alertmanager (what is firing right now, with silences). It also reads Prometheus alert history, which catches alerts that flap faster than a page.
The assistant handles each alert once. If the same rule fires on twenty filesystems, the assistant merges them into one "this is widespread" case. It does not run twenty investigations.
The assistant resolves the alert's target (kind, namespace, name) from its labels. Before the investigation starts, it fetches a snapshot of the live object and its git manifest. If the alert body is thin, it reads the rule definition itself.
If no tracked repository declares the workload, the outcome is already known: an issue, never an edit to an unrelated file.
The investigation is a loop of read-only tool calls against the cluster. The pod's last state says OOMKilled (the system stopped it for using too much memory), exit code 137, seven restarts. Events confirm the pattern of back-off, restart, back-off.
The cluster identity behind these tools can read. It cannot change anything.
Over the last two days, the memory working set climbs every night during the catalogue reload. It touches the 512 MiB limit, and then the system kills the container. Daytime use is well below.
That is not a leak (it drops back each time) and not random. It is a known peak that no longer fits.
Where you have enabled log access for a cluster, the assistant reads a short tail of the previous container, the one that died. Here the last line before the kill shows the catalogue reload. The reload has a much larger item count than a year ago.
Without log access it does not guess. It says what it could not see and hands you the command to look.
Root cause: the catalogue grew, so the reload's peak outgrew a memory limit that someone set long ago. A limit is a manifest value, so the answer is yes, and the output is a draft MR.
Suppose the cause is outside git: a node problem, a workload in a repository you do not track, or a bug that needs a code change. Then the answer is no. The assistant does not allow speculative "bump it just in case" fixes.
Before anyone sees the drafted fix, rule-based checks run on it. These checks are code, not an AI model (LLM):
If a check fails, the assistant holds the fix for human review and names the reason.
The MR carries the alert, the evidence trail and the reasoning. The issue carries what was observed, the copy-paste commands to finish the diagnosis, and the likely fix for each outcome.
The loop does not end at the proposal. The assistant closes an issue with a note when its alert stays quiet for long enough. If a merged fix does not stop its alert, the assistant escalates and links back to the MR.
→ prometheus_alert_rule("KubePodCrashLooping") expr: max_over_time(kube_pod_container_status_waiting_reason{ reason="CrashLoopBackOff"}[5m]) >= 1 → read_workload_manifest_from_git(Deployment, shop, orders-api) found: shop-deploy · apps/shop/orders-api/deployment.yaml resources.limits.memory: 512Mi requests.memory: 384Mi last change to resources: 14 months ago
→ kubernetes_describe_object(Pod, shop, orders-api-7d9f8c-x2kqp) lastState.terminated: reason: OOMKilled exitCode: 137 restartCount: 7 → workload_restart_history(shop, orders-api) 7 restarts / 48h · all OOMKilled · all between 02:00 and 02:20 → kubernetes_list_events(shop, orders-api) BackOff Back-off restarting failed container api ×23 Killing Container api exceeded its memory limit ×7
02:10:03 INFO scheduler: starting nightly catalogue reload 02:10:04 INFO catalogue: fetching snapshot from products-db 02:10:41 INFO catalogue: decoded 1,940,212 items (prev 1,211,480) 02:10:58 INFO catalogue: building search index (container terminated: OOMKilled)
orders-api in deployment.yaml, sized from the measured peak.resources: requests: - memory: 384Mi + memory: 512Mi limits: - memory: 512Mi + memory: 768Mi
Illustrative, shaped like the real output. The alert, the investigation trail and the sizing reasoning travel with the one-file change.
gitops-ai/alert-orders-api-oomkilled-5c81e0a2 into main · 1 file · +2 −2KubePodCrashLooping (critical) on shop/orders-api, container api, firing since 02:14. Third firing in 7 days.
The container is OOMKilled during the nightly catalogue reload (02:10). The catalogue grew from about 1.2M to 1.9M items. The reload briefly holds the old and new copies. The peak now exceeds the 512 Mi limit set 14 months ago. Daytime usage (≈300 Mi) is unaffected. Memory drops back after each restart, so this is not a leak.
Request 384 Mi → 512 Mi, limit 512 Mi → 768 Mi. The old limit caps the observed peak, so the true need is not known exactly. The new limit leaves 1.5× the last observed peak. The post-merge checks below confirm it.
Streaming the reload instead of double-buffering it would remove the peak altogether. That is an application change. The MR notes it for the service owners and does not propose it.
| # | Tool (read-only) | Finding |
|---|---|---|
| 1 | prometheus_alert_rule | Crash-loop rule, 5 min window |
| 2 | read_workload_manifest_from_git | apps/shop/orders-api/deployment.yaml, limit 512Mi |
| 3 | kubernetes_describe_object | lastState OOMKilled, exit 137, 7 restarts |
| 4 | workload_restart_history | All 7 restarts between 02:00 and 02:20 |
| 5 | kubernetes_list_events | BackOff ×23, memory-limit kills ×7 |
| 6 | prometheus_query_range | Nightly climb to the limit, daytime ≈300Mi |
| 7 | kubernetes_pod_logs (previous) | Killed while building the index after reload |
Second pass (OOM class): confirmed under-resourced, not leaking. Memory returns to baseline after every restart.
containers:
- name: api
resources:
requests:
cpu: 250m
- memory: 384Mi
+ memory: 512Mi
limits:
- memory: 512Mi
+ memory: 768Mi
orders-api. Pods restart once with the new limits.OOMKilled in kubectl get pod -n shop -l app.kubernetes.io/name=orders-api after 02:30.KubePodCrashLooping for shop/orders-api stays resolved.The assistant watches for it. If a merged fix does not stop its alert, the assistant opens an escalation issue and comments on this MR.
Restart times match the reload window in the cited logs. The change touches only the alerted container's memory. Request stays below limit. LGTM.
An alert is a symptom. Most of the rules below exist to stop a confident-sounding fix for the wrong thing.