01 · Read everything
Flux status and an hour of Warning events.
Every two hours the scan lists all HelmReleases and Kustomizations
and picks out the ones that are Ready=False for a real failure reason. Examples are upgrade failed, build failed, health check failed and dependency not ready.
It also collects every Warning event from the last hour and groups
them by the object they are about. An object needs at least five
warnings before it is worth a look.
02 · Spot churn is not an incident
A node going away on schedule is filtered, carefully.
On a spot cluster, the provider reclaims and replaces nodes all the time. Each replacement produces a handful of node warnings. The assistant drops those only when both conditions hold. First, the node carries the provider's spot label. Second, every reason in the group is a known churn reason.
Anything else on the same node, such as memory pressure or a
read-only filesystem, is still reported, spot or not.
03 · A pod starting up is not broken
Probe bursts on healthy workloads fold into one digest.
Rescheduled pods fail a few readiness checks while they boot. If a
group of warnings is only probe failures and the workload is fully available right now, it does not get its own issue. Fully available means every pod is Ready, with zero restarts and no OOM (out-of-memory) kills.
Those groups fold into one low-risk digest per cluster, so the
pattern stays visible without paging anyone.
04 · Drift, once, not per release
Twelve drifting releases, one digest.
With Helm drift detection in warn mode, healthy releases send a drift warning on every reconcile. The assistant does not open twelve "investigate this HelmRelease" issues. Drift goes to a single drift digest for each cluster.
The digest only reports. It never proposes switching drift correction on
or ignoring fields. That decision stays yours.
05 · What is left is real
One release that stopped trying.
payments-api failed its upgrade and exhausted its
retries. That matters, because a stalled HelmRelease will not retry on its own. Its parent Kustomization shows as failing too.
The issue for the parent Kustomization does not stop at "dependency not ready". It names the child release that holds it up. It adds that release's recent events and, where the assistant can find the pods, a log tail.
06 · Which change did this?
A join with the MRs merged just before it broke.
The assistant joins the first-seen time of the failure with MRs merged in the 30 minutes before it. It scores each candidate on the files it touched, the names it shares with the failing object and how close in time it landed. It discards weak matches instead of guessing.
Here one MR stands out: the chart upgrade for
payments-api, merged seven minutes earlier.
07 · Grouped by cause
From 63 events to a short list.
On a cluster that is simply out of memory, scheduling failures become one capacity issue, not one per workload. It links an ExternalSecret that cannot sync to the workloads that consume its Secret. So the chain reads as one story.
Grouping findings that share an identical error message is
In development. One example is a missing version of a CRD (custom resource definition) behind eleven failing releases.
08 · Closed when it is fixed
Quiet is not the test. Healthy is.
The assistant closes a Flux failure issue when the release is back to
Ready=True at or above the version that failed, and
stays that way. A lull in events is not enough on its own.
Warning-event findings are short-lived by design. They re-alert daily while the problem continues, and stop when it does.