Incidents do not respect team boundaries. The cause of a Kubernetes alert can be a network uplink. The cause of a failed Ansible run can be a Vault (secrets store) pod three layers down that nobody noticed was sealed (locked). This story shows where the assistant is going: it will join what it knows across inventory, network, CI, cluster and cloud. Then you get one root-cause finding instead of a pile of symptoms. Most of it is designed but not yet built. Each step says which.
orders-api is timing out, payments reports
its database unreachable, a node exporter goes quiet, a queue backs
up. Each alert, read on its own, points at a different service and
a different owner.
Today each one gets its own investigation and, if it warrants one, its own issue. Sixteen honest findings about the wrong thing.
A rule-based correlator of co-firing alerts knows that
KubeletDown explains the pod alerts on that node. It also knows that a scraper going dark makes its targets look down. It elects a root and annotates the others as dependents. It stays within one tenant, so it never mislabels an outage elsewhere as "fix the root".
It ships switched off and only annotates. It does not yet collapse N issues into one. It also cannot see what is outside the cluster: a switch, an uplink or a VLAN.
The assistant will join the instance or node label of each alert to a NetBox record, inside the same process. The join goes from address to host, host to interface, and interface to switch port and VLAN. Fourteen of the
sixteen alerts land on hosts behind the same access switch uplink.
The join is strict. A key that matches two hosts abstains instead of guessing. The assistant says "not in NetBox" only after a complete read.
For VM and bare-metal infrastructure that Zabbix monitors, the assistant will treat Zabbix problems as one more alert source. It will also use the Zabbix host inventory (OS, hardware, resources) as observed context beside NetBox.
Here Zabbix raised an interface-down problem on the switch uplink about ninety seconds before the first Kubernetes alert. Two independent sources, one cause.
Inventory is core-infrastructure detail. The assistant will use addresses, MAC addresses, serials, VLAN ids, racks and site names for the join, then drop them. What reaches an AI model (LLM) is a closed-schema card. The card holds the role, platform family, status, update cadence, whether an owner is declared and an opaque team handle.
The assistant will express topology as relations between opaque handles, not as invented addresses. By default, it will drop any field that is not on the allowlist. A canary test will prove that no raw value survives.
The assistant will fold the dependents into a single issue. The issue names the shared uplink, lists every alert it explains, and states what could not be joined. The two alerts that did not fit the pattern keep their own investigation.
The scheduled Ansible apply mints its SSH certificate from Vault
before any play runs. Since last night it fails at that step:
HTTP 503, Vault is sealed. The playbook repository is fine.
The problem is somewhere else.
Rules, not an AI model, will match the failure signature (vault … 503 … sealed). The CI script itself names the Vault address and auth mount, so the dependency is declared, not guessed. The cluster map will resolve that Vault to its HelmRelease in the cluster repository.
In the cluster, the Vault pods are Running, 0/1 ready,
zero restarts. The one auto-unseal attempt at startup failed against Google Cloud KMS. Nothing ever tried again.
A rule in the availability pack looks for exactly this shape: a Vault HelmRelease with KMS auto-unseal and no liveness probe. KMS auto-unseal means a cloud key service unlocks Vault at startup. A liveness probe is a check that restarts a stuck pod. Readiness only takes a sealed pod out of the Service. A liveness
probe on /v1/sys/health restarts the pod, and the restart retries the unseal.
The rule is meant to fire on its own, outage or not. The planned attribution in step 8 is what lets the MR cite the playbook failures as evidence.
name: db-07.dc1.example primary_ip: 192.0.2.47 interface: eth0 · aa:bb:cc:… switch_port: sw-dc1-r4 Gi1/0/17 vlan: 112 serial: XK4… owner: (a person) comments: free text
host: h-5c1e role: database platform: vm status: active update_cadence: monthly owner_declared: true owner_team: team-91a0 address_scope: private behind: sw-3f02 (uplink u-1)
$ ci/mint-ssh-cert requesting fleet SSH certificate from Vault vault PUT auth/fleet/login: HTTP 503 ['Vault is sealed'] ✕ job failed before ansible-playbook started scheduled applies, newest first ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ … 12 in a row
spec: chart: { spec: { chart: vault } } values: server: ha: enabled: true config: | seal "gcpckms" { key_ring = "…" crypto_key = "vault-unseal" } # no livenessProbe
Illustrative, and on the roadmap as a whole. The probe values and the reasoning are the ones we use to build the rule. The cross-repository evidence section is the planned attribution step.
gitops-ai/vault-autounseal-liveness into main · 1 file · +7 −0This Vault unseals itself through Google Cloud KMS. Vault attempts auto-unseal once, at startup. If that attempt fails, the pod stays sealed and Running indefinitely. Readiness only removes it from the Service. Nothing restarts it.
This MR enables the chart's liveness probe on /v1/sys/health?standbyok=true. A sealed pod fails the probe. The cluster restarts it, and the restart retries the unseal.
initialDelaySeconds: 60, then failureThreshold: 6 × periodSeconds: 10 with a 3 s timeout. So a KMS blip of up to a minute does not cause a restart loop.standbyok=true keeps healthy standbys at 200. It does not mask 474 (a standby that lost its active node), so that case restarts too.If this Vault is uninitialised (501) or runs as an Enterprise DR / performance standby (472 / 473), this probe would restart-loop it. Check that neither applies before merging.
The scheduled Ansible apply in the playbook repository has failed 12 times in a row at the certificate step with HTTP 503 ['Vault is sealed']. The CI script declares this Vault's address and auth mount, so the assistant attributes the failures here, to the repository that owns the Vault.
| Evidence | Source |
|---|---|
| 12 consecutive failed scheduled applies, same signature | playbook repository · CI job logs |
| Vault address + auth mount named in the CI script | playbook repository · declared edge |
| 3 pods Running, 0/1 ready, 0 restarts, sealed | cluster · read-only |
| KMS auto-unseal, no liveness probe | cluster repository · this HelmRelease |
A comment on the latest failed run links this MR. The assistant proposes no change there, because the playbook is correct. It was only the first thing to notice.
valuesFrom or post-renderer could override the probe.vault status shows Sealed: false on all replicas.Revert this MR. The probe is the only change.
server:
ha:
enabled: true
config: |
…
+ livenessProbe:
+ enabled: true
+ path: "/v1/sys/health?standbyok=true"
+ initialDelaySeconds: 60
+ periodSeconds: 10
+ failureThreshold: 6
+ timeoutSeconds: 3
Confirmed the seal stanza is live config and KMS-backed, and that the probe path tolerates standbys. The restart-on-sealed behaviour is the intended fix. LGTM.
Joining domains multiplies what can go wrong. These are the design rules, not afterthoughts.