A database migration is booked for Saturday morning. Without a silence (a rule that mutes alerts), it pages the on-call engineer four times. A silence clicked in the UI might still mute critical alerts in three years. Here the window is a small YAML file in git with a hard end time. Everything else follows from that file.
A real audit of one company's infrastructure found silences created years earlier. They had an end date more than a century away and a two-word comment. Their matchers pointed at alert rules that no longer existed anywhere. They muted nothing useful. They also hid whatever came next under the same name.
A better UI is not the fix. The fix is to make every silence a reviewed change with a reason and an end.
Silences live in your alerting-config repository under
silences/<tenant>/<id>.yaml. Each file has matchers, a
start, an end at most 30 days later, a comment and a link to
the change that justifies the silence.
A genuinely long-lived silence is rare. It has a loud escape hatch,
permanent:. This needs a reason, an approver and a
review_by date.
The merge request (MR) pipeline runs a small, self-contained validator. It refuses unknown keys, duplicate keys and multi-document files. The tenant must match its directory. The id must match the file name. The window must have an end. Every regex must compile.
It runs with no network and no secrets, because in many Alertmanager setups a "read-only" certificate can also write. The assistant uses the same validator as its loader. So review and runtime never disagree about the schema.
After merge, a job on the default branch prints a plan. Then it creates the silence. Each silence carries a sentinel comment with its id and generation. So the job never re-creates a past window. If you edit an end time, the next pass reconciles it.
The lint ships with the assistant. The applier is a small, tested script that runs in your pipeline. If the job cannot read a tenant, it fails and does not guess.
An Alertmanager silence does nothing for a PagerDuty incident that is already open. The escalation policy keeps sending notifications. So every 15 minutes the same pipeline snoozes open incidents that a live declared silence covers. The snooze lasts until the silence ends.
It also resolves incidents whose alert group has stopped firing. A silenced alert would never do this on its own.
The migration trips PostgresReplicationLag at 06:40.
Alertmanager matches it to the silence and sends nothing.
The assistant's alert triage still sees the alert, with the silence attached as context. The assistant annotates silenced alerts and does not drop them. If something outside the planned change breaks, the triage knows which alerts were expected.
Two hours before the end, and again thirty minutes before, a check runs. If alerts that
the silence covers are still firing, a Slack message says so.
At that point you can extend the silence on purpose (edit
ends_at in a new MR) or accept the page.
At ends_at the silence expires. Nobody has to remember it or clean it up.
A read-only audit checks every live silence of every tenant against the loaded rule set. Rule-based checks flag a silence with no end, or with an end more than 30 days away. They also flag a silence that matches no loaded rule, and an empty or broken matcher. A permanent grant whose review date has passed is a finding too.
A second report lists the other ways to mute an alert. These are inhibit rules, mute intervals and severity demotions. The report also shows their expiry debt (mutes that should have ended). If the assistant cannot read a source, the answer is "unknown", never "clean".
Planned: when the assistant proposes a risky upgrade hop, it also drafts the matching silence file as a companion MR. The silence fits the window the hop needs. We also plan recurring maintenance windows, such as a weekly patch slot, as a route-level mute interval.
Today your team writes the silence file. The rest of this page runs in production.
+tenant: acme # must match the directory +doc_id: orders-db-pg16-2026-09-26 # must match the file name +generation: 1 # bump only to re-issue +matchers: + - name: namespace + value: orders-db + - name: alertname + value: Postgres(ReplicationLag|Down|TooManyConnections) + isRegex: true +starts_at: 2026-09-26T06:00:00Z +ends_at: 2026-09-26T10:00:00Z # ≤ 30 days after start +comment: "orders-db Postgres 15 → 16 migration, primary failover expected" +link: https://git.example.com/acme/platform/-/merge_requests/318
$ python3 ci/silences_lint.py --self-check silences/ ✓ self-check: validator matches the vendored checksum ✓ acme/orders-db-pg16-2026-09-26.yaml schema · keys · types ✓ tenant "acme" matches directory ✓ window 4h (cap 30d) ✓ 2 matchers · regex compiles note: regex checked with Python re; Alertmanager uses RE2 0 failing · 0 skipped
tenant acme live sentinels: 3 declared: 4 CREATE orders-db-pg16-2026-09-26@g1 06:00Z → 10:00Z skip batch-reindex-2026-09-12@g1 past window, never re-created ok search-rebuild-2026-09-24@g2 content matches comment: "gitops-silences:orders-db-pg16-2026-09-26@g1: orders-db …" applied 1 · 0 errors
SNOOZE incident PostgresDown orders-db until 10:00Z covered by orders-db-pg16…@g1 RESOLVE incident CronJobFailed batch alert group no longer firing in any tenant note added to both incidents with the silence id
orders-db is inside a declared silence for the Postgres 16 migration.orders-db-pg16-2026-09-26 ends in 2h. 1 covered alert still firing: PostgresReplicationLag.Today your team opens this MR. Drafting it alongside an upgrade is on the roadmap. Everything after "merge" runs in production now. This is an illustrative example.
silence/orders-db-pg16 into main · 1 file · +13 · pipeline passedThe orders-db cluster moves from Postgres 15 to 16 on Saturday. The operator performs a switchover. PostgresDown and PostgresReplicationLag will fire for a few minutes each. This MR silences that alert family for orders-db only, for four hours.
Any other namespace. Any other Postgres alert name. The application's own SLO alerts. If orders-api starts to fail at checkout, the pager still wakes you.
+tenant: acme +doc_id: orders-db-pg16-2026-09-26 +generation: 1 +matchers: + - name: namespace + value: orders-db + - name: alertname + value: Postgres(ReplicationLag|Down|TooManyConnections) + isRegex: true +starts_at: 2026-09-26T06:00:00Z +ends_at: 2026-09-26T10:00:00Z +comment: "orders-db Postgres 15 → 16 migration, primary failover expected" +link: https://git.example.com/acme/platform/-/merge_requests/318
CREATE orders-db-pg16-2026-09-26@g1 and applies it.ends_at in a new MR. The reconcile job picks it up.The silence has an end (4h) and matches 3 loaded rules in tenant acme. The next audit has no findings.
A silence stops people from hearing about a problem. It gets stricter handling than any other config.