Backups fail quietly. The nightly run of Velero, a Kubernetes backup tool, reports
PartiallyFailed because one volume could not be snapshotted. The database backup has been failing since someone rotated a credential. Nobody looks until the day a restore is needed. The assistant checks every few hours and opens one issue per broken schedule, with the evidence and the likely cause.
Backup objects and CloudNativePG (CNPG, a database operator) Backup objects in the cluster.
The backup scan lists Velero Backup objects and
CloudNativePG Backup objects created in the last 48
hours (both configurable). It reads them straight from the cluster API, read-only.
Here: 14 Velero runs and 8 database backups in the window.
Velero runs in Failed or
PartiallyFailed count. A partial failure often means
Velero saved the Kubernetes objects but not a volume. The volume is exactly the part you need for a restore.
The assistant groups runs by their Schedule, so seven failed nights become one finding, not seven.
For each failing schedule the scan collects the error messages of the most recent runs and the Schedule's spec and status from the cluster. It also collects the Schedule and BackupStorageLocation manifests from your repository.
The error usually names the layer: storage credentials, bucket, snapshot permissions, a node agent.
The assistant groups failed CNPG backups by the Cluster that owns them. It attaches the Cluster's status and its spec.backup block
(object store, credentials reference, retention) as context.
An AI model (LLM) reads the evidence and writes a short diagnosis. It gives the most likely cause, what to check first and how to confirm the fix. If the analysis is unavailable, the assistant still opens the issue with the raw phase and error. It never drops the finding.
The assistant opens backup findings as issues, not draft merge requests (MRs). The issues carry the labels reliability and high risk. The fix is often outside git: a cloud permission, a bucket policy or a rotated key. A wrong storage change can make things worse.
That caution is earned. Once, removing a storage-location prefix to fix one error made the whole location invalid, because the bucket was shared. The full issue is below.
If the BackupStorageLocation is Unavailable, Velero attempts nothing, so there is no failed backup to find. We plan to check the storage location itself and the age of the newest successful backup per schedule. Then silence no longer reads as health.
We also plan to flag runs whose volume snapshot list is empty when the schedule includes volumes.
We plan draft MRs for schedule and retention changes, and for a missing BackupStorageLocation. The assistant will first check the actual layout of the bucket before it touches a prefix.
When an upgrade bumps a backup plugin, the assistant will also compare the cloud permissions that each version documents. Then a new required permission shows up in the upgrade MR, not in next month's failed restore.
phase: PartiallyFailed errors: 3 error: snapshot shop/data-orders-db-0: 403 permission denied (create snapshot not allowed for the backup identity) volumeSnapshotsAttempted: 3 completed: 0 # Schedule (cluster) spec.template: { includedNamespaces: [shop, payments], snapshotVolumes: true, ttl: 720h } status.lastBackup: 2026-09-23T04:00:12Z # git: infrastructure/velero/schedule_daily-full.yaml · backupstoragelocation_default.yaml
spec.backup: barmanObjectStore: destinationPath: s3://acme-backups/billing-pg s3Credentials: { accessKeyId: { name: billing-pg-s3, key: ACCESS_KEY_ID } } retentionPolicy: 30d status.lastFailedBackup: 2026-09-23T00:00:04Z
403. The backup identity can read disks but may not create snapshots.shop and check that volumeSnapshotsCompleted equals volumeSnapshotsAttempted.spec: schedule: "0 4 * * *" template: - ttl: 72h0m0s + ttl: 720h0m0s # match the 30-day retention the runbook promises snapshotVolumes: true
backup plugin v1.7 → v1.9, documented cloud permissions + one permission added in v1.9, missing from the current role
Illustrative, shaped like the real backup issue. It has the same title format, the same labels and the same evidence sections. Backup findings are issues today, not merge requests.
prod · namespace veleroThe Velero schedule daily-full has produced PartiallyFailed backups on the last 2 runs in the 48-hour window. Kubernetes objects are saved. No volume snapshot completed (0 of 3), so persistent data in shop and payments is not restorable from these backups.
Every snapshot error is a 403 on snapshot creation. The backup identity can list disks but lacks permission to snapshot them. The Schedule and BackupStorageLocation in git are unchanged in this period, which points at the cloud role rather than the manifests.
Velero schedule daily-full: 2 backup runs failed. Restores from these runs would bring back objects without their data.
| Run | Phase | Error |
|---|---|---|
| daily-full-20260923040012 | PartiallyFailed | snapshot shop/data-orders-db-0: 403 permission denied |
| daily-full-20260922040009 | PartiallyFailed | snapshot shop/data-orders-db-0: 403 permission denied |
infrastructure/velero/schedule_daily-full.yamlinfrastructure/velero/backupstoragelocation_default.yamlvelero backup create verify-snapshots --include-namespaces shop.volumeSnapshotsCompleted equals volumeSnapshotsAttempted in velero backup describe.A backup alert that cries wolf gets muted, and a muted backup alert is how data gets lost.
PartiallyFailed as a failure, because the missing part is usually the data.