Home / Stories / Orphans & relics
Story · Hygiene & cost In development

The things nobody owns any more, found and written up with evidence.

Every long-lived setup collects leftovers. One example is a directory of YAML that no Kustomization (a Flux object that says what to deploy) points at. Another is a ServiceMonitor (a rule that tells Prometheus what to scrape) from a chart that you uninstalled long ago. There is also a 50 GiB volume that nothing mounts, and a database that you hibernated "for now". The assistant looks at git and the cluster side by side. It proves what is unused and gives you an issue. It never deletes anything itself.

Starts fromYour GitOps repositories and the live objects in each cluster.
Looks atFlux Kustomizations and their inventories, HelmReleases, monitors and policies, PVCs (volume claims), pods and workload templates, and CloudNativePG clusters.
You getIssues that name each leftover, why the assistant thinks it is unused, and what it checked. Cleanup stays your decision.
01 · Two directions

Git without a cluster, and cluster without a git.

An orphan is a break in the chain between what is written down and what runs. The assistant checks the chain from both ends. One end is YAML in git that nothing deploys. The other end is live objects that nothing in git declares.

Each direction is a separate, rule-based scan. No AI model (LLM) decides whether something is an orphan.

02 · YAML nothing deploys Live

A directory that no Kustomization reaches.

The daily wiring audit computes the closure of every Flux Kustomization over the repository tree. The closure is the set of files that the Kustomization reaches. Flux never applies files outside that closure, however correct they look.

Sometimes the fix is a one-line stitch into a parent kustomization.yaml. Then you get a draft merge request (MR). A whole unreferenced directory becomes an issue, because only you know whether it is future work or a forgotten copy.

03 · Objects nothing declares Live

Uninstall residue: still running, owned by nobody.

Every four hours the assistant checks the cluster side the other way. It looks at ServiceMonitors, PodMonitors, PrometheusRules and NetworkPolicies. It reports in an issue any that no scanned repository declares and no live HelmRelease manages.

A leftover NetworkPolicy still shapes traffic. A stale PrometheusRule still pages. That is why these are worth finding.

04 · What prune: false keeps alive Live

Removed from git, never removed from the cluster.

Flux only removes deleted objects when a Kustomization has prune: true. With prune off, deleting a file in git leaves the object running. The assistant compares each Kustomization's own applied inventory with the live objects that carry its labels.

An object must show up as orphaned on three consecutive runs. Only then does the assistant report it, in one digest per cluster. The assistant never proposes flipping prune. That is your call.

05 · Idle volumes In development

A volume nothing has mounted for months.

The relic census lists PersistentVolumeClaims older than a threshold (90 days by default). It checks them against every live pod and every workload template. This includes CronJobs and Deployments scaled to zero, so a paused consumer still counts as a consumer.

The assistant matches StatefulSet claims to their ordinals. It leaves alone a volume reserved by a StatefulSet parked at zero replicas. The relic census is built and tested, but it is opt-in and not yet switched on in production.

06 · Hibernated databases In development

A database put to sleep, that nothing will ever wake.

A CloudNativePG cluster with hibernation on keeps its storage and costs money. The census looks for any reference to its -rw, -r or -ro services in pod and workload env, args, commands and volumes.

If nothing references the cluster, the assistant reports it as a relic. It does not read Secret and ConfigMap contents, and the issue says so.

07 · Absence must be proven

"I could not see it" is never "it is not there".

An orphan claim is a claim of absence. So every read behind it must be complete. Sometimes a list of HelmReleases, pods or workloads is denied, cut short or malformed. Then the scan withholds the verdict for that cluster and logs why. It does not report everything as unused.

You can put a skip annotation on anything that you want left alone. The scan will not mention it again.

08 · Where it is going Roadmap

Owners who left, siblings that drifted, and a price on each.

Planned: resources whose declared owner or team no longer exists. Also planned: environments that drifted apart from their siblings. One example is a file that was identical in staging and production until someone changed only one copy.

Planned: an estimated monthly cost next to each relic, so you can sort the list by what cleanup is worth. Until then, the issue shows the size and storage class, not a price.

gitwhat is written down
Fluxclosure + inventory
clusterwhat is running
four orphan shapes
  • in git, never applied wiring audit
  • in cluster, declared nowhere cluster-side orphan scan
  • deleted from git, still live prune posture digest
  • declared and live, but unused relic census
wiring audit · Kustomization closure over the repo tree
clusters/production/apps.yaml  → path: ./apps/production
apps/production/kustomization.yaml
  ✓ shop/orders-api/
  ✓ shop/checkout-web/
  ✓ payments/
✗ apps/production/shop/reports-worker/   # 3 files, in no resources: list
✗ apps/production/legacy-search/         # 7 files, whole directory unreferenced
reports-worker → draft MR (one-line stitch) · legacy-search → issue
cluster-side orphan scan · cluster prod-eu
  • ServiceMonitor shop/orders-api declared in git
  • PrometheusRule monitoring/kube-prometheus-stack managed by HelmRelease
  • PrometheusRule monitoring/elasticsearch-alerts chart uninstalled · declared nowhere
  • NetworkPolicy shop/allow-legacy-search declared nowhere
2 objects neither in git nor owned by a live release → issue
Kustomization flux-system/apps · prune: false
status.inventory # what Flux applied last reconcile
  shop_orders-api_apps_Deployment
  shop_orders-api__Service
  shop_checkout-web_apps_Deployment
live, labelled kustomize.toolkit.fluxcd.io/name=apps
  shop/orders-api        Deployment  in inventory
  shop/checkout-web      Deployment  in inventory
  shop/promo-banner      Ingress     not in inventory
  shop/promo-banner      Service     not in inventory
seen orphaned on run 1 · 2 · 3 → included in this cluster's digest
relic census · PersistentVolumeClaims older than 90 days
  • shop/data-orders-db-0 mounted by pod orders-db-0
  • shop/exports referenced by CronJob nightly-export (suspended)
  • search/data-opensearch-2 StatefulSet at 0 replicas · reserved
  • platform/data-auth-postgresql-0 1,280 days · 50Gi · no mount anywhere
  • shop/import-scratch 412 days · 200Gi · no mount anywhere
2 unattached volumes · 250Gi retained · opt-in lane
Cluster.postgresql.cnpg.io platform/auth-db
metadata:
  annotations:
    cnpg.io/hibernation: "on"
# consumer search in pods and workload templates
auth-db-rw   0 references
auth-db-r    0 references
auth-db-ro   0 references
# the live auth service connects to auth-db-v2-rw
hibernated, no consumer → relic (Secret/ConfigMap payloads not read)
read completeness · cluster staging-eu
  • list PersistentVolumeClaims complete
  • list pods complete
  • list workloads (CronJobs) 403 · incomplete
planned signalsroadmap
  • owner team no longer exists planned
  • staging and production twins diverged planned
  • estimated monthly cost per relic planned
# planned: sibling drift
apps/staging/shop/orders-api/configmap.yaml     sha 3f1c…
apps/production/shop/orders-api/configmap.yaml  sha 9a07…  # identical until last month
Illustrative. Not built yet.

The issue you get.

Orphans and relics arrive as issues, not merge requests. Deleting data is a decision, not a diff. This is an illustrative example of the relic digest. It has the same shape as the digest that the census writes.

Issue

Relic PersistentVolumeClaim digest (unattached-pvc-relic, 2 resources) on cluster prod-eu

opened by gitops-assistant · issue only, no branch, no diff
category:costsource:orphan_cluster_resourcerisk:low
OverviewEvidenceHow it decidedNext steps
Summary

Two PersistentVolumeClaims on prod-eu are older than 90 days. No live pod mounts them. No workload template references them, including suspended CronJobs and StatefulSets scaled to zero.

Is this retained data still needed? Or should you consider it for garbage collection after you verify consumers and recovery requirements? This is an issue-only advisory. The assistant proposes no destructive action.

PVCAgePhaseStorage classSize
platform/data-auth-postgresql-01,280 daysBoundssd50Gi
shop/import-scratch412 daysBoundstandard200Gi

Both rows: no PVC mount reference found in live pods or workload pod templates.

Checked before the assistant opened this issue
  • Every live pod's volumes: no reference to either claim.
  • Every Deployment, StatefulSet, DaemonSet, Job and CronJob template, including paused and scaled-to-zero ones.
  • The assistant matches StatefulSet claim templates by ordinal, so it does not report a volume reserved by a parked StatefulSet.
  • All listings are complete. A denied or partial read would have withheld this issue.
  • Neither volume carries the skip annotation.
  • Confirm nobody outside the cluster (a laptop kubectl cp, a migration plan) still expects the data.
  • Confirm a backup exists if the data may be needed later.
  • Remove the claim through git if it is declared there, so Flux does not recreate it.
  • Or keep it: add the skip annotation and this issue will not come back.
G
gitops-assistant deduplicated per volume set

The same set of volumes produces one issue, not one per run. A new unattached volume starts a new digest.

What keeps it honest.

An orphan report is a claim that something is unused. Every guard below exists because a wrong one costs data.

Issues, not deletionsThe assistant reports relics and cluster-side orphans as issues. Its cluster access is read-only. It never proposes a delete for retained data.
Absence needs a complete readA denied, truncated or malformed listing withholds the verdict for that cluster. A failed read never turns into "everything is orphaned".
Paused is not abandonedSuspended CronJobs, Deployments at zero and StatefulSets parked at zero replicas still count as consumers of their volumes.
Persistence before reportingThe assistant must see prune: false leftovers on three consecutive runs, so it does not report an object that is mid-rollout.
Never flips your policyThe prune digest names the leftovers. It never proposes turning prune on. That stays an operator decision.
Opt out per objectA skip annotation on the resource silences it for good. Use it for relics that you keep on purpose.

Where this stands.

Live
  • Wiring audit: YAML no Kustomization reaches (stitch MRs, directory issues)
  • Cluster-side orphans: monitors, rules and NetworkPolicies declared nowhere
  • prune: false leftover digest from Flux inventories
  • Orphaned PodDisruptionBudgets in cluster-state checks
In development
  • Relic census: unattached PVCs past an age threshold (built, opt-in, not yet on in production)
  • Relic census: hibernated CloudNativePG clusters with no consumer (same)
Roadmap
  • Resources whose owner or team has left
  • Drift between sibling environments
  • Estimated monthly cost on each relic
← Previous storyNaming conventions