Secrets hygiene — GitOps Assistant

Every Secret gets a source of truth, and it isn't git.

Someone created a webhook signing key with kubectl create secret during an incident. Another key sits base64-encoded in a YAML file in the repository. Both work, until someone rebuilds the namespace or rotates the key. The assistant turns each one into an ExternalSecret (an object that reads a secret from a store) backed by your Vault. It seeds Vault from the running value. It also checks every diff it writes for credentials that should not be there.

Starts fromA Secret in the cluster with no ExternalSecret behind it. Or a Secret manifest committed to git. Or a credential inside Helm values.

Looks atSecret types and key names (never values). Your ClusterSecretStores. Which chart or Job owns each Secret. The workloads that use each Secret.

You getA draft merge request (MR) that adds the ExternalSecret and a PushSecret to seed Vault. It removes the committed Secret. It also says what must be true before you merge.

01 · Which Secrets have no source?

Every Secret without an ExternalSecret behind it.

The assistant lists Secrets in application namespaces. For each one, it looks for an ExternalSecret that produces it. It reads names, types and key names only. Its cluster access is read-only, and it never sees a secret value.

Here the assistant finds payments/stripe-webhook, with two keys, created by hand. It also finds a committed smtp-relay Secret in the repository.

02 · Not every Secret should move

Some belong to a chart, some aren't secret at all.

The assistant leaves some Secrets alone. These are Secrets that a chart's own bootstrap Job writes, and Secrets named after a Helm release in the same namespace. If the assistant shipped them again, it would fight the chart.

TLS and CA material goes to review, because an empty key-value store path would overwrite a working certificate. If a committed Secret holds values that are plainly not credentials, the assistant proposes a ConfigMap instead.

03 · Write the ExternalSecret

Same name, same keys, backed by your store.

The ExternalSecret mirrors the Secret's keys, with one remoteRef per key. If the cluster has exactly one ClusterSecretStore, the assistant uses it. If there are several, your configured default wins. If you set no default, the MR asks you to choose.

For a committed Secret, the same MR deletes the static manifest. Git and the ExternalSecret then never both own it.

04 · Seed Vault on merge

No manual `vault kv put`.

An ExternalSecret that points at an empty path would replace a working Secret with an empty one. So the MR pairs it with a PushSecret. The PushSecret copies the live value into Vault once, on merge. It never overwrites or deletes the value afterwards.

The value goes from the cluster to Vault inside your cluster. It never passes through the assistant, the AI model (LLM) or the MR.

05 · Secrets hiding in Helm values

Credential-shaped values become references.

When the assistant migrates or adopts a HelmRelease, it scans the values for credential-shaped leaves. Examples are a password key or a token pasted into a startup script. It rewrites each one to a reference. One ExternalSecret then supplies them all through valuesFrom.

The assistant declines anything ambiguous and does not guess. The MR then carries a scaffold and a note instead.

06 · Scan every diff

Before any MR opens, not after.

The assistant checks every diff from every agent for literal-looking credentials. These include API keys, tokens, passwords and client secrets. A hit withholds the MR.

ExternalSecret edits get their own checks. The checks look for a renamed remoteRef key, a template that reverts to the old engine, a rewrite that matches everything, and a deletionPolicy that flips to Delete.

07 · Before a restart bites

A missing Secret only fails when the pod restarts.

The assistant holds a diff that makes a workload use a Secret that exists nowhere, in git or in the cluster. The next pod start would fail with CreateContainerConfigError.

When an ExternalSecret stops syncing, the issue names the workloads that use its Secret. You can then fix the stale credential before the next rollout picks it up.

08 · Keeping ExternalSecrets current In development

Operator upgrades that remove old API and template versions.

When the External Secrets Operator stops serving an old API version, every Helm release that still renders it fails at its next upgrade. The assistant reports these failures today, with one issue per release. Grouping them into one issue per cause is built, but not yet switched on.

Roadmap Before an operator upgrade, the assistant will sweep live ExternalSecrets for defaults that the new version removes (such as a stored template engine version). It will then propose to declare them.

apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: { name: stripe-webhook, namespace: payments } spec: refreshInterval: 1h secretStoreRef: { kind: ClusterSecretStore, name: vault } target: { name: stripe-webhook, creationPolicy: Owner } data: - secretKey: signing-secret remoteRef: { key: payments/stripe-webhook, property: signing-secret } - secretKey: api-key remoteRef: { key: payments/stripe-webhook, property: api-key }

kind: PushSecret spec: updatePolicy: IfNotExists # never overwrite what Vault holds deletionPolicy: None # removing this never deletes the Vault data selector: { secret: { name: stripe-webhook } }

values: db: host: orders-db - password: s3cr3t-from-2021 migrate: command: - - sh -c "curl -H 'Token: 9f2c…' http://…" + - sh -c "curl -H \"Token: ${MIGRATE_TOKEN}\" http://…" + env: [ { name: MIGRATE_TOKEN, valueFrom: { secretKeyRef: … } } ] + valuesFrom: + - kind: Secret + name: orders-api-externalized

✗ shop/orders-api no matches for kind "ExternalSecret" in version "external-secrets.io/v1alpha1" ✗ shop/cart-api no matches for kind "ExternalSecret" in version "external-secrets.io/v1alpha1" ✗ payments/billing-api no matches for kind "ExternalSecret" in version "external-secrets.io/v1alpha1" … 8 more, same message

The merge request you get.

This example is illustrative. It has the shape of the real conversion MR, with the same title format, PushSecret pairing and pre-merge conditions.

Draft

Convert Secret to ExternalSecret: payments/stripe-webhook

from gitops-ai/convert-secret-payments-stripe-webhook into main · 3 files · +34 −0

category:securitysource:flux_migrationrisk:mediumxrev::LGTM

OverviewBefore mergeAfter mergeChanges

Summary

Secret payments/stripe-webhook exists in the cluster with no ExternalSecret and no manifest in git. Nobody can recreate it if the namespace is rebuilt. To rotate it, you must edit the cluster by hand. This MR makes Vault its source of truth.

What is in this MR

An ExternalSecret with the same name and the same two keys. It uses ClusterSecretStore vault, the only store in this cluster.
A PushSecret that copies the current value into Vault once, on merge (updatePolicy: IfNotExists, deletionPolicy: None).
Both files wired into apps/payments/kustomization.yaml.

Could not verify

The assistant reads Secret metadata only. It cannot see the sync status of the PushSecret. Check it after merge (below).

The Vault role of the ClusterSecretStore can write payments/stripe-webhook. The PushSecret needs to write once.
No other process writes this Secret in the cluster. From now on, the ExternalSecret owns it.

After merge, expect

kubectl -n payments get pushsecret stripe-webhook: Synced.
kubectl -n payments get externalsecret stripe-webhook: SecretSynced, same two keys.
The Secret now carries an owner reference to the ExternalSecret. Its content stays the same, so pods do not restart.

Note on deletion

With creationPolicy: Owner, if you delete this ExternalSecret later, that also deletes the Secret. deletionPolicy: Retain does not prevent that.

+ apps/payments/externalsecret_stripe-webhook.yaml   (new, 17 lines)
+ apps/payments/pushsecret_stripe-webhook.yaml       (new, 15 lines)
  apps/payments/kustomization.yaml
+   - externalsecret_stripe-webhook.yaml
+   - pushsecret_stripe-webhook.yaml

independent reviewer second opinion, no pipeline context

Key names match the live Secret. The store name matches the only ClusterSecretStore in the cluster. No value appears in the diff. LGTM.

What keeps it honest.

Secret changes fail quietly and late. Most rules below exist because of a failure of exactly that kind.

Never reads a live valueThe assistant lists cluster Secrets by name, type and key names only. It scrubs secret-shaped content found in git before anything reaches the AI model.

No empty overwriteAn ExternalSecret over a path that may be empty would blank a working Secret. So the assistant pairs it with a PushSecret or a populate-first condition. Certificates go to review.

One ownerThe MR that adds an ExternalSecret also deletes the committed Secret. It deletes only the document for that Secret, not its neighbours in the file.

Charts keep their SecretsThe assistant never converts Secrets that a chart's bootstrap Job writes, even after the Job is gone.

Your opt-out winsA skip annotation on the live Secret or on its manifest stops the conversion of that Secret.

Says when it takes effectEvery Secret or ConfigMap change states how it reaches the pod (restart, checksum annotation, reloader or nothing). It also states what you need to do.

Where this stands.

Live

Secret → ExternalSecret conversion with Vault seeding through PushSecret
Committed Secrets replaced; non-credentials moved to ConfigMaps
The assistant moves credentials in Helm values out to references during migration and adoption
Credential scan and ExternalSecret checks on every diff. The assistant holds references to missing Secrets
The assistant links failed ExternalSecret syncs to the workloads that use them

In development

One issue per cause when many releases fail on an unserved ExternalSecret API version (built, switched off)

Roadmap

Pre-upgrade sweep for stored defaults a new operator version removes
Templated ExternalSecrets (per-key Vault entries assembled into a config file in git) instead of whole config blobs
Key-level consumer tracing on every Secret change