Home / Stories / Vault audit logs
Story · Secrets & access In development

A service keeps getting "permission denied". Is it a missing grant, or someone probing?

Vault writes every request to its audit log, including the requests it refuses. Most teams look at that log only after an incident. The assistant reads it every day. It groups the denials and decides which ones are a policy gap and which look like probing. It proposes a policy change only for the first kind.

Starts fromThe audit log of Vault. Your existing log collector sends it to Loki (a log store).
Looks atDenied and successful requests for each entity. The policies bound to the entity. Your Vault policy repository in git.
You getOne rolling digest issue for everything suspicious. A small draft merge request (MR) for a clear, low-risk policy gap.
01 · The log you already have

Vault's audit device, tailed into Loki.

A stdout audit device makes Vault print one JSON record per request and one per response. Your log collector already sends pod logs to Loki. So the audit trail lands there with no extra agent.

The assistant runs a read-only LogQL query (a Loki query) against that stream. It holds no Vault token and cannot change Vault.

02 · Find the refusals

Parse every record. Keep the denials.

The assistant parses each audit record as JSON. A denial is a response that carries permission denied. The record also shows who asked (the entity and its bound policies), what they asked for (path and operation) and when.

The assistant skips records that do not parse, and counts them. A failed read never turns into "no denials".

03 · Group them

One row per (entity, path), not one alert per line.

246 denials in about two and a half days is usually one misconfigured client, not 246 problems. The assistant clusters denials by entity and path. Each cluster shows a count, the first and last time seen, and how many distinct path prefixes Vault refused the entity on.

A single occurrence is noise. A cluster must repeat before the assistant considers it.

04 · Widen, probe or ambiguous

A fixed rule decides, not a model.

The classifier is rule-based code. No AI model (LLM) is involved:

  • Probe: the path is sensitive (sys/, auth/, the audit device, key-management paths). An unknown entity also counts as a probe, and so does a refusal on three or more different prefixes.
  • Widen: a known entity that Vault refuses again and again on one ordinary path prefix, with plain operations.
  • Ambiguous: everything in between.
05 · Probes and grey areas stay issues

One digest issue, refreshed every run.

The assistant never turns probe and ambiguous clusters into a policy change. They roll into a single audit-denial digest issue for each Vault repository. A cause table in the issue is recomputed on every run. So you get one place to look, and not a new issue for each entity.

Each row says how the denials arrived. A steady rhythm suggests a misconfigured client. A burst suggests something else. If there are too few timestamps to support that label, the assistant withholds it and states the shortfall.

06 · A clear gap becomes a draft MR In development

Append one stanza to one policy. Nothing else.

Take a widen cluster with exactly one candidate policy and one operation. The assistant renders the change against the policy file in git. It adds the missing capability to an existing path block, or appends a new block for the denied path. The rest of the file stays byte for byte as it was.

This path is built and switched on. As of October 2026, it has not fired on a real production denial. That is why it is marked in development.

07 · What it will never widen

A safety layer that fails closed.

Every finding passes one guard before the assistant sends it. A widen can never target an escalation surface (sys/, auth/, identity/, CA and key management). It can never grant sudo, touch an audit device or edit the protected root policy.

If the renderer is unsure about anything, the finding stays an issue.

08 · Did the grant land?

The next run checks for successes, not just silence.

For each entity, the assistant also looks for its successful responses in the same window. You can see "the denials stopped and the service now reads the path" in the issue itself.

That lookup runs only when the entity is a real, unique Vault entity id. A shared login name or IP address would mix up clients. In that case the assistant skips the lookup, and the issue says so.

Loki · {namespace="vault"} · audit streamlive tail
09:41:02 request  entity=8f2c… orders-api  read   secret/data/shop/orders-api/db
09:41:02 response entity=8f2c… orders-api  read   secret/data/shop/orders-api/db     200
09:41:07 request  entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key
09:41:07 response entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key  permission denied
09:43:15 response entity=1a90… ci-deployer update secret/data/platform/ci/token    200
09:51:07 response entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key  permission denied
09:52:40 response entity=—      approle     list   sys/policies/acl                  permission denied
10:01:07 response entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key  permission denied
Vaultstdout audit device
Log collectorpod logs
Lokiaudit stream
Assistantread-only LogQL
one audit record, parsed
{
  "type": "response",
  "time": "2026-09-18T10:01:07Z",
  "auth": {
    "entity_id": "8f2c…",
    "display_name": "kubernetes-shop-orders-api",
    "policies": ["default", "shop-orders-api"]
  },
  "request": { "operation": "read", "path": "secret/data/shop/payments/psp-key" },
  "error": "1 error occurred:\n\t* permission denied\n\n"
}
  • entity real entity id · bound policy shop-orders-api
  • operation read → capability "read"
  • timestamp from the record itself, sanity-checked
denial clusters · last 3 days
entitypathcountprefixes
orders-apisecret/data/shop/payments/psp-key2461
approle (no entity)sys/policies/acl311
report-jobsecret/data/* (5 prefixes)585
billing-workersecret/data/shop/billing/…122
ci-deployersecret/data/platform/tmp1below threshold
348 denials → 4 clusters worth a look · 1 single occurrence ignored
classify_cluster · deterministic
  • orders-api · psp-key known entity · 1 prefix · read → WIDEN
  • approle · sys/policies/acl sensitive path · no entity → PROBE
  • report-job · 5 prefixes breadth ≥ 3 → PROBE
  • billing-worker · 2 prefixes neither → AMBIGUOUS
widen → draft MR candidateprobe → issueambiguous → issue
Issue · Vault audit-denial digest · 3 causesupdated this run
category:securitysource:vaultrisk:high
causeclasstimeline
sys/policies/acl · approleprobe31 over 4h, single burst
report-job · 5 prefixesprobe58 over 2d, ~1 per 50m (steady)
billing-worker · 2 prefixesambiguousrate withheld (3 of 12 timestamped)
policies/shop-orders-api.hcl+4
path "secret/data/shop/orders-api/*" {
  capabilities = ["read"]
}

path "secret/metadata/shop/orders-api/*" {
  capabilities = ["list"]
}
+
+path "secret/data/shop/payments/psp-key" {
+  capabilities = ["read"]
+}
category:securitysource:vaultrisk:medium
safety.passes_all_guards · fail-closed
  • widen sys/policies/acl escalation surface · refused
  • widen pki/root/sign-intermediate CA management · refused
  • add sudo never renderable
  • touch sys/audit/* audit device is untouchable
  • edit the root policy protected · operator-only
  • read on secret/data/shop/payments/psp-key ordinary path · one policy · allowed
anything the guard cannot prove safe stays an issue
orders-api · psp-key · denials vs successes
policy merged denied succeeded
Interleaved successes: 212 since 14:05 · last denial 14:01 · the grant landed
Illustrative. Absence is stated too: "Interleaved successes: NONE found".

The merge request you get.

This MR is illustrative, and every number on this page comes from one example run. The widen path is built and enabled. As of October 2026, it has not fired on a real production denial. So this is the shape of an MR, not a copy of one the assistant sent.

Draft

Vault: grant shop-orders-api read on secret/data/shop/payments/psp-key

from gitops-ai/vault-widen-shop-orders-api-5b21 into main · 1 file · +4
category:securitysource:vaultrisk:mediumxrev::LGTM
OverviewEvidenceChangesBefore you merge
Summary

Vault refused the entity for orders-api (Kubernetes auth, namespace shop) read on secret/data/shop/payments/psp-key 246 times in about two and a half days. Its only non-default policy is shop-orders-api. That policy grants nothing under shop/payments/.

Classification: widen

The entity is known. There is one path prefix and one plain operation. There is no wildcard, and the path is not an escalation surface. This is the pattern of a missing grant, not of probing.

Is this access legitimate?

The assistant cannot know if orders-api should read the payment provider key. That is a product decision. Confirm it before you merge. Or close this MR and fix the client instead.

Timeline

246 denials over 2d13h, ~1 per 15m (steady). First seen 2026-09-16 21:04Z, last seen 2026-09-19 10:01Z. A steady rhythm like this usually means that a client retries on a schedule.

Interleaved successes

1,380 successful responses for the same entity in the window, all on secret/data/shop/orders-api/*. None are on the denied path.

Source query
{namespace="vault"} |= "permission denied" | json
  | auth_entity_id="8f2c…" | request_path="secret/data/shop/payments/psp-key"
 path "secret/metadata/shop/orders-api/*" {
   capabilities = ["list"]
 }
+
+path "secret/data/shop/payments/psp-key" {
+  capabilities = ["read"]
+}
Checked before this MR was opened
  • The target path is not under sys/, auth/, identity/, an audit device, or CA / key management.
  • The capability set is read only. It has no sudo and no new write verb.
  • The policy is not the protected root policy. The rest of the file is unchanged.
  • Exactly one candidate policy is bound to the entity.
After merge, expect
  • The CI of your Vault repository applies the policy on the default branch.
  • Denials for this cluster stop within the retry interval of the client.
  • The next run reports successes on the path. This closes the loop in the issue history.
R
independent reviewer second opinion, no pipeline context

The diff is a single read grant on one exact path, with no wildcard. The grant scope matches the evidence. Whether the service should hold this key is a product decision. LGTM on shape.

What keeps it honest.

A mistake in Vault can do a lot of damage. Everything here is rule-based code, with no AI model in the loop.

No Vault credentialsThe assistant reads the audit trail from Loki and the policies from git. It cannot read secrets or change Vault. Your own CI applies merged policy.
Escalation surfaces are off limitsThe assistant refuses a widen on sys/, auth/, identity/, audit devices or CA and key management, whatever the evidence says.
Probes are never "fixed"A denial that looks like probing becomes an item to investigate. It never becomes a policy change that would let the probe succeed.
No invented factsThe assistant withholds rates and "steady" labels when it has few samples. It skips success lookups if it cannot identify the requester uniquely, and the issue says why.
Fails open for the run, closed for the findingA Loki or parse error never blocks the daily run, and never produces a finding. If the guard is unsure, it suppresses the finding.
One digest, not a ticket stormIssue-only causes share one rolling issue for each repository. The assistant marks partial Loki evidence, so fewer rows cannot imply that the problem went away.

Where this stands.

Live
  • Audit-log denial triage from Loki, clustered by entity and path
  • Rule-based widen / probe / ambiguous classifier
  • Rolling audit-denial digest issue with timeline and success telemetry
  • Fail-closed Vault safety layer
In development
  • Draft MRs for unambiguous widen findings (enabled, awaiting its first real production witness)
Roadmap
  • A separate paging alert for probe patterns on sensitive paths
  • Comparing declared policy with what is live in Vault (needs a read-only Vault token)
← Previous storyReviewing your team's merge requests