Home / Stories / Ansible CI/CD
Story · Beyond Kubernetes Roadmap

Your Ansible CI is green. Is your fleet?

A well-run Ansible repository already lints every change. It also shows a --check --diff plan (a dry run) on each merge request (MR). Nobody automates the layer above. That layer would notice that the hourly apply has failed since Tuesday. It would also notice a green run that did nothing, or an MR that shows another MR's plan. The assistant is moving into that layer. Some parts are built, but the lanes that open draft MRs are on the roadmap.

Starts fromAn Ansible repository (roles, inventories, playbooks) whose CI runs the playbooks. The CI runs a check on merge requests and an apply on a schedule.
Looks atCI run history and job logs, the workflow files, role defaults and version pins, the repositories those playbooks depend on.
You getRun-health alerts on the scheduled apply, draft MRs for CI hygiene and role pin upgrades, and failures traced to their cause.
01 · A different kind of repo

No HelmReleases, no Kustomizations. Just roles.

An Ansible repository looks nothing like a Flux repository (the git repository that a Kubernetes cluster follows). The Kubernetes-shaped agents parse nothing in it. So the failure mode is not a wrong proposal. It is silent zero coverage: the assistant scans the repository, finds nothing and says nothing.

Treating Ansible repositories as first-class targets starts with admitting this. The next step is to build lanes that understand them.

02 · What CI already does

Lint and plan on every MR. Apply every hour.

In a typical setup, a merge request runs syntax-check, ansible-lint and a --check --diff plan. The main branch applies on push and on a schedule. The scheduled apply is the reconciler of this repository, the Ansible equivalent of Flux.

The MR gate is good at what it checks. But by its structure, it cannot report that its own reconciler has died.

03 · Watch the reconciler In development

OK, stale, red, or unknown. Never "probably fine".

A rule-based run-health check reads the scheduled runs and gives one of four verdicts. Stale: no scheduled run in two cadences. Red: the last K scheduled applies all failed. Unknown: no token, an API error or incomplete pages. Unknown never reads as OK.

We tune the thresholds by replaying the full run history, not by waiting a week. The check only reports. Opening an issue from its verdict is on the roadmap.

04 · Read the run log In development

A PLAY RECAP is structured data.

A parser turns an ansible-playbook job log into facts: plays, tasks, per-host results and the recap. In --check mode, changed>0 is a genuine drift signal. It means the live hosts differ from the repository.

The parser is honest about its limit. A run log cannot prove that a named resource exists, so the parser answers "can't tell", never "absent". The parser is built. No production lane uses it yet.

05 · CI hygiene Roadmap

The plan on this MR belongs to a different MR.

The plan step writes to a fixed /tmp/plan.txt on a self-hosted runner whose /tmp survives between jobs. The publish step runs if: always(). When this MR's plan never ran, reviewers see yesterday's plan from someone else's change.

That misleads reviewers and discloses diffs across merge requests. A rule can detect this from the workflow file alone.

06 · CI hygiene Roadmap

Green every hour, doing nothing every hour.

A guard step finds that a required secret is missing. It sets ready=false and lets the job finish green, "rather than failing hourly". Every scheduled apply becomes a successful no-op. A run-health check that only reads the status says OK.

The planned rule flags the pattern. The fix is to fail loudly on scheduled runs, or to read step outcomes and demote a skipped run to unknown.

07 · Role pins Roadmap

Versions live in role defaults, not in requirements.yml.

In practice the pins that matter sit in roles/*/defaults/main.yml (*_version, *_ref, image digests) and in the CI image's build arguments. A rule-based lane will read them and look up upstream releases. Then it will propose the same evidence-based target that the Kubernetes upgrade lane uses.

08 · Failure triage Roadmap

When the playbook is the victim, not the cause.

The assistant will compare the log tail of a failed job with known signatures. Examples are a sealed Vault, a registry pull error and DNS. When the repository declares the dependency, the assistant will attribute the failure to the repository that owns it. The proposal then lands there. The cross-domain story follows one such failure all the way to a missing probe.

09 · The proposal

A small workflow fix, with the reason spelled out.

The fix changes three lines. Two move the plan to a per-run path, and one guards the publish step. The MR explains the mechanism and shows the evidence from the repository's own history. It also warns that the MR itself triggers the check job on the fleet. The full MR is below.

fleet-ansible · repository tree
.ci/workflows/ansible.yaml
inventories/prod/hosts.yml
inventories/prod/group_vars/all.yml
roles/node_exporter/defaults/main.yml
roles/inference/defaults/main.yml
roles/hardening/tasks/main.yml
site.yml
Dockerfile
  • HelmRelease / Kustomization lanes 0 files understood
  • naming, resources, availability nothing to parse
silent zero coverage is the default, not a misfire
MRsyntax-check
ansible-lintproduction profile
--check --diffplan in job summary
mergemain
scheduled applyevery hour
fleethosts converge
the MR gate never sees the scheduled apply fail
scheduled applies · last 36 hours · oldest first
OKsuccess in window
STALE> 2 cadences
REDK in a row
UNKNOWNnever OK
backtest over full history: the single blip at run 4 does not fire; the streak does
job log · ansible-playbook site.yml --check --diff
TASK [node_exporter : install binary] ****************
ok: [host-01]
changed: [host-02]
PLAY RECAP *******************************************
host-01 : ok=41 changed=0 unreachable=0 failed=0
host-02 : ok=41 changed=1 unreachable=0 failed=0
  • mode: check dry-run drift probe
  • host-02 · node_exporter : install binary drift: live differs from repo
  • does resource X exist? can't tell, never "absent"
.ci/workflows/ansible.yaml · check job
runs-on: self-hosted   # host mode: /tmp persists
steps:
  - id: plan
    run: ansible-playbook site.yml --check --diff | tee /tmp/plan.txt
  - name: Publish the plan
    if: always() && steps.guard.outputs.ready
    run: cat /tmp/plan.txt >> $STEP_SUMMARY
plan step failed · summary shows a plan stamped the previous day
.ci/workflows/ansible.yaml · apply job · guard
  - id: guard
    run: |
      if [ -z "$KNOWN_HOSTS" ]; then
        echo "runner busy, skipping this run"
        echo ready=false >> $OUTPUT
      fi
12 green runs · 0 tasks executed · status-only health says OK
roles/node_exporter/defaults/main.yml
node_exporter_version: 1.8.1
node_exporter_checksum: sha256:…
node_exporter_web_listen: ":9100"
1.8.1pinned
1.8.2patch
1.9.1target
1.10.0-rcskipped
upstream releases read · no breaking change applies · checksum updated with the version
Scheduled apply · REDsame failing step, 12 runs
Log-tail signaturevault … 503 … sealed
Declared dependencyCI script names the Vault
Owning repoproposal lands where the cause is
Draft · Publish the check-mode plan from a per-run path
-    run: ansible-playbook site.yml --check --diff | tee /tmp/plan.txt
+    run: ansible-playbook site.yml --check --diff | tee "$RUNNER_TEMP/plan.txt"
-    if: always() && steps.guard.outputs.ready
+    if: always() && steps.plan.outcome == 'success'
category:securitysource:ansible_cirisk:low
↓ the full merge request is below

The merge request you would get.

Illustrative and on the roadmap. The rule, its evidence and its fix come from a real finding in an Ansible repository we run. The assistant does not yet produce this MR automatically.

Draft

Publish the check-mode plan from a per-run path, only when the plan ran

from gitops-ai/ansible-ci-plan-per-run-path into main · 1 file · +3 −3
category:securitysource:ansible_cirisk:low
OverviewEvidenceValidationChanges
Summary

The check job writes its plan to the fixed path /tmp/plan.txt. The runner is self-hosted in host mode, so /tmp persists between jobs. The publish step runs under always() and does not check whether the plan step succeeded.

Result: when the plan step fails or is skipped, the MR's job summary shows the previous job's plan, which may belong to a different merge request.

Why it matters
  • Reviewers approve a change based on a diff that is not its own.
  • Plan output from one merge request is disclosed on another.
Fix

Write the plan under $RUNNER_TEMP, which is per job, and publish it only when the plan step succeeded.

CheckResult
Job runs on a persistent (host-mode) runnerruns-on: self-hosted
Producer writes a fixed shared pathtee /tmp/plan.txt
Publisher runs regardless of producer outcomeif: always() && …, no steps.plan.outcome
Observed in historyA failed check job published a plan stamped the previous day.
Checked before this MR was opened
  • The workflow still parses, and every step id it references exists.
  • No other step reads /tmp/plan.txt.
  • The limit of open assistant MRs for this repository (1, to protect a runner shared with the hourly apply) is not exceeded.
After merge, expect
  • A merge request whose plan step fails shows no plan, instead of a stale one.
  • A merge request whose plan succeeds shows exactly its own plan.
       - id: plan
-        run: ansible-playbook site.yml --check --diff | tee /tmp/plan.txt
+        run: ansible-playbook site.yml --check --diff | tee "$RUNNER_TEMP/plan.txt"
       - name: Publish the plan
-        if: always() && steps.guard.outputs.ready
+        if: always() && steps.plan.outcome == 'success'
-        run: cat /tmp/plan.txt >> $STEP_SUMMARY
+        run: cat "$RUNNER_TEMP/plan.txt" >> $STEP_SUMMARY

What keeps it honest.

The CI of an Ansible repository touches real hosts. The guards are about not making that worse.

Unknown is not OKA missing token, an API error or an incomplete page of runs gives UNKNOWN. A health check that cannot see is never reported as healthy.
Tuned on history, not on hopeWe choose run-health thresholds by replaying every past scheduled run. We count would-fire and false-fire cases for each threshold.
Explicit opt-in for CI side effectsAn MR from the assistant triggers a check job on the fleet. So each Ansible repository needs a dated acknowledgement in the configuration before the assistant writes to it. Open assistant MRs also have a cap.
Drafts only, verifiedProposals are draft MRs. Where the forge marks drafts by title prefix, the assistant reads the draft state back after it creates the MR. If the draft state did not stick, the assistant closes the MR and the proposal fails.
Negative controls per ruleEach hygiene rule ships with a workflow that does it right. Examples are a per-run path, outcome gating and failing on a missing secret. The rule must stay quiet on that workflow.
No plan hunks in fixturesWe strip diff hunks and mask hosts in test fixtures made from real job logs. A person reviews them before we commit them.

Where this stands.

Live
  • The general GitLab reviews of access, CI/CD and runners also cover Ansible repositories on GitLab, like every other repository. Nothing Ansible-specific is live yet.
  • Evidence-based upgrade targeting (the Kubernetes lane the role-pin lane will reuse)
In development
  • Scheduled-apply run-health (OK / STALE / RED / UNKNOWN), report-only
  • Run-log parser for ansible-playbook job logs, not yet wired into a lane
  • Read access to Ansible repositories on Forgejo
Roadmap
  • Run-health as an issue on the repository
  • CI hygiene rule pack: stale artefacts on persistent runners, guard-skips that hide a dead reconciler
  • Role and collection pin upgrades as draft MRs
  • Failure triage from CI job logs, attributed across repos
  • Inventory and group_vars understanding
← Previous storyDNS as code