Spot capacity (spare cloud machines at a low price) is cheap because the provider can take a node back at any moment. Some teams even treat that as a free chaos monkey. That only works if every workload can survive the loss of a node. This page shows how the assistant finds the workloads that cannot. It also shows what the assistant proposes, so the next preemption is a non-event.
The cluster-state scan records how many availability zones each cluster spans and whether it has spot nodes. It also records whether the cluster runs only on spot. It reads node labels and nothing else.
Here: three zones, four spot nodes. checkout-api runs
two replicas, but both landed on the same node.
payments-operator and orders-worker run a
single replica each, on that same node.
The provider reclaims node a-1 with a few seconds of
notice. Everything on it stops at once.
checkout-api had two replicas and still went fully
dark. Replicas that share a node share its fate. Until the scheduler
finds room and the new pods pass their readiness checks, nothing
serves checkout.
The same gaps are visible in git long before any node dies. The availability rules walk every workload daily and check it against the cluster it runs on:
This agent does not call an AI model at all. Each rule is a plain check with a fixed fix. So you can reproduce a finding and explain it line by line. The pack also costs nothing to run every day.
A rule that needs topology skips itself when the zone count is unknown, rather than guess. When the assistant cannot write a clean diff, the finding becomes an issue instead of a risky MR.
On a multi-zone cluster, a multi-replica workload without
topologySpreadConstraints (Kubernetes rules that spread pods apart) gets two constraints. One uses the node as key and one uses the zone. Each has maxSkew: 1 and the
workload's own selector labels.
Both are ScheduleAnyway, a best-effort spread that can never leave a pod Pending when capacity is short. You can tighten it
to DoNotSchedule if you have guaranteed room.
A multi-replica workload with no PodDisruptionBudget (PDB, a limit on how many pods can stop at once) gets one. The assistant adds it in a new file next to the workload and wires it into the directory's kustomization.
Be clear about what that buys: a PDB governs only voluntary evictions, such as node drains, pool upgrades and autoscaler scale-downs. A spot preemption does not ask permission. That is what the spread in step 05 is for. The two cover different failures, so you get both.
A controller-manager running alone gets a proposal for 3 replicas. Leader election means only one replica works at a time. The others take over when the leader's node dies. It ships as a "judgment call" draft. The assistant skips it for controllers whose standbys cannot pass readiness.
On a spot-only cluster, a terminationGracePeriodSeconds
above 30 is trimmed to 30. Spot never waits longer, and a long grace
also slows every voluntary drain. Trimming further, to 15 s when
every live pod is provably on spot, is
In development.
With the replicas spread, losing a-1 costs one
checkout-api replica, not the service. The operator's
standby in another zone takes the lease. The cluster reschedules the lost pods in the background.
orders-worker still has one replica, so it still
blips. Proposing extra replicas for ordinary single-replica apps,
with evidence that the app tolerates it, is
Roadmap: some singletons are singletons
on purpose.
if kind in (Deployment, StatefulSet, HelmRelease) and replicas ≥ 2 and not has_topology_spread(workload) and cluster.availability_zones ≥ 2: # unknown → skip propose(spread_by_host_and_zone(workload)) # no clean diff → issue
template: spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: checkout-api + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: checkout-api containers: - name: api
+apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: checkout-api-pdb + namespace: shop +spec: + minAvailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: checkout-api
resources: - deployment.yaml - service.yaml + - poddisruptionbudget_checkout-api.yaml
spec: - replicas: 1 + replicas: 3 template: spec: containers: - args: ["--leader-elect"] # standbys wait for the lease
spec: - terminationGracePeriodSeconds: 120 + terminationGracePeriodSeconds: 30
One rule, one small MR. This is the spread proposal for checkout-api. The PodDisruptionBudget arrives as its own MR, so you can review and revert each one separately. Illustrative, shaped like the real output.
gitops-ai/topology-spread-checkout-api-7b2e into main · 1 file · +14Deployment shop/checkout-api runs 2 replicas on a 3-zone cluster but declares no topologySpreadConstraints. Multiple replicas on the same node defeat the high-availability goal. One node failure takes all replicas down at once.
Two soft constraints on the pod template: spread by kubernetes.io/hostname and by topology.kubernetes.io/zone, maxSkew: 1, selecting on the workload's own labels.
A hard DoNotSchedule can leave a replica Pending when the cluster cannot satisfy the skew. This can happen, for example, while a spot node is being replaced. Tighten it only if you have guaranteed node and zone capacity.
A separate draft adds a PodDisruptionBudget (minAvailable: 1) for the same workload.
To opt out for this resource, add the skip annotation with the value availability.
| Signal | Value | Source |
|---|---|---|
| Availability zones | 3 | node zone labels, last cluster scan |
| Spot nodes present | yes | node spot / preemptible labels |
| Spot only | yes | no on-demand pool |
| Effective replicas | 2 | spec.replicas in git |
| Existing spread / PDB | none / none | workload and sibling files |
No AI model was used for this finding or this diff.
template:
spec:
+ topologySpreadConstraints:
+ - maxSkew: 1
+ topologyKey: kubernetes.io/hostname
+ whenUnsatisfiable: ScheduleAnyway
+ labelSelector:
+ matchLabels:
+ app.kubernetes.io/name: checkout-api
+ - maxSkew: 1
+ topologyKey: topology.kubernetes.io/zone
+ whenUnsatisfiable: ScheduleAnyway
+ labelSelector:
+ matchLabels:
+ app.kubernetes.io/name: checkout-api
containers:
- name: api
kubectl get pods -n shop -l app.kubernetes.io/name=checkout-api -o wide shows the two replicas on different nodes, in different zones.Revert this MR. The constraints are soft, so they never block scheduling in the meantime.
Selector labels match the Deployment's pod template. The constraints are soft. The cluster has three zones. LGTM.
A rule that fires on the wrong workload is worse than no rule. Most of the work is in the cases where it stays quiet.