Monitoring & Grafana dashboards

From "nobody scrapes this" to a dashboard that actually has data.

A team ships a new service. It exposes Prometheus metrics (numbers about how the service runs). Nobody wrote the ServiceMonitor, so nothing collects the metrics, and there is no dashboard. This story shows what the assistant does about it, step by step. At the end, you get two draft merge requests (MRs).

Starts fromA workload with a metrics port and no ServiceMonitor or PodMonitor.

Looks atLive Services and monitors, the metrics endpoint itself, your metrics backend (Prometheus / Mimir) and grafana.com.

You getA ServiceMonitor MR. Then a dashboard ConfigMap MR where the assistant has shown that every panel returns data.

01 · Notice the gap

A service with a metrics port, and nothing scraping it.

Every day, the monitoring scan lists the workloads in the cluster. It matches them to their Services and to every existing ServiceMonitor and PodMonitor. These are Kubernetes objects that tell Prometheus what to scrape. A ServiceMonitor selects Services, not pods. So the scan follows the real chain: monitor → Service → workload, and it respects namespace selectors.

orders-api has a port called metrics on 9102. No monitor reaches it. The scan skips workloads that someone scaled to zero on purpose, so a parked relic never gets a monitor.

02 · Check the endpoint really serves

It asks the endpoint before it writes a line of YAML.

A port name is a hint, not proof. Before it proposes anything, the assistant fetches /metrics with a read-only probe. It checks that the endpoint answers in Prometheus exposition format.

The same response shows which metric families exist. Later, the assistant builds the dashboard from these families.

03 · Propose the ServiceMonitor

Labels copied from the live Service, not guessed.

The generated ServiceMonitor selects the Service by its actual labels and names the actual port. It also carries the release label. Your Prometheus operator or Alloy instance uses that label to find monitors.

Sometimes the Helm chart (a packaged application) already ships a monitor behind a values flag. Then you get the values flip instead of a hand-written manifest. The chart stays the owner.

04 · Merged. Data starts flowing.

Your collector does the scraping. The assistant reads the result.

After you merge, Flux applies the monitor. Your collector (Prometheus, or Grafana Alloy) starts to scrape. The series arrive in your metrics backend.

The assistant never writes to that pipeline. It only reads from the backend, one tenant at a time, when it builds and checks the dashboard.

05 · Find or write the dashboard

Community dashboards first. Generated only when none fit.

For well-known exporters (node, Redis, PostgreSQL, Traefik…), the assistant prefers the maintained grafana.com dashboard. It adapts the dashboard to your datasource and namespace variables.

orders-api is your own code, so there is nothing to import. The assistant drafts a dashboard from the metric families it saw: request rate, latency histogram (how long requests take), errors, business counters and runtime.

06 · Prove every panel

Each query runs against your data before you see it.

The assistant runs every panel expression against the metrics backend. It drops panels that return nothing and counts them. A grounded panel is one that is proven to return data. If no panel is grounded, the assistant does not ship the dashboard quietly. It holds the dashboard for review and names the reason.

The assistant catches rate() over a gauge and escalates it.
Every $variable needs a declaration.
Datasource and namespace are variables. They are never hard-coded.

07 · Ship it the GitOps way

A ConfigMap in git. Grafana's sidecar does the rest.

The assistant wraps the dashboard in a ConfigMap with the grafana_dashboard label. It places the ConfigMap in the tenant's folder. It also records on the ConfigMap how many queries were grounded, and when.

You merge it, and Flux applies it. The Grafana sidecar picks it up. There is no click-ops (manual work in the UI). No dashboard exists only in one Grafana's database.

08 · The result

A dashboard that maps one-to-one onto what the service exposes.

Every panel shows one of the metric families the probe saw in step 2. Nothing is invented. Nothing is empty.

The assistant audits committed dashboards for known defects and repairs them by MR. A scheduled re-check of these dashboards is in development. It will also find a panel that goes dark after a service change.

$ GET http://orders-api.shop:9102/metrics 200 OK content-type: text/plain; version=0.0.4 # HELP http_requests_total Requests by route and code. # TYPE http_requests_total counter http_requests_total{route="/orders",code="200"} 184223 # TYPE http_request_duration_seconds histogram http_request_duration_seconds_bucket{le="0.25"} 171004 # TYPE orders_created_total counter # TYPE outbox_queue_depth gauge # TYPE go_goroutines gauge → 5 metric families recorded for later

+apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: orders-api + namespace: shop + labels: + release: kube-prometheus-stack # what your collector selects on +spec: + selector: + matchLabels: + app.kubernetes.io/name: orders-api # from the live Service + endpoints: + - port: metrics # named port, verified + interval: 30s

+apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-dashboard-orders-api + namespace: monitoring + labels: + grafana_dashboard: "1" # sidecar picks it up + annotations: + grafana_folder: Shop + gitops-assistant/grounded-queries: "6/7" + gitops-assistant/grounded-at: "2026-09-18T04:12:09Z" +data: + orders-api.json: | + { "title": "orders-api", "templating": { … }, "panels": [ … ] }

The merge request you get.

You get two small MRs, in order. First comes the ServiceMonitor. The dashboard follows once there is data to ground it on. This is the second MR.

Draft

Add Grafana dashboard for shop/orders-api

from gitops-ai/dashboard-orders-api-3f1c into main · 1 file · +214

category:dashboardsource:dashboardrisk:lowxrev::LGTM

OverviewChangesValidation

Summary

Prometheus has scraped orders-api in namespace shop since the ServiceMonitor in !212 merged, but it has no dashboard. No grafana.com dashboard matches an in-house service. So the assistant generated this one from the five metric families the service exposes.

Panels

Panel	Source family	Grounded
Requests / s, by route	`http_requests_total`	✓ 12 series
Latency p50 / p95 / p99	`http_request_duration_seconds`	✓ 4 series
5xx ratio	`http_requests_total`	✓ 1 series
Orders created / h	`orders_created_total`	✓ 1 series
Outbox queue depth	`outbox_queue_depth` (gauge)	✓ 1 series
Goroutines	`go_goroutines`	✓ 1 series

Dropped: "Refunds / h" (orders_refunded_total returned no series in the last 7 days).

+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-orders-api
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+  annotations:
+    grafana_folder: Shop
+    gitops-assistant/grounded-queries: "6/7"
+    gitops-assistant/grounded-at: "2026-09-18T04:12:09Z"
+    gitops-assistant/grounded-panels-dropped: "1"
+data:
+  orders-api.json: |
+    {
+      "title": "orders-api",
+      "templating": { "list": [
+        { "name": "datasource", "type": "datasource", "query": "prometheus",
+          "current": { "text": "mimir-shop", "value": "mimir-shop" } },
+        { "name": "namespace", "type": "query", "datasource": "${datasource}",
+          "current": { "text": "shop", "value": "shop" } } ] },
+      "panels": [ … 6 panels, all bound to ${datasource} … ]
+    }

Checks run before this MR was opened

The assistant ran 7 panel queries against mimir-shop. 6 returned data. It dropped 1 panel.
No counter function runs over a gauge. The assistant checked metric types against the backend's metadata. outbox_queue_depth is a gauge, and the dashboard graphs it as one.
All template variables have a declaration. No panel has a hard-coded datasource UID.
The ConfigMap sits under the tenant folder and carries the label your Grafana sidecar watches.

After merge, expect

Flux applies the ConfigMap in the next reconcile of the monitoring kustomization.
The dashboard appears in Grafana under Shop within a minute, after the sidecar reloads.
All six panels show data for the last 6 hours on first open.

independent reviewer second opinion, no pipeline context

Verified: the ServiceMonitor from !212 is Ready, and the series exist in the tenant. Queries and folder placement match the committed conventions. LGTM.

What keeps it honest.

The AI model (LLM) drafts. Rule-based checks (code, no AI model) decide whether the draft may go out.

Endpoint probed before proposingThe assistant proposes no monitor for a port that does not serve Prometheus format.

Every panel groundedQueries run against your backend. The assistant drops empty panels. It holds a zero-data dashboard for review and never ships it quietly.

Metric types respectedThe assistant uses the backend's own metadata to catch a counter function over a gauge. It escalates the problem and does not rewrite it silently.

Portable by constructionDatasource and namespace are variables. So one dashboard works across clusters and tenants.

Chart owns chart outputIf the dashboard lives inside a Helm chart, the fix goes to the chart's values. It does not go to the rendered output.

Audited after shippingThe assistant checks committed dashboards for known defects (broken variables, hard-coded datasources). It repairs them by MR.

Where this stands.

Live

Missing ServiceMonitor / PodMonitor detection and MRs
grafana.com import with datasource adaptation
Generated dashboards with per-query grounding
Multi-tenant folders and sidecar labels

In development

Scheduled re-grounding of committed dashboards as exporters change
Checking datasource variables against Grafana's own API

Roadmap

Alert rules generated alongside the dashboard, executed against your backend first

From "nobody scrapes this" to a dashboard that actually has data.

A service with a metrics port, and nothing scraping it.

It asks the endpoint before it writes a line of YAML.

Labels copied from the live Service, not guessed.

Your collector does the scraping. The assistant reads the result.

Community dashboards first. Generated only when none fit.

Each query runs against your data before you see it.

A ConfigMap in git. Grafana's sidecar does the rest.

A dashboard that maps one-to-one onto what the service exposes.

Requests / s

p95 latency

5xx ratio

Orders / h

Request rate by route

Latency p50 / p95 / p99

Outbox queue depth

Goroutines