Runbooks

Tell the agent what your team knows — environments, ownership, how you fix your usual alerts — and it follows it.

The agent investigates with your tools, but it doesn't know your team's habits: which cluster is production, who owns checkout, that rollbacks go through Argo CD. Runbooks is where you tell it.

Who can: Owners, Admins and SREs write them. Everyone in the organization reads them, in the console and the iPhone app (More → Runbooks).

The team's notes

A short page the agent reads before every investigation: in the console, in Slack, from an alert, and in scheduled reports. Good things to write:

  • Your environments: Production is the prod-eu-1 cluster; staging is staging-eu-1.
  • Who owns what, and where to reach them: checkout and payments: the storefront team, #storefront-oncall.
  • Conventions: Rollbacks go through Argo CD, never kubectl.
  • Alerts to take lightly: CPUThrottlingHigh on batch jobs is expected at night.

The agent follows them unless the evidence shows they're out of date. Up to 8,000 characters.

Runbooks

A runbook says how to diagnose and fix one alert or symptom, in Markdown. The agent searches them (by the alert's name, the service, the symptom) when an investigation starts, reads the one that fits, follows its diagnosis steps, and says which runbook it followed. The searches and reads are in the audit log, like any tool call.

Tips:

  • Name it after the alert (Checkout latency (CheckoutLatencyP99High)): that's what the agent searches for.
  • Give steps the agent can check with your tools: List checkout deploys in the last 2 hours, search the logs for "pool exhausted".
  • Say how to fix it, for the human: the agent never changes anything, but it passes the steps on.

Runbooks has a search box (and the iPhone app a search bar): it finds runbooks by their words, the same way the agent does.

On this page