A runbook can be accurate and still be unusable. Whoever wrote it knew which server, which account and which order of work was meant, so the page never needed to say so. The reader at three in the morning has none of that. They have a pager, half a night's sleep and the page itself. To check one runbook, read it one step at a time and ask, for each step, whether a stranger could carry it out without asking anybody. The answer is usually no at several steps, and the gaps fall into the same few kinds: a step with no exact command or place, no result to compare against, no point where you may safely stop, access or tools that were never mentioned, wording that fits two readings, and an order that only works if you already know it.

Why the reader's state is the design constraint

Google's SRE book gives the reason: under stress the body's hormones degrade thinking and decisions suffer, and among the remedies it names is a well-defined, easy-to-follow set of steps for incidents (Being On-Call, Google SRE book). The SRE workbook says playbooks exist to lower stress, repair time and the risk of human error, and it admits that the authors disagree about style: some keep entries general, others prefer step-by-step entries that reduce variation between people (On-Call, Google SRE workbook). For a procedure that someone will run half asleep, the second camp is the safer one. A step that relies on judgement should say so and say what the judgement is about.

The same chapter offers a warning about runbooks that grow old: details in a playbook go out of date as fast as the production environment changes. A check on wording therefore has a shelf life, and the date of the last real run belongs on the page.

Gap one: the step has no exact command or place

"Clear the cache" names an action, not a command. "Open the dashboard" names a tool without an address. A rested author fills these in from memory, and the tired reader cannot. A usable step carries the literal text to run or the exact location to click, written so that it can be pasted. When the command contains a value that changes between runs, such as an identifier or a date, the step names where that value comes from.

The AWS guidance lists reliance on memory as an anti-pattern in its own right, together with manual changes made without a checklist (AWS Well-Architected, OPS07-BP03). A step that cannot be run without remembering something is exactly that anti-pattern in miniature.

Gap two: no expected result to compare against

"You should see the service come back" is a hope, not a test. The reader needs something to look at and compare: a status value, an entry in the output, a count on a screen, a page that loads. Without it, the reader moves to the next step on trust, and a failure at step three surfaces as a confusing failure at step seven.

Write the expected result next to the command, and add what it looks like when it went wrong. The second half is rarely written, and it is the half that the tired reader most needs, because a wrong result is when they are about to make things worse.

Gap three: no stop point before the irreversible step

Some steps cannot be taken back: dropping a table, rotating a key that other systems still use, promoting a replica. Before such a step, a runbook should say so in words and tell the reader what to confirm first. After it, the step should either explain how to reverse it or state outright that no reversal exists.

The Google SRE book tells the story of an outage that got longer because the team had never tried its rollback procedure in a test environment, and the procedure turned out to be flawed (Emergency Response, Google SRE book). A rollback nobody ever ran is only a promise, not a safeguard. If a step has a rollback line, someone should have run that line at least once.

Gap four: prerequisites nobody listed

A runbook that begins with step one often hides step zero: the access you need, the tool that has to be installed, the environment you must be pointed at. The AWS guidance asks that special permissions and tools be documented, and its template has columns for tools used and special permissions (AWS Well-Architected, OPS07-BP03). PagerDuty's on-call guidance adds the personal side: the laptop, the environment and the repositories should already work on the person's machine before the shift begins (PagerDuty, Being On-Call).

Which environment is the most common hidden prerequisite. A step that says "connect to the database" is dangerous when production and staging both exist and nothing says which one this run concerns. Name it in the step, and show how the reader can confirm it, for example by a value that differs between the two.

Gap five: wording that fits two readings

"Restart the service" leaves two questions open: which one, and where. A runbook for a system with several services, several regions or several copies needs the exact name each time. The same goes for "the primary", "the old node" and "the usual account". A useful habit is to replace every definite article with a name. If a sentence uses "the", ask: the what, exactly?

Gap six: steps that only work in an unstated order

Numbered lists suggest an order, but the dependencies are often hidden: step five assumes that the lock taken at step two is still held, or that the traffic drained at step three has not come back. When steps depend on a condition rather than on the previous step, state the condition. If two steps may run in parallel, say that too, since a reader will otherwise assume strict order and lose time.

The check that can fail: count the questions

Reading a runbook for these six gaps catches most of them, but the only check that can truly fail is a different one: hand the runbook to someone who has never done the task and count their questions. The AWS guidance recommends exactly this validation: have someone else on the team run the runbook, and update it wherever something is missing or unclear (AWS Well-Architected, OPS07-BP03). The Google SRE workbook describes a related practice, in which a team role-plays recent incidents to practise debugging, a practice it names the Wheel of Misfortune (Google SRE workbook).

To make the count meaningful:

  1. Pick a reader who has the access but not the knowledge, and give them no help.
  2. Write down every question they ask, and every time they stop and look puzzled. Each one marks a step that needs repair.
  3. Do the run in a safe environment, so that a wrong guess costs nothing.
  4. After each repair, give the new version to a different person. The same reader will remember the answers.

A runbook that produces zero questions from a stranger has passed. A runbook that produces one question per step has just shown you where to start. The count is also a small trend you can follow over time: if the next review finds more questions, the procedure drifted.

Pair this with the way a runbook is found. The PagerDuty guidance says that information which is hard or impossible to find should be written down, and a runbook nobody can locate during an incident has the same effect as one that does not exist. When an alert always leads to the same list of commands, the SRE workbook points the other way: that is a sign automation is worth building (Google SRE workbook).

A procedure is also a kind of report that someone will act on, and the habit of testing it against what actually happens is the same one described in how to audit what an agent says it did.

When this does not apply

Only procedures that a person runs by hand, written as numbered steps, are covered above. It does not judge whether the commands are technically correct, whether a step is missing, or whether a better order exists; it only asks whether the text as written lets a stranger proceed. Fully automated procedures are better reviewed as code. A runbook for a team whose members all know the system by heart will show fewer questions than a stranger would raise, so the count measures the page, not the team. The advice here comes from the cited guidance and from reading procedures; we ran no study of how many gaps a typical runbook holds, and the six gaps are a working list, not a complete one.

What a skill does for the review

Our runbook-step-check skill reads one pasted runbook and lists only the steps that fall short. For each such step it prints one line naming which of five items is absent: an owner, a time estimate, a visible sign of success, a reversal, and a contact for trouble. It accepts a success sign in any wording, honours an owner or contact named once in a header, and does not demand a reversal where a step merely looks at something. If nothing is missing, its answer is the single sentence No findings. It does not rewrite the runbook, add steps or judge the commands, and a stranger's walk-through is still the check that tells you about commands, places and order.

Read this post as Markdown: /blog/runbook-steps-tired-operator.md · Atom feed.