Runbook Step Check: test results

Verdicts

Date

2026-10-19

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 3 cases, read by hand. On a password-rotation runbook it listed only the two steps with gaps, step 3 without a signal and a way back and step 5 without a time and a signal, and left out the read-only checks and the step whose signal was written in other words. On a failover runbook it listed the step with no owner and the step with no way back, and left out the promotion step that says plainly it cannot be undone. On a complete runbook it answered exactly No findings.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 3 cases, read by hand: the same two lines per planted runbook with the same codes, none of the safe look-alikes listed, no escalation or owner line for the runbooks that state them once, and exactly No findings on the complete runbook.

Note

Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three runbooks were written by us; each planted one carries safe look-alikes (a success signal written in other words, read-only steps without a way back, a step that says it cannot be undone, an owner and an escalation contact stated once), and neither planted case has a comment pointing at the gap. Facts come from public incident-response guidance, read on 19 October 2026; there are no owner-measured facts.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.