IAM Policy Review: test results

Verdicts

Date

2026-10-19

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 3 cases, read by hand. On the deploy role it gave the registry wildcard, the pass-role grant and the policy-version grant, and left out the read-only bucket and the scoped log group. On the bucket policy it gave the open principal and the NotAction allow and left out the organization-limited principal, though in the opposite order from the list. On the clean policy it answered exactly No findings.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 3 cases, read by hand. It matched the strong model on the deploy role and on the bucket policy, adding a sensible note that the public statement may be intended. It put a blank line between the two findings of the bucket policy, which the check accepts, and answered exactly No findings on the clean policy.

Note

Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three inputs were written by us; each planted one carries a safe look-alike and neither planted case has a comment pointing at the defect. Facts come from public documentation, read on 19 October 2026; there are no owner-measured facts.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.