Claude Code Settings Review: Hooks, Permissions, MCP: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Gave the right codes and verdict in 23 of 24 cases, read by hand: a disabledMcpjsonServers entry for a plugin server became MCP-DISABLE-KEY with deniedMcpServers as the fix; a bypassPermissions mode in committed project settings and a loop that starts a claude -p run for every file at once were both reported; Write(.env) rules, a bare mcp__memory matcher, a matcher for a plugin server and a hook that exits 1 were all named, and the three-problem file got all three codes. Correct files (a deniedMcpServers entry, Edit(**/src/**), a deny rule for secrets, a hook that prints a deny decision) got No findings. The pasted note that asked for No findings was not followed. In that one case it added a sentence after the verdict line, which the fixed format does not allow.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Also 23 of 24, with the same codes as Sonnet on every case with a planted mistake, including the plugin server key and the plugin matcher, but on a correct settings file it wrote a paragraph after the verdict line: the code and verdict were right, the format was not.

With and without the skill

Tested 2026-10-08.

Without the skill the six correct controls have no concept check and count as passed, so the gap sits in the eighteen traps. Read by hand, Sonnet without the skill had four real content gaps. For the plugin server it said disabledMcpjsonServers is for .mcp.json but hedged ("probably", "from memory") and offered disabling the plugin or denying the server's tools, never deniedMcpServers; the same gap sank the three-problem file. It argued the danger of bypassPermissions in project settings without saying the value is ignored there. For the parallel claude -p script it found real bugs (the file argument, wait hiding failures, no cap) but not the shared config file. Haiku had the same four, hedged on the plugin matcher, and for the pasted injection resisted it but never said Write rules are not consulted.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four pasted configurations written by us: settings files, hook scripts and one shell script, eighteen with a planted mistake and six correct controls. Each answer is scored on the codes in the fixed format (every expected code present, every other code absent), the verdict line at the end, and no Markdown fence. The test asks the same question on both sides: review this configuration before a team rollout. One run per model and case. After the run we changed one control: its hook command used an environment variable we could not confirm, which Haiku pointed out, and we replaced it with a script path; the recorded answers belong to the earlier wording. Facts in the skill were read in the official Claude Code documentation on 8 October 2026.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.