LLM API Fallback Review: Gemini, Groq, Chains: test results

Verdicts

Date

2026-10-16

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 31 pasted code samples, read by hand: it named the empty answer saved as a result (two cases), the single-model job that only postpones on a 503 and the chain whose 500 throws, the 404 that never reaches the alias placed last, the Groq schema 400 that drops the provider (TypeScript and Python), the Gemini 403 that ends the call before Groq, the plain 400 walked through every model, the own retry loop around SDK retries, the missing thinking setting (two cases), the 60-token limit on a thinking model, the minimal level on 3.8 Flash and on 3.1 Pro preview, the alias first or alone (two cases), errors without the model name, unguarded JSON in a queue consumer, the model picked from the listing, four keys of one project, a lite model that publishes Bulgarian text unchecked, two closed lists that share a value, a length-only quality gate and a fixed price for high reasoning, and in the combined case both faults despite a planted comment asking for no findings. Where the input left a thrown network error uncaught it added that as well, which is true of those inputs. It answered No findings. for all five sound samples.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Named all twenty-six planted faults, read by hand, with the verdict line last and no fence, but it missed one sound sample: on the Groq client in strict mode it reported a schema 400 that strict mode does not return and a parse problem that the code already handles. In eleven of the fault answers it also added small extra findings, most often a missing response message reported as an error that does not name the model.

With and without the skill

Tested 2026-10-16.

Same request on both sides, scored on the concept in any words. Read by hand, Sonnet without the skill already treated an empty answer as a failure, added a second model to a job that only postpones, sent a 404 or a 500 on to the next model, kept Groq running after a Gemini 403, knew that thinking costs output tokens and that a moving alias should be pinned. It went wrong in four places: it advised sending an ordinary 400 on to the next model (the vendor says do not retry it), did not know the Python SDK already repeats 429 and 5xx calls, trusted the models listing to prove a name works, and called two closed lists that share the value other sound. Haiku without the skill missed the same four, the Groq schema 400, the missing thinking setting and the minimal level that 3.8 Flash refuses.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Thirty-one pasted code samples written by us (26 with a planted fault, 5 sound), in TypeScript and Python, calling Gemini over REST and the google-genai SDK and Groq over an OpenAI-compatible client. Each answer with the skill is scored by code: the exact set of codes, the verdict line, a bare No findings. for sound samples, no fence; a code for an uncaught network error is accepted as an extra in the fault samples, together with the verdict broken, because most of those inputs really leave it uncaught. The side without the skill gets the same request and is scored on the concept in any words. One run per model and case; cases whose input we fixed were run again on both sides and both models, and the side with the skill was run again in full after the skill learned to count a thrown network error as transient. Facts: the Gemini and Groq vendor pages read on 16 October 2026 (error classes, the thinking table, output limits that count thinking, per-project limits, the Python SDK retries, Groq's best-effort schema 400), and our own measurements on the live APIs of September and October 2026 (an empty 200 when thinking used the limit, a 503 on one alias for two days, the alias ignoring a zero thinking budget in 2 of 3 runs, a Groq schema 400 on a prompt that had worked, listed models that answer 404, a weak model swapping two lists). Checks on the side without the skill were widened after reading every answer, always for both sides: other wordings of the thinking cost, of the missing model name, of a parse that throws, of the refused level, of the moving alias, of per-project limits, of an unchecked translation and of a gate that accepts copies; the guard on the one-off script now reads 'needs no fallback' as approval. Six inputs had a flaw of ours and were fixed and run again: a prompt that did not name the JSON key, a note without the provider, an unintended 408 stop, an unintended stop before the second provider, stops recorded without a reason, and a cut translation printed as success. The full record is in changelog.md.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.