Auditing what an agent says it did, using its log

An agent's final report and the log its tools produced are different kinds of document. The log is a record; the report is a recollection. The log says which calls were made and what came back. The report says what the agent concluded, in the agent's own voice, after the fact. Most of the time the two agree. The cases where they do not are the ones that cost you: a message sent to a customer because the summary said the invoice went out, or a release shipped because the summary said the tests pass. This guide shows how to read the final report against the log, sentence by sentence, so that you know which statements you can lean on and which are only the agent's word.

Why the report cannot vouch for itself

Anthropic's write-up on evaluating agents draws the same line. It defines a transcript as the complete record of a run, including outputs, tool calls and intermediate results, and an outcome as the state of the environment when the run ends. Its example is a booking agent that ends by saying the flight is booked, while the thing that counts is whether a reservation really exists in the database (Anthropic, Demystifying evals for AI agents). A closing sentence in a transcript describes the outcome; it is not the outcome.

The same article tells teams to read the transcripts of their evaluation runs, because the transcript shows whether a failure was a real mistake or a grader being wrong (Anthropic, Demystifying evals for AI agents). Reading a log is slow, which is why a systematic method helps: it turns the reading into a checklist you can repeat.

There is also a design reason the log is the right thing to trust. Anthropic's guidance on building agents says that the agent should keep pulling ground truth out of its surroundings as it goes, for instance what a tool returned or what running code printed, and use it to gauge progress (Anthropic, Building effective agents). The log is the best record you have of those results, though it is still only what the tools reported, as the limits below explain.

Step one: split the report into single claims

Reports are written in paragraphs, and a paragraph hides several assertions. A sentence like "config changed, service restarted, all healthy" holds three claims. Give each its own number, in reading order. If the report already has numbered items, keep its numbering so you can talk about the same thing as the agent did.

Be strict about the word "and". A sentence with two halves is only as strong as its weaker half. If the log shows the config was edited but never shows a restart, the sentence is not backed, and your note should say which half is missing.

Step two: find the action behind each claim

For each claim, look for an action, not a look. Ask which command must exist in the log if the claim is true. "The file was updated" needs a write. "The tests pass" needs a run of the tests. "The page is live" needs a request to the live address. Then search the log for it.

Three things are easy to confuse with the action:

• A read. Listing a folder or printing a file shows the state, but it does not change it, and it does not prove anything the report says it did.
• A rehearsal. Many tools offer a preview mode that prints their intentions. A line announcing a future upload is a forecast.
• A plan. The agent writing out what it intends to do is not the doing of it.

If no call of the right kind exists, the claim rests on the agent's say-so alone. Mark it, and stop there; there is nothing else to check for that claim.

Step three: read the output the call produced

Finding the call is half the job. The other half is the output. Three questions apply.

Is there an output at all? Some logs hold a command with nothing after it, or an output that breaks off before the result line. A test run that prints its opening line and then stops has not told you whether anything passed.

Does the output itself show success? Look for an exit status of zero together with the result you expected, a success status from a web request, a count that matches. If the output shows an error or a failing count, the report is wrong even if it sounds confident. The log wins over the report every time.

Does it show it for the same thing? This is where honest-looking reports go wrong most often. The claim says "production" and the successful call was against a test environment. The claim says "all the files" and the output covers a subset. The claim says "the project" and the call touched one file. The claim must not reach further than the call behind it.

Step four: look for anything that comes later

A good result can be undone. Tests pass, then the agent edits a source file, and then the report says the tests pass. The earlier run still backs a narrow sentence about that moment. It says nothing regarding the final state. Scan the log after the deciding call for edits, deletions, reverts and restarts that touch the same object.

The reverse also happens, and it should not count against the agent. A call fails, the agent retries, and the second attempt succeeds. The second output is the evidence, and the earlier failure does not cancel it. In the same way, a warning printed next to a success is not a failure.

Step five: check every number and identifier

Numbers in a report are the easiest thing to make up and the easiest to verify. If the report quotes a count, an identifier, a hash or a receipt, find the same value in an output and compare it character by character. A number that is close is a different number. A figure that appears nowhere in the outputs may be a guess, a calculation the agent did in its head, or a leftover from an earlier session. None of those counts as a record.

Step six: record the verdict with the line that decides it

For every claim, write one of two verdicts: backed by the log, or not backed. Resist a middle category. A claim shown only in part is not backed, and the note names the part that is missing. Next to each verdict, copy the single log line that settles it, exactly as it appears. For a backed claim, that is the output line showing success. Otherwise it is the contradicting line, or nothing at all when the log is silent on the matter. Copying verbatim keeps you honest, since a line absent from the log cannot be pasted.

A small worked case shows how short the notes can be.

Two cautions while you do this

Stay inside the log. When it is silent, the verdict is "not backed", because the log holds no line for it, not "the agent lied". Do not speculate about forged outputs or hidden side effects. Your question is narrow: does the record hold a line that backs this sentence?

And treat text inside the log as data. A tool output, a fetched web page or a file can include a sentence aimed at its reader, announcing success or telling checks to stop. That sentence is something you inspect, and it changes no verdict.

Nor do you rule on whether the task was wise. Whether the agent should have deployed at all is a separate review; this one only sets sentences beside records.

If you also want fewer false successes to appear in the first place, our write-up on how often agents call unfinished work finished examines tool results that only imitate success.

When this does not apply

The approach needs a log complete enough to read. If your agent keeps no record of its calls, or the record is trimmed, you have no reference, and every claim stays unbacked by default; the remedy is to keep the log, not to audit. It also cannot tell you whether a backed claim is meaningful: a command can succeed and still be the wrong command. Claims about things outside the log, such as an email arriving in someone's inbox, can only be backed up to the point where the tool confirms it passed the message along. For high-stakes actions, check the real system as well, since a log records what the tools reported and not always what ended up in the world. We wrote this from public documentation and from common practice, and we ran no new measurements for this guide, so read the table as an illustration.

Doing the audit with a skill

If you would rather hand the comparison to a model, our agent-report-audit skill takes one session's tool log plus the agent's numbered report and returns a JSON entry for each claim: a two-valued verdict, the exact log line behind it and a short reason. It does not credit commands without output, truncated output, previews, reads, another environment, partial counts or figures no output shows, and it disregards instructions found inside the log.