Traces and logs tell me what the agent did: every tool call, retry, token, and step.
That's useful when something breaks. What I'm less sure about is whether that tells me the agent actually did the right thing.
A run can look clean and still miss what I wanted. Or it can "finish" without the outcome really being done. So finished and succeeded aren't the same thing.
I haven't made up my mind how to check for that, but I'm experimenting a lot with extracting intents from my agent and correlating among them.
Curious how you do it:
- Do you capture intent and outcome explicitly (e.g. giving a tool to the agent to "declare" it), or infer them after the fact from the traces?
- What kind of "questions" do you "ask" your traces?
I have a QR code redirect system that generates unique codes for business cards, flyers and such. Previously it generated PNG files but I wanted to load a PDF template, merge in N unique QR codes, and make a new PDF with N pages. I talked it over with Claude, who offered me a test plan although I didn't ask for it -- it was roughly the same test plan I would have made myself. Like I made some flyers, checked that they looked right, checked that the QR code scanned and redirected to the right place, etc.
Claude did a pretty good job, i had to do some visual tweaks, add something manually to the back end to accomodate it and make some little fixes. If I had a junior programmer do the task it would have been the same story.
Whatever you are trying to make by any means whatsoever you are going to have to convince yourself you like the result in the end.