A web server fails by throwing. You get a 500, a stack trace, a line number. An agent fails by succeeding at the wrong thing. It loops for twenty turns, refuses a reasonable request, gets talked out of its system prompt, closes the ticket by deleting the test. Every call returns 200 OK.
That's the gap. Your traces, your logs, your latency graphs all watch the plumbing. None of them watch the conversation.
Every call returned 200. The failures that matter are in the text, and nothing in the trace is looking at the text.
The stakes keep climbing while the visibility stays flat. Agents pull from more tools every month, call their own subagents, run for hours with no human in the loop, and now sit in healthcare, finance, and support queues where a quiet failure costs real money. The harder the agent works, the less a 200 tells you.
The failures live in the text
Error rate and latency throw, which is why you already watch them. The signals that actually tell you whether the agent is any good never make it to your logs at all, because they're semantic. They're buried in what the user and the agent said to each other. A user typing "this is the third time I've asked" and then giving up. A prompt that coaxes the agent out of its own rules. An agent hitting the same tool eight times, getting the same error every time, and trying a ninth. A perfectly reasonable request the agent refused for no reason anyone can find. The agent spilling its chain of thought into the reply.
None of that surfaces on its own. Something has to read the turn and put a label on it, and that something is what we call a reflex: a small classifier that looks at one turn and gives you back one answer. Not a 1-to-10 score that slides around every time you tweak the wording. An actual label, the kind of thing you can count, watch across a release, and get paged over when it jumps.
Eight reflexes, one API call
Nothing to train up front. Send a turn to the API and the label is back in under 90ms. Eight of them come ready to go:
| Reflex | Fires when |
|---|---|
| User Frustration | the user is angry, repeating themselves, or giving up |
| Jailbreak | the input is trying to break the agent out of its instructions |
| Stuck in a loop | the agent is repeating an action with no progress |
| Guardrail | a turn crosses a policy line you defined |
| Incomplete thought | the agent stopped mid-task or left the job unfinished |
| Leaked thinking | internal reasoning leaked into a user-facing reply |
| Difficulty | the turn is hard or the agent needs more information |
| Ambiguity | the request is underspecified |
One call, one classifier, one label:
Run every turn through it. frustrated at 0.97 becomes a row in your dashboard instead of a number sitting in a transcript nobody's ever going to open.
A rate you can alert on
One label on its own is just a data point. Point that same reflex at all of your traffic and it turns into a rate, and a rate is what you actually run a product on. Frustration sits flat at some baseline for a while. Then a deploy goes out and it jumps:
Frustration held near 12% until v2.4 shipped, then doubled within hours. No one read a thousand transcripts to find it. The reflex ran on every turn and the rate moved.
The new system prompt went out at v2.4 and the frustration rate doubled inside a few hours. Nobody had to sit down with a thousand transcripts to catch it. The reflex read every turn and the chart just showed you.
Now shipping becomes an experiment. Push a prompt or model change to 5% of traffic, keep the rest as control, and watch the rates move. If refusals or frustration climb once the change lands, you shipped a regression, and you found out that same day instead of a week later through the churn numbers. That's A/B testing on what the agent actually did, not on what your eval set guessed it would do.
When we don't ship the signal, train it
Eight reflexes cover the common failures. Your product has failures that are yours alone: a medical agent that should never give dosing advice, a coding agent that quietly stubs out a failing test, a support agent drifting off-policy. Describe the behavior, bring a few labeled examples or let us generate a synthetic set, and you have a custom reflex in under an hour. It serves on the same API, at the same latency, beside the built-ins. The public surface is OpenAI-fine-tuning-compatible, so if you've trained a model through the OpenAI SDK, you already know the shape.
The labels feed back into the agent
A reflex is a detector, but the output is training signal. Every label is a row that can become an eval case, a fine-tune example, or a reward term in RL. The turn that flagged as frustrated today is the behavior your main agent learns to avoid tomorrow.
A big model on every turn is too expensive to run on all of production. The detector has to be tiny, or the loop never closes.
I learned this at Tesla, building the stack that went through driving data at scale to surface the frames engineers wanted to train on: the obscured camera, the rare cut-in, the long-tail case the model kept missing. The fleet produced petabytes. Nothing accurate enough to be useful was cheap enough to run on all of it, so the whole game was finding a filter you could afford to run on everything, because the rare event you need is never in the sample.
Treat your production traffic as a funnel. At the top, reflexes run on 100% of turns and flag a few percent. That flagged slice is small enough to hand to the expensive review you could never point at all of production: your team reads the turns that matter, and your own agents dig into the rest. Reflexes don't replace that judgment. They decide what's worth it, so nobody drowns in transcripts that returned 200 and meant nothing.
The cost of the top filter sets how much of production you can afford to look at. Cheap enough to run on 100%, and the expensive review below it only ever touches the slice worth the spend.
That only works because the detector is cheap enough to leave running on everything. One backbone pass labels every turn with many heads, so the tenth reflex costs almost nothing. Read how we built it.
The label was never the hard part. Running it on all of production was, and that's the part we solved.