> ## Content Index
> Fetch the complete content index at: https://www.reasonforoutage.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Your Agent Answered. Did It Do the Job?
- URL: https://www.reasonforoutage.com/your-agent-answered-did-it-do-the-job/
- Published: 2026-10-12T12:00:00.000Z
- Updated: 2026-10-20T11:47:44.000Z
- Description: A fluent answer, a tidy References section, and four links that did not exist.
- Author: Christy Steyn
- Tags: Agents, Architecture

A new hire asked the platform a question. The answer was clear, well organized, and ended with a References section: four official-looking links.

None of them were real.

Nobody noticed at first, which is the whole story. The agent answered. It just had not done the job.

## Symptom

A fluent answer with a References section. It looked exactly like the good ones. If you did not click the links, you would never know.

That is the point I keep coming back to. An invented citation does not look like a failure. It looks like diligence.

## What the system did

I traced it back. It was not one mistake. It was a short chain of small ones, each easy to defend on its own:

1. The path that handled this question had no constraint on the shape of the output, so the model was free to write a References section if it felt like it.
2. The fallback logic that normally builds citations skipped this type of query.
3. Nothing checked the answer after it was generated. Once the stream started, whatever came out went to the screen.

Any one of those, I would have shrugged at in a review. Together they let a made-up source list reach a new hire.

## Why I missed it

My checks asked whether an answer came back, and whether it came back in time. They did not ask whether the sources behind it existed.

That is the blind spot. I was measuring whether the agent responded. I was not measuring whether the response was the job.

The fix for that is not a smarter prompt. It is being able to look at what the system did, stage by stage, so a fluent answer cannot hide a broken one.

## What I changed

I stopped trusting the prompt to behave. A prompt rule is a request, not a guarantee.

I added controls in layers, and one of them is a positional contract: the model is only allowed to cite a source by number, and each number is checked against the list of sources that were really retrieved. If it cites something that is not on the list, it does not get through.

That stops invented links. I want to be straight about what it does not do. It does not prove the claims in the answer are actually supported by the sources it cites. A real link beside an unsupported claim is still a bad answer. That is a separate check, and I treat it as a separate problem.

## What "did the job" means

This incident is why I now ask the same question of every request before I tune anything. Three requests can show up in the same chat box and need three different things.

"What does this alarm mean?" needs a good, sourced explanation.

"Is it affecting anything right now?" needs current evidence, not a documented procedure.

"Open a ticket with the findings" changes another system.

Treating those as one workload is how a system gets good at sounding done. The first needs an answer people can verify. The second needs evidence. The third needs a boundary around what the system may do and a clear difference between "drafted" and "submitted."

If a model can lie about where its answer came from, an interface can just as easily say something happened before it did. Same bug, higher stakes.

## The decision for you

Pick one task your system is supposed to complete. Write down what "done" means in one sentence, including what it must not do.

Then ask what evidence you would keep to prove it did. If the only evidence is that it answered, you have found your next gap.

Treat every model output as untrusted input. Enforce the things that matter with code, not with a polite instruction in a prompt.

*I write post-mortems for agentic systems at Reason for Outage: real failures, traced all the way down. Subscribe if you want the next one.*