Five Levels of RAG, and the Failure That Ends Each One.

The demo is impressive. It always is. Somebody splits a few hundred documents into chunks, embeds them, wires up a similarity search and a prompt, and by Friday there is a chatbot that answers questions about the wiki. Leadership sees it. Leadership is delighted.

That pattern is retrieval-augmented generation, or RAG: find relevant source material, put it in front of a language model, and ask the model to answer from it.

Then it meets real documents, real questions, and real users, and it turns out that "answers questions about the wiki" was doing a lot of work in that sentence.

I came up through network operations: sixteen years, Cisco IT and Verizon Federal among them. For the last year I have been building the agentic RAG platform that a network operations center at a global enterprise uses every day. Around month three I needed a way to explain to the people funding it why the thing they saw in week two was not the thing they were going to get in week forty, and why that gap was the whole job. So I wrote down the kinds of failure I had run into and the decisions each one forced me to make.

That is what this is: five lenses drawn from one platform's history. They are useful questions to ask of another system, not mandatory stages every RAG project must pass through. A narrow application may never need some of these components.

The labels help organize the stories. They are not a certification, and a strength in one area does not cancel a dangerous gap in another.

What each level means

The names come from how the field already talks about RAG. The research literature describes three paradigms: naive, advanced, and modular. Industry maturity models extend that ladder through production hardening and usually end somewhere around "self-optimizing." My five levels follow the same shape, with the definitions sharpened by what each one cost me.

Level 1, Naive RAG. The tutorial architecture. Documents are split into fixed-size chunks, each chunk is embedded as a vector, and the closest matches to a query are pasted into the prompt. One index, one retrieval path, no awareness of document structure. Every team builds this first, and it is a real architecture. It answers easy questions from clean documents.

Level 2, Advanced Retrieval. Everything that improves what reaches the model. Chunking that follows the document's structure instead of a character count. Hybrid search: dense vectors for paraphrased questions, keyword search for exact terms like model numbers. Query rewriting. A reranker, which is a second model that re-scores the candidates. Parent-child retrieval, which matches on a precise chunk and returns its surrounding section for reading.

Level 3, Modular and Specialized. The single pipeline becomes components with contracts. A router classifies each request and sends it down a specialist path: different indexes, different prompts, different rules per domain. Tools let the system act instead of only answering. Citations are enforced structurally rather than requested politely. The research calls this modular RAG. I think of it as the point where the system becomes a small organization.

Level 4, Production-Hardened. The level that has nothing to do with retrieval quality. Timeouts, concurrency and GPU memory budgets, health checks that do real work, degradation paths instead of indefinite waits, and guardrails that survive a debug flag. This level exists because production failures stop announcing themselves.

Level 5, Instrumented. You can see what the system did, and you can prove whether a change helped. End-to-end tracing on every request. User feedback attached to the trace that produced the answer. A golden set of queries with expected outcomes, including the ones where the right answer is a refusal. Regression runs before anything ships. The industry models usually put "autonomous and self-improving" above this. I will get to why I stopped below it.

Level 1 is not hard. That is the problem.

Fixed-size chunks, one dense vector index, top-k into the prompt. Any engineer can build it in a day from a tutorial, and that is exactly why it is a trap: nothing about building it teaches you what it cannot do.

Fixed-size chunking slices tables, code blocks and multi-step procedures mid-sentence, so the retriever finds the right document and the wrong piece of it. A single dense vector is good at meaning and bad at exact terms; ask about one hardware model number and you get its cousin, because the embeddings sit next to each other. And when the honest answer needs evidence from forty chunks, a stuff-everything prompt cannot hold it.

None of this announces itself as an architecture problem. It shows up as a user asking a precise question, getting a confident near miss, and quietly going back to the old wiki. You do not get many of those before the tool has a reputation, and a knowledge tool with a reputation for near misses does not get a second launch.

One design constraint mattered from the start: operational documentation had to stay inside the network. I built around a locally hosted stack for embeddings, inference, and retrieval. That constraint shaped later model swaps, capacity choices, and latency work. It did not make those changes easy, but it kept the decisions within the environment I was designing for.

The cost of leaving Level 1 is admitting the demo was a demo.

Level 2 is where the chunks lie to you

Advanced retrieval is where interactions start to matter. Hybrid search combines meaning-based retrieval with exact-term search, then needs a rule for merging the results. A reranker is a second model that judges which candidate passages best answer the question. It helps, and it adds another model lifecycle and more latency.

Structural chunking differs by document format. Query expansion may improve recall, but it can also drag the neighbouring product's documentation into the context and dilute everything. These are options to test against actual misses, not a checklist every system must implement.

Here is the failure that caught my attention.

The ingestion checks looked healthy. Chunk sizes were plausible. Citations resolved. The answers could still mix up vendor procedures. The source pages used tabs, one per vendor. A human saw separate sections; the processing path did not reliably preserve those boundaries. My best guess is that the tab titles lived in markup the parser ignored, but I have not confirmed that against the source page.

The pipeline completed. That is what made the missing structure easy to miss.

I planned to compare a PDF export against the HTML path, and to carry the tab context into the chunk metadata. I have not finished that comparison, so I am not claiming PDF fixes it. What I did build was retrieval of small, precise chunks that return the larger parent for reading. Retrieve small, read big, but first check that the parent still belongs to the right section.

Why this failure deserves its own level: a blended chunk does not look wrong. The citation resolves, the prose reads well, and the mistake only surfaces when someone who already knows the procedure reads closely, or when someone who does not follows it. In operations, that is not wrong trivia. That is the wrong vendor's steps under a real alarm. And every layer above this one inherits the damage, because the best reranker in the world can only reorder chunks that were broken at birth.

Before you trust any ingestion pipeline, take your ugliest real document and read the chunks by hand. The structure a human sees is often not in the markup, and no metric will tell you that.

Level 3 is where one pipeline becomes many

Routing to specialists. Prompt templates per domain. Tables, figures and code, not just prose. Tools, so the system can act and not only answer. Citations enforced structurally. At Level 3 the system stops being a pipeline and becomes a small organization, and it fails the way small organizations fail: the wrong person gets the request, nobody knows what the memo says any more, and someone confidently cites a source that does not exist.

Three receipts, briefly.

The router got 20 of 39 benchmark queries right. For that test set, a wrong route could undermine everything downstream. I treated the classifier prompt like code: explicit disambiguators for confused pairs, ordered intent rules, few-shot examples from real misses, and a held-out run before and after. The revised version got 38 of 39 on that benchmark. Later I moved most unambiguous routing to an embedding-based path because LLM classification latency had become noticeable. Those figures describe this test, not a general production accuracy guarantee.

I exposed 57 tools to the agent, and selection got noisier. Every request carried 57 schemas whether it needed them or not. I narrowed the default visible set to 24 through a capability gate that considers tool risk, side effects, and cost. The prompt shrank. Visibility is only one boundary: permission to execute and the result of an action still need separate checks.

The citations were beautiful and invented. A new-hire question got a fluent answer and a References section with four official-looking links. All four were made up. The cause was a chain: no output-format constraint on that path, fallback citation logic skipped that query type, and the stream was not checked after generation. I added layered controls, including a positional contract: the model may cite only by number, checked against a list of real sources. That can stop invented URLs; it does not prove that the answer's claims are supported by those sources.

The thing these three have in common: a prompt rule is a request, not a guarantee. Treat every model output as untrusted input and enforce the things that matter structurally.

Skip that enforcement and the failures compound quietly. A misroute caps everything downstream, so your retrieval improvements stop mattering. An oversized tool list is a distraction surface and an attack surface at the same time. And invented citations teach your newest users to trust fabricated evidence, which is exactly the population least equipped to catch it.

Level 4 is where the failures go quiet

Production means the failures have consequences, and the failures stop announcing themselves.

An LLM request holds GPU memory for the length of a generation, so one enthusiastic user running four deep queries can starve everyone else, and no requests-per-second rate limiter will notice. Poor retrieval does not error; it produces a confident answer from bad context. And the safety mechanism somebody disabled "just for testing" is the one that is off when it matters.

The inference server passed its health checks while failing to complete generation.

The API answered. The model showed as loaded. Every dashboard was green. Meanwhile a document had been stuck in metadata extraction through four attempts of seven-plus minutes each, no error, no timeout, silence. I bypassed everything and sent the smallest request I could think of straight to the server: say only the word OK. It hung for 39 seconds before I killed it. The runner had deadlocked internally, and because it never returned an error, nothing upstream ever gave up. One process quietly blocked the entire ingestion pipeline behind a green light.

The cost of this failure mode is not an outage anyone sees. It is ingestion stalled for a day while every dashboard says fine, a knowledge base quietly going stale, and answers that are correct against last month's procedures. Nobody files a ticket for that. They just stop trusting the answers.

The response was not subtle: hard timeouts on model calls, with degradation instead of indefinite waiting; a three-phase health check covering API response, model load, and a bounded real generation; and a written recovery path for whoever is on call at two in the morning. Only the generation probe catches this deadlock.

On one fixed GPU slice, context length and concurrent generation slots competed for memory. The KV cache holds working memory for active generations; quantizing it to 8-bit let that configuration run three slots instead of two. Four spilled to CPU. I also built a retrieval-confidence gate so low-scoring evidence constrains the response, with a check against a production bypass flag. Those controls address specific failure paths; they do not turn every low score into proof that an answer must be refused.

For LLM infrastructure, "the endpoint responds" is not a health check. "A real generation completes" is. And a safety mechanism that can be turned off by a flag is a suggestion.

Level 5, honestly

The source framework calls its top level "autonomous and self-improving." I am going to be straight about this one, because being straight about it is the point of the whole model.

What I want from an instrumented system is unglamorous. End-to-end tracing with enough metadata per request to investigate a bad answer later. Human feedback attached to the same trace as the pipeline decisions, so a bad answer can be blamed on retrieval or on generation rather than on "the AI." A golden set of queries with expected outcomes, including the negative cases where the right answer is a refusal. Regression runs before any model or prompt change ships. And a review cadence where a person looks at the data and decides what to fix.

None of that comes in a box. There is no standard for LLM observability; you instrument it or you do not have it. Feedback is easy to collect and almost never acted on, because nobody links the thumbs-down to the stage that caused it. Natural language has no unit test. And writing golden queries takes domain knowledge, because you need to know what the wrong-but-plausible answer looks like, not just the right one.

Skip this level and every bad answer stays an anecdote. You cannot say whether retrieval or generation caused it, so you cannot fix it. You cannot prove a change helped, so every model upgrade is a coin flip you do not get to score. And when the system is right and a skeptic says it is not, you have nothing to show them.

What tracing changed for me: diagnosing a bad answer once meant log archaeology across six services. Which classifier fired, which specialist handled it, what was retrieved, which prompt version rendered, what the model saw. With that metadata on one trace, I can open a flagged answer and see all of it, and compare patterns across sessions. I used to spend roughly thirty minutes diagnosing a bad answer. Now it takes about thirty seconds. The trace store includes around three hundred operator sessions; those sessions are material for evaluation, not a validated test set by themselves.

Where the platform actually is: tracing, feedback linked to traces, a golden suite, a regression harness, and confidence measurement exist. The automated deployment gate and ticket from a flagged answer are not wired. I call this instrumented, with a human review cycle. I do not call it self-improving. Someone has to read the evidence, decide what changed, and act. In an operations environment, that is a deliberate boundary.

If you cannot say which pipeline stage caused your last bad answer, that is the next gap to fix, whatever the slide says.

Find the next failure before it finds you

These are checks, not points on a scorecard. Keep the evidence behind each answer; a good result on one check cannot erase a serious failure on another.

1. Have you read, by hand, the chunks produced from your ugliest real document?
2. Does an exact-term query, a model number or an alarm code, retrieve the exact document rather than its semantic neighbour?
3. When a query is routed to the wrong specialist, do you have a test that would have caught it?
4. Can you stop an invented citation before it reaches the screen, and check whether the cited passage supports the claim?
5. Is the visible tool set scoped to the request, with authorization checked again before execution?
6. Does your health check complete a real generation, or just a handshake?
7. Is the production safeguard protected against a debug bypass?
8. For your last bad answer, can you say whether retrieval or generation caused it, in under a minute?

Start with the first question you cannot answer with evidence. A system can have excellent tracing and still invent citations or execute an unauthorized action. No total score makes that acceptable.

The demo was the easy part. I will write up the router separately: 20 of 39 to 38 of 39 on a test set, and why I changed its design again.