Your RAG System Isn't a Prompt Problem. It's a Distributed Systems Problem.

Most teams building RAG treat it like a prompt engineering exercise. Stuff some chunks into a context window, tell the model to "only use the provided context," ship it. When it hallucinates, rewrite the prompt. When it misses information, crank up top-k. When it cites the wrong source, add "always cite your sources" in bold.

None of that works. I think we're framing RAG wrong.

A production RAG system is really three subsystems interacting with each other. An ingestion pipeline, a retrieval engine, and a generation layer. Each one has its own failure modes, its own consistency requirements, its own scaling profile. It's a distributed system. The quality of answers coming out of it is an emergent property of the whole pipeline, not any single piece.

When you deploy RAG, you're not deploying "an LLM that reads your docs." You're running three things at once.

An ingestion pipeline. Document parsing, chunking, metadata extraction, embedding computation, index upserts. It's a data pipeline and it fails like a data pipeline. Schema drift, partial failures, backlog buildup, stale data. A PDF parser that chokes on scanned documents won't throw an error. It will silently produce garbage chunks that sit in your index for weeks before someone notices the answers have gone sideways.

A retrieval engine. Vector index, sparse index, metadata filters, ACL enforcement, reranking. This is search infrastructure and it breaks like search infrastructure. Index staleness, recall degradation, latency spikes, access control leaks. A vector index that falls out of sync with your metadata store won't crash. It will return chunks the user technically shouldn't have access to, and nobody finds out until legal calls.

A generation layer. LLM inference, prompt templates, context assembly, citation extraction, safety filtering. A non-deterministic compute layer that fails in ways no traditional system does. Hallucination, prompt injection, refusal, format violations, cost overruns.

Here's the thing. Each subsystem can work fine on its own while the overall system gives wrong answers. The retriever pulls relevant chunks but the LLM ignores them. The LLM generates a grounded answer but cites a chunk the user shouldn't see. The chunks are great but they're from a document version that was updated last month.

RAG quality is emergent. You can't unit-test your way to it.

If you come from search, you expect bad results to be harmless. The user skips to the next link. RAG flips that. A bad result is a confident, well-formatted paragraph that looks authoritative. The failure mode is not "irrelevant results the user ignores." It's "a fluent answer the user trusts and acts on." Traditional search has a built-in safety valve: the user's own judgment. RAG takes that away.

If you come from predictive ML, you expect evaluation to mean precision and recall on a labeled test set. RAG evaluation means judging faithfulness, groundedness, and citation accuracy. Often that means using one LLM to evaluate another LLM's output. Your existing eval infra probably can't do this. And unlike a classification model where you compute AUC on a holdout set, RAG correctness is statistical. You're maintaining a target quality distribution and watching for when it degrades.

If you come from backend engineering, you expect determinism. Same inputs, same outputs. RAG will hand you the same query, the same retrieved context, and a different answer. Your integration tests will flake. Your regression suite needs fuzzy matching. Your incident response playbook needs a section for "the model changed its mind."

This is where most teams get it wrong. They see hallucination and they go fix the prompt. But groundedness, meaning whether the answer is actually backed by the retrieved context, is a system-level property. It depends on at least six things.

Retrieval quality. If the retriever returns irrelevant chunks, the model has two options: hallucinate or refuse. Neither is good. And "irrelevant" is more subtle than it sounds. A chunk that's topically adjacent but doesn't actually contain the answer is worse than something completely off-topic, because it gives the model just enough rope to make things up.

Chunk quality. Too small and they lack context. Too large and they dilute relevance. Bad boundaries split answers across chunks. Think about a policy document where the answer starts at a heading on one page and continues in the body text on the next. If your chunker splits at the page break, neither chunk has the full answer. The model has to guess.

Context assembly. If relevant chunks are buried in noise, the model might ignore them. A retriever that returns 3 relevant chunks and 7 irrelevant ones is handing the model a context window that's 70% garbage. Your grounding instruction is now fighting the noise for attention.

Prompt design. Yes, the system prompt matters. But it's one factor out of six. A solid grounding instruction can't make up for a retriever that surfaces the wrong documents.

Model capability. Smaller models hallucinate more. Models with poor instruction-following will blow past grounding constraints. A model that handles 3 clean chunks fine might fall apart with 10 noisy ones.

Citation mechanism. If the model can't point to specific chunks, groundedness becomes unverifiable. For you and for the user. The fix for "the model doesn't cite sources" is usually not adding citation instructions. It's tagging each chunk with a source identifier like [Source: Employee Handbook, p.42] before it enters the context window. Give the model something concrete to reference.

Most teams look at latency as one number. But the breakdown tells you where to spend engineering effort and where you're wasting it.

A typical RAG query end to end, assuming self-hosted embedding inference and a well-tuned pipeline:

The LLM call is 90% of the time. Two things follow from that. First, streaming is non-negotiable. The user needs to see tokens within 500ms, not stare at a spinner for 3 seconds. Second, shaving the retrieval pipeline from 200ms to 100ms barely registers in the user experience. But improving retrieval quality (better recall, better reranking) has a huge effect on the answer.

The retrieval stages are cheap in time but expensive in quality. That's where engineering effort belongs.

A production RAG system has at least four caching layers, and they have very different characteristics.

Query embedding cache (exact query text to vector). Low hit rate in practice since most queries are unique, but it's trivial to implement and has no downside.

Semantic cache (similar queries to a cached answer). Research from GPTCache shows that roughly 30% of LLM queries exhibit semantic similarity, so the theoretical ceiling is real. Actual hit rates depend heavily on your query distribution. Saves the entire pipeline when it hits. But invalidation is painful. When a source document gets updated, every cached answer that cited a chunk from that document has to go. Most teams skip this until cost becomes a real problem.

Chunk cache (chunk_id to text and metadata). High hit rate for workloads where a subset of popular documents get queried repeatedly. Saves storage reads. Simple to implement.

Retrieval result cache (query embedding plus filters to ranked chunks). Useful for repeated queries from dashboards or automated workflows. Lower hit rate for ad-hoc user queries.

The gotcha everyone hits is invalidation. When a document gets updated or deleted, you need to kill cached answers that cited it. Not just the chunk cache. Most caching bugs in RAG are not stale chunks. They're stale answers built on chunks that don't exist anymore.

After spending enough time building and debugging these systems, there's a clear order for improving quality.

Fix retrieval first. Most RAG quality issues are retrieval problems wearing a generation disguise. If recall@10 is below 0.80, there's a hard ceiling on answer quality and no prompt will fix it. Put the work into hybrid search (BM25 plus dense), better chunking, and a cross-encoder reranker.

Fix the prompt second. Once the retriever consistently surfaces the right chunks, tighten the system prompt. Grounding instructions, output format constraints, few-shot citation examples.

Fine-tune last. Only when the model keeps failing at something domain-specific even with good retrieval and a solid prompt. Fine-tuning is expensive, hard to iterate on, and tends to cause regressions on general capabilities.

Most teams work in reverse. Weeks on prompt engineering, months on fine-tuning experiments, and they never once measure retrieval recall.

A diagnostic framework that saves a lot of debugging time: treat retrieval quality and generation quality as independent measurements.

If retrieval metrics look good (high recall, high precision after reranking) but answers are still bad, the problem is downstream. Context assembly, prompt design, or the model itself. Check the signal-to-noise ratio in the context window. Check whether the model is actually following grounding instructions. Check whether your few-shot examples are guiding output format the way you think.

If retrieval metrics are weak, stop touching the prompt. Fix chunking, fix the embedding model, fix the search pipeline. No prompt will compensate for a retriever that can't find the right documents.

If both look fine but users are still unhappy, it's probably freshness. The index is stale. The answer is grounded and faithful... to information that was updated two weeks ago and hasn't been re-indexed yet.

One more thing worth noting. Adding context beyond the most relevant 5 to 10 chunks tends to make answers worse, not better. The model gets distracted by the less relevant material. A tight reranker selecting 5 good chunks will beat a pipeline that dumps 20 into the context window every time.

This was the framing issue. Why RAG is a systems problem, not a model problem. Upcoming issues will go deeper.

When BM25 beats embeddings. Hybrid search in practice, how Reciprocal Rank Fusion works, and why your retriever needs both sides.

Chunking strategies that don't kill your recall. Fixed-size, recursive, hierarchical, and the tradeoffs that don't show up in tutorials.

Evaluating RAG without losing it. Regression suites, LLM-as-judge pipelines, and eval gates that block bad deploys before they ship.

Agents vs. deterministic workflows. A decision framework for when an LLM should plan its own actions. Plus the compounding error math that should make you nervous.

The ingestion pipeline nobody designs. Document parsing, metadata enrichment, blue-green index swaps, and what breaks when you swap your embedding model.

If you're building RAG, search, or LLM systems for production, that's what this newsletter covers. Architecture decisions, failure modes, no hand-waving.

See you next week.

Cosine Weekly. Real-world AI engineering: RAG, agents, vector search, and LLMs in production.