Building a production RAG pipeline: the latency, cost and accuracy trade-offs

Retrieval-augmented generation is easy to build and hard to make good, and the reason is that almost every decision improves one of latency, cost and accuracy at the expense of the other two. A pipeline that is "slow and expensive and still wrong" is usually one where nobody chose which of the three to give up.

This walks the pipeline in order, naming the trade at each stage.

The failure is almost never the model

When a RAG system answers badly, the instinct is to change the model or the prompt. Measure first, because the fault is usually further up:

1. Retrieval did not return the right chunk

   By far the most common. Check by hand: search the query and read what came back. If the answer is not in there, no model can produce it.

2. It returned the right chunk, cut in the wrong place

   The relevant passage is split across a boundary, so each half looks half-relevant and neither ranks.

3. It returned the right chunk and the model ignored it

   Genuine, and much rarer than people assume. Usually a symptom of too many chunks — the signal is buried.

Chunking: the decision that constrains everything downstream

Chunk size is the single most consequential parameter, and it is a direct trade.

Two techniques that improve the trade rather than just moving along it:

Chunk on structure, not on character count. Split at headings, then paragraphs, and only fall back to a fixed window inside an oversized block. A chunk that begins mid-sentence embeds badly and reads badly.

Retrieve the small chunk, send the larger parent. Index precise chunks for matching but pass the surrounding section to the model. Precision where it helps — the search — and context where it helps — the generation.

// Index the child, keep a pointer to the parent.
const chunks = splitByStructure(doc, { target: 220 }).map((child, i) => ({
  id: `${doc.id}:${i}`,
  text: child.text,          // what gets embedded
  parentId: child.sectionId, // what gets sent
}));

Retrieval: dense alone loses on the queries that matter most

Vector search finds semantic neighbours, which is exactly what it is for and exactly why it fails on identifiers. A query for an error code, a part number, a person's surname or an API name is a lexical query, and an embedding blurs precisely the token that mattered.

Hybrid retrieval fixes it: run both, fuse the rankings.

-- pgvector for meaning, full-text for the exact string, fused by rank.
WITH dense AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks ORDER BY embedding <=> $1 LIMIT 50
),
lexical AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
  FROM chunks, plainto_tsquery($2) query WHERE tsv @@ query LIMIT 50
)
SELECT id, SUM(1.0 / (60 + rank)) AS score
FROM (SELECT * FROM dense UNION ALL SELECT * FROM lexical) fused
GROUP BY id ORDER BY score DESC LIMIT 20;

Reciprocal rank fusion is the whole algorithm — no score normalisation, no tuned weights, and it is robust because it only uses ordering. The single largest accuracy improvement available to most RAG systems is adding lexical retrieval to a dense-only pipeline, and it costs one index and one query.

Reranking: the best accuracy-per-millisecond in the pipeline

A cross-encoder reads query and passage together rather than comparing two independently computed vectors, so it is far more accurate — and far too slow to run over a corpus. Used as a second stage over 20–50 candidates, it is affordable.

Send fewer, better chunks. Eight well-ordered passages beat twenty mixed ones on accuracy and on cost — which is the rare case where two of the three move together.

Caching: the only lever that improves all three at once

1. Exact-match cache on the normalised query

   Trivial, and in most production systems it hits more than anyone expects. Support and internal-docs traffic is extremely repetitive.

2. Semantic cache on the query embedding

   A near-duplicate question reuses the previous answer above a similarity threshold. Tune that threshold carefully and log near-misses — too loose and it answers a different question confidently.

3. Embedding cache on content hash

   Re-embedding unchanged documents on every ingest run is pure waste, and it is the default behaviour of most naive pipelines.

4. Prompt-prefix caching where the provider supports it

   A stable system prompt and instruction block in front of a varying question is exactly the shape prefix caching rewards.

Where the latency actually goes

Before optimising, instrument each stage separately. The distribution is usually not where people guess:

Stream the answer. It does not reduce total latency by a millisecond and it changes the experience completely, because time-to-first-token is what a person perceives as speed.

Freshness is a design decision, not an implementation detail

The question nobody asks until it bites: how stale may an answer be?

1. Ingest incrementally, keyed by content hash

   Rebuilding the whole index on every change is the default and it does not survive a real corpus.

2. Delete properly

   A chunk whose source document was removed will keep being retrieved and cited. Deletions are the part of ingestion that gets skipped and the part that produces the worst failure — confidently citing something that no longer exists.

3. Carry a timestamp into the citation

   So the answer can say when its source was last updated, and a reader can judge it.

Citations are a correctness mechanism

Requiring the model to cite the chunk each claim came from is usually framed as a UX feature. It is a correctness one: a claim with no supporting chunk is detectable, automatically, and can be dropped or flagged before it reaches anyone.

The short version

1. Build the evaluation set first

   Otherwise you cannot tell retrieval failure from generation failure.

2. Chunk on structure; retrieve the child, send the parent

   Precision where it helps, context where it helps.

3. Add lexical retrieval to your dense search

   Usually the largest single accuracy gain, and it costs one index.

4. Rerank 50 down to 8

   Best accuracy per millisecond in the pipeline, and it reduces generation cost too.

5. Cache at four levels

   Exact, semantic, embedding, prompt prefix. The only lever that helps all three axes.

6. Verify citations mechanically

   A cited chunk that was never in the context is a detectable hallucination.