CAG vs RAG: When Preloading the Corpus Beats Retrieval — and When It Does Not

The choice between CAG and RAG is decided by corpus shape, not by the acronym. Preloading is the better fit when the corpus fits comfortably inside the model's effective context, changes slowly, and is safe for every user to see. Retrieval stays necessary when the corpus is large, fast-moving, permission-partitioned, or must leave an auditable evidence trail per answer. Vendor prompt caching is a third thing again: a cost and latency layer for any stable prompt prefix. It helps both patterns and replaces neither design decision.

CAG and RAG in production terms

CAG vs RAG (production sense): Retrieval-augmented generation fetches the passages relevant to each query from an index and places only those in the prompt. Cache-augmented generation loads an entire, bounded knowledge collection into a long-context model once, stores the resulting KV cache, and answers every query against it with no retrieval step. Corpus size, change rate, access control and provenance decide between them.

Two ways to put knowledge in front of a model

RAG runs a retrieve-then-generate path on every request: query, retrieve top-ranked passages from an index, assemble context, generate. It scales to corpora far larger than any context window, enforces entitlements at retrieval time, and leaves the retrieved passages as an audit trail. The price is a retriever, an index, chunking, ranking and refresh pipelines — each of which can drop the right evidence before the model sees it. The stage-by-stage failure modes of a production RAG pipeline are mapped in a companion Insight. Moving Generative AI from Demonstration to Production covers the controls retrieval needs in production; this article asks the prior question — when to retrieve at all.

CAG, as proposed by Chan et al. in Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks (WWW Companion 2025), splits the work in two. Offline, the document set is encoded once and its KV cache stored. Online, the cache is loaded with the query and the model generates with no retrieval hop. The authors cite three costs of RAG it avoids: retrieval latency, document selection and ranking errors, and the complexity of operating retriever and generator as one system.

Removing retrieval removes those failure modes and introduces others: the corpus must fit, long-context reasoning must hold up at that length, and every corpus change means rebuilding the cache.

What the "Don't Do RAG" paper shows — and what it does not

The title is a provocation; the evidence is narrower.

• Setup. The experiments use Llama 3.1 8B on the SQuAD and HotPotQA question-answering datasets, with small, medium and large reference sets. The largest is 64 HotPotQA documents, roughly 85,000 tokens. Baselines are sparse (BM25) and dense retrieval at top-3 and top-10.
• Quality. CAG achieved the highest BERTScore in most configurations; with the whole reference set preloaded, there is no retrieval error to make. The gap narrowed as the reference text grew, which the authors connect to long-context degradation.
• Latency. CAG removes retrieval time and is far faster than naive in-context loading that reprocesses the reference text on every query, because the KV cache is computed once. Generation time still rises as the preloaded set grows.
• Stated limits. The authors write that the method requires loading all relevant documents into context, making it well suited to internal knowledge bases of small companies, FAQs and call centres, and impractical for significantly larger datasets. They expect that limit to ease as context lengths grow — an expectation, not a result.
• Stated direction. Their conclusion proposes a hybrid: preload a foundation context and use retrieval only for edge cases or highly specific queries.

Read in that scope, the paper is a strong argument against reflexive RAG for small, stable corpora — not evidence that retrieval is obsolete. Public QA benchmarks scored on one 8-billion-parameter model say little about enterprise knowledge with conflicting versions, entitlements and corpora orders of magnitude larger.

Long context is not free accuracy

A corpus that fits in the window is not the same as a corpus the model uses reliably.

Liu et al. (Lost in the Middle) found that performance on multi-document question answering and key-value retrieval is often highest when the relevant information sits at the beginning or end of the context, and degrades significantly when it sits in the middle — including for models built for long context. Hsieh et al. (RULER) evaluated 17 long-context models: near-perfect needle-in-a-haystack scores did not translate into robust long-context behaviour, almost all models dropped sharply as length increased, and only half of those claiming 32,000 tokens or more held satisfactory performance at 32,000. These are 2024 results on models of that period — evidence of a mechanism, not a current leaderboard.

The consequence: size the corpus against the effective context you measure on your own evaluation set, not the advertised maximum, with margin for instructions, history and output. Effective context is model-specific, so re-run that evaluation whenever the model changes.

Vendor prompt caching is not CAG

Most teams meet "caching" through a provider API, not by holding KV tensors themselves. The mechanics overlap — OpenAI's documentation states that its prompt cache stores KV tensors for a reusable prefix — but the operating model is different.

• Who holds the cache. In CAG as the paper implements it, the application encodes the corpus and holds the KV cache inside an inference stack it controls; the authors used an open-weight model. With vendor prompt caching, the provider holds the cache on its own infrastructure, for a limited lifetime.
• Lifetime and hits. A CAG cache lives as long as you keep it. Vendor caches expire unless reused, and hits are not guaranteed; Amazon Bedrock tells users to check the cache usage fields.
• What is cached. Vendors cache an exact prefix — typically tools, then system instructions, then messages, in that order. A change anywhere before the breakpoint invalidates the cached prefix from that point on.
• Where it applies. Any request with a stable prefix benefits, including RAG requests whose instructions and tool definitions repeat on every call.

Model rosters, minimums and multipliers change frequently; confirm them against the provider's current page before sizing a design.

Two consequences follow. First, a cache write costs more than ordinary input, so caching pays only when the prefix is reused within its lifetime. OpenAI's own worked example puts one write plus one full reuse at 1.35× ordinary input cost, against 2× without caching; across ten requests, 2.15× against 10×. An assistant whose requests arrive further apart than the cache lifetime pays the write premium repeatedly and collects few reads.

Second, cached tokens are discounted, not free. At a 0.1× read multiplier, every query against a 100,000-token cached corpus still pays the input-equivalent of about 10,000 uncached tokens — which can exceed the input of a tightly scoped retrieval context. Preloading through a vendor cache shifts cost from retrieval infrastructure to per-query context rather than removing it — a cost that belongs in the capacity planning described in GenAI FinOps Is a Capacity and Commitment Problem, Not a Dashboard Problem.

Corpus shape decides: a decision frame

Neither pattern wins in general; the table states when each fits better.

Access control is the hard line

One preloaded context means one audience. If two users are entitled to different documents, a single shared cache either over-shares or under-serves. The options are separate caches per entitlement partition — which multiplies write cost and invalidation work — or filtering at retrieval time, which is what a governed RAG pipeline already does.

Provider caches are isolated by organisation (on Anthropic's API, by workspace), not by your end users; OpenAI suggests separate cache keys per customer partly to prevent cache-hit probing across users. Treat cache partitioning as an access-control decision, held to the same least-privilege access controls as any other path to production data.

Freshness is an invalidation problem

Prefix caches invalidate from the first changed token, so prompt order is an architectural decision: the most stable content first (tool definitions, system instructions, foundation documents), volatile content last. In CAG, a corpus change means re-encoding the cache. Treat that like a data pipeline build: a versioned corpus, a scheduled rebuild and an evaluation gate before the new cache serves traffic. Google Cloud warns that changing a storage object referenced by an explicit cache can make the cached content unusable — another reason to version rather than edit in place.

Provenance has to be engineered

RAG produces evidence as a by-product: retrieved passages are logged with the answer. CAG answers from everything at once, so citations depend on the model naming its source. Embed stable document identifiers in the preloaded text, require them in the output, and verify in evaluation that every cited identifier exists and supports the claim. Some providers support citations over cached document blocks, which helps; it does not replace the check.

Hybrid is the common production answer

The paper's conclusion and most enterprise constraints agree: preload what is stable and shared; retrieve what is large, private or fresh.

1. Cached foundation plus selective retrieval. A product manual or policy set is held as a cached prefix. Retrieval runs only for long-tail, customer-specific or recently changed documents, appended after the cache breakpoint.
2. RAG with a cached template. Retrieval is unchanged; instructions, tool definitions and examples are cached — usually without architectural change.
3. Partitioned preloads. A few entitlement partitions each carry their own cached corpus — viable only while partitions stay few and stable.

In every variant, retrieved or user-specific content goes after the breakpoint; placed ahead of the stable material, it invalidates the cache on every query.

Checklist: preload or retrieve

1. Measure the corpus in tokens for the target model, including expected growth.
2. Establish effective context on your own evaluation set at the intended preload size, against a retrieval baseline on the same questions.
3. Map entitlements. If any two users may see different documents, design for retrieval-time filtering or explicit cache partitions.
4. Set a freshness target: how quickly a changed document must affect answers. It fixes re-encode or index refresh cadence.
5. Decide the provenance requirement: answer-level citations verified in evaluation, or per-answer retrieved evidence retained for audit.
6. Model cache economics from the provider's documented write and read multipliers, minimum prefix and lifetime, against real request cadence.
7. Order the prompt: tools, system instructions, foundation documents, then retrieved or user-specific content, with breakpoints at the end of the stable prefix.
8. Instrument cache usage (cached and written tokens) alongside quality metrics; a falling hit rate is a design signal.
9. Re-evaluate on change: model version, corpus size or caching configuration.

What this is not

• Not a verdict that RAG is obsolete. The CAG evidence covers corpora of tens of thousands of tokens on public benchmarks.
• Not an argument to put everything in context. Long prompts are an engineering trade-off.
• Not the same as vendor prompt caching, which is a provider-controlled cost and latency mechanism for any stable prefix.
• Not a benchmark. No Yottalyte measurements are presented here; vendor figures are documented mechanics, not results.

Operational implications

Treat CAG versus RAG as a corpus-shape decision written into the architecture record: size against effective context, change rate, entitlements, provenance and request cadence. Ownership follows. The corpus owner owns re-encode cadence or index refresh; the platform owner owns cache configuration, partitioning and cost monitoring; the evaluation gate covers both and reruns whenever the model, corpus or cache configuration changes.

Preloading is a legitimate production pattern for bounded, stable, shared knowledge; retrieval remains the pattern for large, fast-moving or permission-partitioned knowledge. Most production assistants will run both, with prompt caching paying down whatever prefix stays stable. Our AI & Intelligent Systems practice works through these trade-offs during architecture and implementation, and Managed Data, AI & Cloud Operations covers the monitoring and FinOps work that follows handover.

Frequently asked questions

What is the difference between CAG and RAG?

RAG retrieves the passages relevant to each query and passes only those to the model. CAG preloads a whole bounded corpus, computes its KV cache once, and answers every query against it without retrieval.

Is prompt caching the same as cache-augmented generation?

No. Both reuse precomputed model state, but published CAG has the application hold a KV cache for a whole corpus for as long as it chooses. Vendor prompt caching is provider-managed, prefix-exact, time-limited and priced per write and read — and benefits RAG prompts as much as preloaded ones.

When does CAG beat RAG?

When the corpus fits within the model's measured effective context with margin, changes slowly, is safe for every user to see, and retrieval latency or ranking errors are the main problem — policy handbooks, product manuals and FAQs, for example.

Do larger context windows make RAG unnecessary?

No. Long-context research shows accuracy can fall with length and with where relevant information sits, and large corpora still exceed any window. Retrieval also remains the practical way to enforce per-user permissions and keep per-answer evidence.

References

Vendor documentation reflects the pages as of 8 October 2026.

1. Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks — Chan et al., WWW Companion 2025
2. Lost in the Middle: How Language Models Use Long Contexts — Liu et al.
3. RULER: What's the Real Context Size of Your Long-Context Language Models? — Hsieh et al.
4. Prompt caching — Anthropic
5. Prompt caching — OpenAI API
6. Context caching — Gemini API, Google AI for Developers
7. Context caching overview — Google Cloud
8. Prompt caching for faster model inference — Amazon Bedrock