Production RAG Is a Pipeline of Failure Modes — Not an Embeddings Demo

A production RAG pipeline should be engineered and owned as a chain of failure modes and controls, not judged by the embedding model behind the demo. Most wrong answers trace to a specific stage: the answer was never indexed, was ranked out of the top results, was dropped while context was packed, or was present but not extracted. This article maps those stages, the failure points researchers have documented, and the controls architecture and platform teams should own through handover.

What is a production RAG pipeline?

Production RAG pipeline: An information path that prepares enterprise content into a searchable, permission-tagged index and, at query time, retrieves, filters, ranks and assembles only authorised context for a generator — with citations, evaluation and monitoring — so that every answer can be grounded, traced to its sources and operated after go-live.

The index path runs before any question is asked; the query path runs per request. Each fails differently and needs different owners.

What the original design solved — and what production still owns

Lewis et al. introduced RAG as two memories working together: the parametric memory inside a pre-trained generator and a non-parametric memory — a dense vector index — retrieved at generation time. Knowledge can be updated by replacing the index rather than retraining the model, and retrieved passages offer a route to provenance (Lewis et al., 2020). Their Wikipedia corpus was split into 100-word passages: a research choice, not a production prescription.

Enterprise corpora add what the paper did not need to solve: multi-format documents, per-user permissions, identifiers that similarity handles poorly, daily change and auditable sourcing. Our Insight on moving generative AI from demonstration to production treats the model as one component in a governed request path. This article opens the context stage of that path and engineers it stage by stage.

The index path decides what can ever be found

Ingestion and parsing

Retrieval cannot return what parsing destroyed: flattened tables, orphaned headings and unextracted scans degrade every later stage. Parsing is also a security boundary: OWASP describes a résumé carrying hidden white-on-white instructions that a RAG screening system later followed, and recommends extraction that detects hidden content plus validation of every document before it enters the knowledge base (OWASP LLM08:2025). The control is a governed intake: approved sources, quarantine on validation failure, and source permissions captured as metadata at ingestion.

Chunking and embedding

Chunk size is a trade-off. Chunks that are too small lose the context a complete answer needs; chunks that are too large dilute relevance and spend tokens on noise. Snowflake recommends Cortex Search chunks of no more than 512 tokens, noting that smaller chunks typically give more precise retrieval and better downstream answers (Snowflake Cortex Search) — a starting point to test, not a rule.

Two disciplines prevent silent regressions. Queries must be pre-processed exactly as chunks were and embedded with the same model, as Microsoft's architecture guidance states (Azure Architecture Center). And the embedding model is a pinned dependency: changing it, or the chunking policy, means re-embedding the corpus and re-running evaluation.

Indexing and freshness

An index goes stale when sources change, and platforms expose freshness as an explicit, priced parameter. Snowflake Cortex Search sets a TARGET_LAG for how far served content may trail the base table; Databricks AI Search Delta Sync indexes stay synchronised as source tables update; Azure AI Search indexers support incremental indexing (Snowflake; Databricks AI Search; Azure AI Search).

Permissions go stale too. Microsoft documents that changes to group membership or source ACLs reach search results only after permission metadata is resynchronised to the index (Azure AI Search document-level access control). Freshness targets must cover access as well as content: a user whose access was revoked this morning should not keep retrieving restricted documents until the nightly sync.

The query path: recall first, then precision, then packing

Query rewriting and planning

Users ask vague or compound questions in language that rarely matches the documents. Microsoft groups the remedies as query translation: augmentation, decomposition, rewriting and hypothetical document embeddings (HyDE). Agentic retrieval goes further, letting a language model plan parallel sub-queries and return structured results with citations. Azure AI Search recommends it for new builds, but its LLM-based query planning is still preview; classic single-query RAG remains simpler, faster and the option when only generally available features are acceptable (Azure AI Search). Amazon Bedrock Managed Knowledge Bases also offer agentic, multi-hop retrieval (Amazon Bedrock Knowledge Bases). Planning buys recall with extra model calls and latency; measure it before making it a default.

Hybrid retrieval

Vector search matches meaning but can miss exact tokens such as SKUs or error codes; keyword search (BM25) misses paraphrase. Hybrid retrieval runs both and fuses the lists, commonly with Reciprocal Rank Fusion (RRF), which merges by rank so incompatible score scales do not matter. Azure AI Search applies RRF to hybrid queries automatically; Databricks AI Search fuses BM25 and vector results with RRF and recommends hybrid search where data contains unique identifiers; Snowflake Cortex Search combines vector and keyword retrieval with semantic reranking by default (Azure Architecture Center; Databricks AI Search; Snowflake). For enterprise corpora, hybrid is the baseline to measure against.

Reranking

Retrieval optimises for recall; reranking optimises for precision. Without it, marginally relevant chunks dilute the context. A common design retrieves broadly (say, the top 50), merges with RRF, applies a cross-encoder or semantic ranker, then truncates to a small final set. Azure's semantic ranker rescores the top 50 results rather than the corpus, and Bedrock Knowledge Bases accept reranking models (Amazon Bedrock Knowledge Bases). The price is latency: Microsoft advises benchmarking relevance and latency on real test queries before adopting a reranker.

Context assembly and generation

More retrieved text is not more truth. Liu et al. found that multi-document question-answering performance was often highest when the relevant passage sat at the start or end of the input and degraded in the middle; longer inputs could hurt even long-context models. In their open-domain study, reader accuracy saturated well before retriever recall, so many more passages bought marginal gains for real cost and latency (Liu et al., "Lost in the Middle").

Context assembly therefore needs its own policy: a token budget, a chunk limit, evidence ordering, de-duplication, and separation of system instructions from retrieved content.

Citation and grounding

Citations let users check answers and let operators reconstruct what the model saw. Bedrock Knowledge Bases can attach citations to generated responses, and Azure agentic retrieval returns grounding data with citations. Whatever the platform, store source identifiers and versions with every answer, and treat claims that cannot be mapped to retrieved sources as a failure, not a stylistic issue.

Seven failure points, mapped to stages and owners

Barnett et al. catalogued seven failure points from experience engineering RAG systems. Each points to a stage, and therefore to an owner (Barnett et al., 2024).

1. Missing content — the corpus lacks the answer, yet the system answers anyway. Owner: corpus scope. Control: coverage reviews and an explicit abstain path.
2. Missed top-ranked documents — the answer exists but falls outside the top-k. Owner: retrieval. Control: hybrid retrieval, reranking, Recall@K tracking.
3. Not in context — retrieved, then dropped when context was consolidated under token or rate limits. Owner: context assembly. Control: an explicit token budget; truncate after reranking, not before.
4. Not extracted — in context, but the model fails to extract it amid noise or contradiction. Owner: assembly and generation. Control: fewer, better-ordered chunks; deliberate conflict handling.
5. Wrong format — required structure ignored. Owner: generation. Control: constrained output and format validation.
6. Incorrect specificity — too general or too narrow. Owner: query understanding and generation. Control: query rewriting; evaluation sets that encode expected detail.
7. Incomplete — a partial answer despite available information. Owner: retrieval breadth and generation. Control: decomposition for multi-part questions; completeness checks.

The authors conclude that RAG validation is only feasible during operation and that robustness evolves rather than being designed in up front. Evaluation and monitoring are therefore standing responsibilities.

Operating controls that make retrieval governable

Enforce permissions before the prompt

Once a chunk is in the prompt, it has been disclosed to the model, so access control belongs inside retrieval rather than in post-processing of the answer. OWASP lists unauthorised access, cross-context leaks, embedding inversion and poisoning among vector and embedding weaknesses, and recommends permission-aware vector stores, logical partitioning and immutable retrieval logs (OWASP LLM08:2025).

Platform support differs and should be checked against current documentation:

• Azure AI Search: string-based security filters for document-level trimming are generally available. Native token-based enforcement with Microsoft Entra ACL and RBAC scopes, Microsoft Purview sensitivity labels and SharePoint in Microsoft 365 ACLs is preview, which Microsoft says carries no SLA and is not recommended for production workloads (Azure AI Search document-level access control).
• Amazon Bedrock: Managed Knowledge Bases filter by document-level ACLs at retrieval time for their connectors, excluding Web Crawler; customer-managed knowledge bases do not include this capability (Amazon Bedrock Knowledge Bases).
• Databricks AI Search: row- and column-level permissions are not supported natively; application-level ACLs are implemented through the filter API (Databricks AI Search).
• Snowflake Cortex Search: access is granted per search service and searches run with owner's rights, so per-user document filtering is an application design responsibility (Snowflake).

The rule is the same everywhere: resolve the caller's identity, filter candidates against synchronised permission metadata before anything reaches the prompt, and log which sources were eligible and which were used.

Treat retrieved text as untrusted input

OWASP states that RAG does not fully mitigate prompt injection, and describes an attacker editing a document in a RAG repository so that retrieved instructions alter the output (OWASP LLM01:2025). The mitigations map onto the pipeline: validate sources at ingestion, segregate and label retrieved content so it cannot pass as instruction, validate output formats, apply least privilege, and require human approval for high-impact actions.

Evaluate retrieval and generation separately

A single end-to-end score hides which stage failed. Measure retrieval on labelled queries with Precision@K, Recall@K and mean reciprocal rank, including negative queries the corpus should not answer (Azure Architecture Center). Measure generation with grounding metrics: Ragas defines faithfulness as the fraction of a response's claims that can be inferred from the retrieved context, from 0 to 1 (Ragas). Add checks for citation correctness, format and abstention.

No single metric proves a RAG system safe or useful. Keep evaluation sets in version control, run them as release gates whenever the embedding model, chunking, prompt, reranker or generator changes, and refresh them from production failures.

Budget cost and latency as pipeline parameters

Candidate-set size, rerank depth, final top-N, planning effort and freshness targets each move both quality and spend. Microsoft suggests starting with a moderate candidate set of 20 to 50 and tuning against evaluation metrics; Liu et al.'s saturation finding points the same way. Treat these as owned parameters with defaults and change control. Governing the wider model spend is covered in our Insight on GenAI FinOps as a capacity and commitment problem.

Monitor the pipeline, not only the model

Telemetry should make a wrong answer diagnosable by stage: query rewrites, retrieved and reranked source identifiers with scores, the permission filter applied, context tokens, dropped chunks, model and prompt versions, citations, abstentions and per-stage latency. Managed services expose part of this, but trace design and alert thresholds remain operating decisions. Log source references rather than sensitive payloads where data policy requires.

Choosing retrieval patterns by constraint, not by vendor

None of these rows is a platform recommendation. The patterns can be built on AWS, Azure, Databricks, Snowflake and others; choose by corpus, access model, latency budget and operating team.

Handover questions for a RAG implementation

1. Who owns source onboarding, validation and quarantine?
2. How often are permissions resynchronised, and what lag after revocation is acceptable?
3. Which embedding model and chunking policy are pinned, and what triggers a re-embed?
4. What are the default candidate-set size, rerank depth, top-N and context token budget, and who may change them?
5. Where do evaluation sets live, and which results block a release?
6. What does the system do when the corpus does not contain the answer?
7. How are citations and retrieval traces stored without unnecessary sensitive logging?
8. Who is on call for retrieval failures, and who for model behaviour?

In white-label and co-delivery engagements the partner retains the client relationship; the answers belong in the handover artefacts either way.

Frequently asked questions

What are the main stages of a production RAG pipeline?

An index path — ingestion and parsing, chunking, embedding, and indexing with permission metadata — and a query path: query planning, hybrid retrieval, reranking, permission trimming, context assembly, generation, citation, evaluation and monitoring. Each stage has its own failure modes and needs a named owner.

Is vector search enough for enterprise RAG?

Usually not on its own. Vector search handles paraphrase but can miss exact identifiers such as product codes or policy numbers. Hybrid keyword-and-vector retrieval followed by reranking is the pattern Azure AI Search, Databricks and Snowflake each document.

Where should access control be enforced in a RAG pipeline?

Inside retrieval, before any content reaches the prompt, using permission metadata captured at ingestion and kept synchronised with the source system. Filtering the generated answer is too late: the model has already seen the restricted content.

What this is not

• Not an argument that RAG is obsolete or that larger context windows remove the need for retrieval engineering.
• Not a comparison of vector, graph-based and other retrieval architectures.
• Not a vendor ranking. Platform names illustrate documented mechanisms and reflect technology capability coverage, not formal partnerships or certifications.
• Not a restatement of the production GenAI operating model; it is the pipeline-level companion to it.

Operational implications

A production RAG pipeline succeeds when it is owned like production software: governed intake, pinned embedding and chunking, measured hybrid retrieval and reranking, permissions enforced before the prompt, context budgets, stored citations, and evaluation on every change. The embedding model is one dependency among a dozen stages that each fail in their own way.

That work sits within AI & Intelligent Systems, with continuity through Managed Data, AI & Cloud Operations when refresh pipelines, permission sync, evaluation gates and retrieval telemetry need an owner after go-live. Prefer a pipeline you can diagnose over a demo that merely answers.

References

Primary sources used for technical claims in this article. Inline citations in the body remain authoritative for the locus of each claim.

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., NeurIPS 2020
2. Seven Failure Points When Engineering a Retrieval Augmented Generation System — Barnett et al., CAIN 2024
3. Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL 2024
4. LLM01:2025 Prompt Injection — OWASP Gen AI Security Project
5. LLM08:2025 Vector and Embedding Weaknesses — OWASP Gen AI Security Project
6. RAG and generative AI — Azure AI Search
7. Develop a RAG solution: information-retrieval phase — Azure Architecture Center
8. Document-level access control — Azure AI Search
9. Retrieve data and generate AI responses with Amazon Bedrock Knowledge Bases — Amazon Bedrock
10. Databricks AI Search — Databricks
11. Cortex Search — Snowflake
12. Faithfulness — Ragas documentation