B2Shift · Published · Updated

Retrieval-augmented generation (RAG) lets a system answer from your documents instead of a model's general training data, with a citation back to the source. The idea is simple; the cost variance in practice comes from a small number of decisions made early.

What a RAG pipeline actually does

Documents are split into chunks, converted into vector embeddings, and stored in a vector database. A question is embedded the same way, the closest matching chunks are retrieved, and the model answers using only those chunks as context — with a citation back to the source document. Remove the citation step and you have a system nobody can audit when it's wrong.

What drives the cost more than model choice

Four things move the number more than which language model you pick. Document volume and format: scanned PDFs need OCR before they can be chunked, and OCR errors propagate into every answer downstream. Chunking strategy: naive fixed-length chunking is cheap to build and frequently wrong on structured documents like contracts or tables. Retrieval quality: a vector database has real infrastructure and query costs at scale that need tuning, not just installation. And evaluation: without a test set of real questions and known-correct answers, you cannot tell if the system is accurate — you can only tell if it sounds confident, which is a different thing entirely.

Where accuracy actually breaks

Poor-quality source documents are the most common failure, not model limitations — a RAG system cannot answer correctly from a document that is itself wrong or ambiguous. Tables, scanned handwriting and multi-column layouts need special handling; naive extraction silently drops or scrambles them. And permission-aware retrieval matters as much as accuracy: a system that can retrieve a document a user shouldn't see is a security bug, not an accuracy bug.

What a realistic build includes

A production RAG pipeline needs a defined confidence threshold below which the system says "I don't have a reliable answer" instead of guessing, a review queue for low-confidence cases, and an accuracy report measured against your own test set — not a generic benchmark, because your documents are not the benchmark's documents. See Document Automation & RAG for how we scope this.

What a RAG implementation estimate should include

A retrieval-augmented generation (RAG) system has costs before a model answers a question. Estimate the source inventory, document cleanup, ingestion frequency, access rules, retrieval quality checks and ongoing ownership. The setup and monthly operating estimate should use the same expected document volume and query volume so the comparison is meaningful.

RAG cost drivers from ingestion through ongoing evaluation
Work areaCost drivers to scopeFailure to test
Ingestion and normalizationConnectors, formats, OCR, deduplication and refresh scheduleScanned or stale files are silently omitted
Chunking and indexingChunk size, overlap, metadata, embeddings and index storageTables or headings are split from their context
PermissionsIdentity mapping, document ACLs and permission refreshA user retrieves a document they cannot access
Retrieval and rankingQuery rewriting, filters, reranking and latencyA relevant passage ranks below a similar but wrong one
Citations and evaluationSource links, test questions, expected passages and review timeAnswers sound confident but cite the wrong version
OperationsRe-indexing, monitoring, model/API usage and supportA source change breaks answers without an alert

Budget setup and monthly operation separately

One-time work commonly includes source-system access, data profiling, ingestion pipelines, permission mapping, index design, a representative evaluation set and user acceptance. Recurring costs can include embedding new or changed files, vector storage, model calls, retrieval infrastructure, monitoring and human review. Ask for low, expected and high volume assumptions, and state whether costs include re-indexing and source-system changes. A small evaluation set should contain real questions, expected source passages and known failure cases; test stale versions, missing access, conflicting documents and questions that have no supported answer.

Inventory sources, owners and update behavior

List each content source before estimating connectors: shared drives, knowledge bases, document stores, databases or approved web pages. Record its owner, format, approximate size, refresh frequency, access model and whether it contains scanned tables or images. A corpus of clean, current HTML is different work from years of duplicated PDFs with inconsistent naming. Ask who can approve ingestion and how the team will identify retired, replaced or restricted material.

Plan for the lifecycle of a source. A document change may require removing the old text, extracting and chunking the new version, regenerating embeddings and refreshing metadata. Decide how quickly updates must appear, whether a full or incremental re-index is required, and how failed ingestion is detected. Keep version and source identifiers on each chunk so a result can be traced back to the authoritative document and withdrawn when access or policy changes.

Choose chunk boundaries from document structure

Chunk size and overlap affect both retrieval quality and operating cost. Very small chunks can separate a rule from its exception or a table heading from its values. Very large chunks can mix several topics, dilute retrieval and use more model context than the answer needs. Start by preserving meaningful boundaries such as headings, sections, table rows and document identifiers; then test a few chunking approaches on representative questions rather than choosing a size from a generic rule of thumb.

Add metadata that helps identify the source, version, section, language and access scope. Keep metadata fields useful for filtering and citations, and avoid embedding values that change for every chunk unless they are needed. A chunk should carry enough surrounding context to make its meaning clear when retrieved on its own. For structured manuals or policies, retain a path such as document title → section → subsection so an answer can link to a readable location rather than a contextless passage.

Combine exact-term and semantic retrieval when the corpus needs both

Vector retrieval can find conceptually similar passages even when the wording differs, while full-text search can be more precise for product codes, names, dates and specialised terms. A hybrid search runs text and vector retrieval together and merges their results; whether that improves a particular RAG system must be tested against its own queries and corpus. Consider filters for document type, effective date, language or business unit when those fields are reliable and maintained.

Tune the candidate count, ranking and filters with examples that expose different failure modes. Search for an exact policy phrase, a paraphrase, a product identifier and a question whose answer is spread across two sections. If a relevant passage exists but is consistently ranked below a similar obsolete one, review metadata, version handling and ranking before changing the language model. Record the retrieval configuration with evaluation results so a cost or quality change can be attributed to a real change.

Enforce document access before context reaches the model

Document-level access must be applied to retrieval for each user's identity or approved group. A prompt instruction such as 'do not reveal confidential information' does not replace a query-time access filter. Test with users who have different permissions and confirm that restricted documents are not returned in results, citations, summaries or cached conversation context. Verify that permission changes propagate to the index within the business's required window.

Treat security trimming as an end-to-end requirement: identity mapping, source ACL ingestion, filter construction, query path, result rendering and logs. If the search provider's filter mechanism does not itself authenticate a user, the application must establish the identity and build the filter from a trusted source; never accept arbitrary group IDs from the client. Keep a documented negative test that attempts to retrieve a known restricted file and confirms no sensitive content is exposed.

Trace one question from retrieval to cited answer

For a policy assistant, take a question such as 'Can I claim a taxi after a late client meeting?' The retrieval layer should find the current travel policy section that covers eligibility and any exception, preserve its title and effective date, and return both passages if the rule depends on a related definition. The answer should cite those sources, state the documented limit and identify what is not specified. If no current source supports the answer, it should say that the material does not establish a rule and route the question to the policy owner.

This trace makes a RAG issue easier to diagnose. If the right passage was never ingested, investigate parsing or source coverage. If it exists but was not retrieved, examine chunk boundaries, filters and ranking. If the right evidence reached the model but the answer contradicts it, review the prompt, context formatting and generation behavior. Keep source excerpts available to an authorised evaluator so each stage can be inspected independently.

Evaluate retrieval separately from generated answers

Create a labelled set of representative queries with the expected source documents or passages, including questions with no supported answer, conflicting versions, restricted content, exact identifiers and paraphrases. Score retrieval using measures such as precision or recall at a chosen cutoff and inspect whether the right evidence appears near the top. Then evaluate the answer for groundedness, relevance, completeness, citation correctness and appropriate abstention. A fluent final response can hide a retrieval defect, so do not collapse both stages into a single satisfaction score.

Run comparisons when changing chunking, embeddings, hybrid search, filters or reranking, and keep the test set stable enough to compare results. Sample domain experts' judgments for ambiguous cases and record the reason for each accepted or rejected result. Include latency and the number of retrieved passages: returning more context may improve coverage but increase token use and can distract the model. Microsoft's RAG evaluation guidance also separates document retrieval quality from response groundedness and relevance; use these as categories for your own acceptance criteria rather than relying on a provider's default threshold.

Build a cost model around indexed content and answered questions

Separate initial setup from ongoing operation. A simple model can be written as: initial cost = source profiling + connectors + parsing and cleanup + metadata/ACL design + index build + test and acceptance. Monthly cost = changed content processed + storage and search + query embeddings and model calls + monitoring + support + human review. The actual billing units depend on the selected services, so record the provider, region, plan and measured assumptions rather than applying a universal price per document.

Estimate low, expected and high cases using both corpus changes and query volume. A large initial backfill may dominate setup, while a frequently changing knowledge base can make re-indexing and embedding the main recurring cost. Track cost per successfully answered query alongside abstentions, correction rate and review time. A cheaper configuration is not a saving if people must repeatedly search for the right source or correct unsupported answers. Refresh provider prices before procurement and include exit costs such as index export, source cleanup and migration.

Sources

Pinecone — Pricing

Microsoft Learn — Chunk large documents for RAG and vector search

Microsoft Learn — Hybrid search using vectors and full-text search

Microsoft Learn — Security filters for trimming search results

Microsoft Learn — RAG evaluators for retrieval and grounded answers

Microsoft Learn — RAG solution evaluation phase

Discuss your workflow