B2Shift · Published 27 August 2026 · Updated 27 August 2026
Retrieval-augmented generation (RAG) lets a system answer from your documents instead of a model's general training data, with a citation back to the source. The idea is simple; the cost variance in practice comes from a small number of decisions made early.
What a RAG pipeline actually does
Documents are split into chunks, converted into vector embeddings, and stored in a vector database. A question is embedded the same way, the closest matching chunks are retrieved, and the model answers using only those chunks as context — with a citation back to the source document. Remove the citation step and you have a system nobody can audit when it's wrong.
What drives the cost more than model choice
Four things move the number more than which language model you pick. Document volume and format: scanned PDFs need OCR before they can be chunked, and OCR errors propagate into every answer downstream. Chunking strategy: naive fixed-length chunking is cheap to build and frequently wrong on structured documents like contracts or tables. Retrieval quality: a vector database has real infrastructure and query costs at scale that need tuning, not just installation. And evaluation: without a test set of real questions and known-correct answers, you cannot tell if the system is accurate — you can only tell if it sounds confident, which is a different thing entirely.
Where accuracy actually breaks
Poor-quality source documents are the most common failure, not model limitations — a RAG system cannot answer correctly from a document that is itself wrong or ambiguous. Tables, scanned handwriting and multi-column layouts need special handling; naive extraction silently drops or scrambles them. And permission-aware retrieval matters as much as accuracy: a system that can retrieve a document a user shouldn't see is a security bug, not an accuracy bug.
What a realistic build includes
A production RAG pipeline needs a defined confidence threshold below which the system says "I don't have a reliable answer" instead of guessing, a review queue for low-confidence cases, and an accuracy report measured against your own test set — not a generic benchmark, because your documents are not the benchmark's documents. See Document Automation & RAG for how we scope this.