Your AI assistant can find the right answer in a small demo, but RAG at scale is a different problem once you connect the full company knowledge base.
Now it has old policies, near-identical contracts, scanned reports, and documents that different employees are allowed to access. A passage can sound relevant while belonging to the wrong customer or year.
To improve RAG accuracy at scale, first identify where the evidence gets lost. Then test retrieval methods against a fixed set of real questions, with response-time and cost limits. A better embedding model can help, but it cannot repair missing source content or decide which policy version applies on its own.
RAG, or retrieval-augmented generation, finds source material and gives it to a model before the model answers. RAG at scale requires both finding useful evidence and using that evidence correctly.
In my work on BioSonic, an acoustic analysis platform, we dealt with millions of audio files and terabytes of data. Part of that work involved moving heavy analytics queries to ClickHouse. It was an analytics and data-delivery problem, not a document-retrieval benchmark. It is why I ask which operation is slow before recommending a larger model or another database.
The comparisons below combine that engineering perspective with published research and product documentation, checked November 5, 2026. They are recommendations to test, not results from a benchmark I ran.
Why can accuracy fall when you run RAG at scale?
A larger collection can introduce more competing passages, duplicate content, conflicting versions, and varied document formats. The relevant evidence can become harder to retrieve or select. Growth does not automatically make a well-designed system less accurate.
However, document count alone is a poor description of scale. A million short records and a million technical manuals produce very different workloads. Record the number of searchable chunks and vectors, document lengths, update frequency, filter patterns, and concurrent requests.
Then separate these failures:
| What went wrong | What to inspect first |
|---|---|
| The answer-bearing passage was never indexed | Parsing, OCR, table extraction, ingestion failures |
| It was indexed but never retrieved | Query interpretation, chunk representation, lexical coverage, filters, approximate-search settings |
| It reached the candidates but was ranked too low | Rank fusion, duplicates, reranker quality |
| It reached the model but the answer was wrong | Missing surrounding context, conflicting versions, unsupported reasoning |
| It was correct but too slow | Time spent in each stage, queueing, disk reads, provider calls, generation |
A reranker cannot recover evidence that never entered its candidate set. A faster index cannot reconstruct a table that the parser discarded.
BM25 vs semantic search vs hybrid search: what works when?
BM25 ranks text using lexical signals such as matching query terms. Dense semantic search compares learned vector representations, which helps when the question and source use different wording. Hybrid retrieval combines the two result lists. Elasticsearch documents this approach and recommends reciprocal rank fusion, or RRF, for combining rankings. Elastic hybrid search documentation
My starting comparison would look like this:
| Question or collection | Approach I would test first | Failure to watch for |
|---|---|---|
| A known invoice number or product identifier | Exact field lookup or filter | Text analysis changing the identifier |
| An error code with descriptive text | BM25, compared with hybrid retrieval | Missing a useful paraphrase |
| A natural-language policy question | Dense retrieval, compared with hybrid retrieval | A similar policy for the wrong region |
| Mixed terminology across a company knowledge base | Hybrid retrieval | Irrelevant candidates from either branch |
| A numerical total across business records | A structured query or calculation over validated data | Treating a plausible passage as a computed answer |
These are starting points for RAG at scale, not fixed rules. Hybrid search should earn its extra work by improving the evaluated result. For a narrow identifier lookup, an exact query may already solve the problem.
Keep retrieval branches under the same authorization rules. Otherwise, combining their results can introduce material the user should never receive. I cover access control in more detail in my guide to local AI for business automation.
Preserve document structure before changing embeddings
For example, consider a hypothetical policy excerpt: “The limit is 30 days.”
It is hard to use correctly without knowing what the limit applies to, which region the policy covers, and when it became effective. Preserve headings, document identity, dates, page references, and useful surrounding text during ingestion.
Contextual retrieval adds explanatory text to chunks before indexing. Anthropic’s published experiments reported a 67% reduction in top-20 retrieval failures when contextual embeddings, contextual BM25, and reranking were combined. That is a relative reduction in a retrieval metric on their evaluation, not a 67-point increase in answer accuracy. Anthropic contextual retrieval research
Late chunking addresses context loss differently: a compatible long-context embedding model processes tokens together before the system pools representations into chunk vectors. It still needs chunk boundaries and is constrained by the model’s input window. Jina’s late chunking explanation
For RAG at scale, both deserve consideration when isolated passages lose meaning. However, neither removes the need to validate extracted content or maintain document versions.
How do you reduce vector-search latency in RAG at scale?
Large collections often use approximate nearest-neighbor search, or ANN, to avoid comparing every query against every vector. The engineering question is how much search work is needed to retain the evidence your application requires.
Measure two different kinds of recall. ANN recall measures agreement with exact vector search. Evidence recall measures whether the retrieved passages actually contain what a person needs to answer the question. An index can reproduce vector neighbors accurately while the embedding representation still favors the wrong evidence.
Tune candidate-search breadth against both measures. Include filters in the test: restricting a graph search by metadata can change its behavior. Elastic’s filtered kNN documentation describes ACORN-based traversal and cases where the system instead scores matching documents directly. Filtered kNN documentation
Compression is also an option. Quantization stores vectors at lower precision to reduce memory use. Oversampling retrieves a wider pool, and rescoring checks candidates using the original vectors. This can recover lost ranking quality, with additional work and potentially disk reads. The balance depends on the collection and storage configuration. Qdrant quantization documentation
When running RAG at scale, I would also check shard distribution, hot partitions, replication, and whether updates compete with queries for resources. Adding shards creates coordination work as well as capacity; benchmark the deployed configuration.
What do Google’s newer approaches change?
Google’s work relevant to RAG at scale spans different parts of the system. It helps to name the layer before comparing the claims.
| Development | What it changes | When I would investigate it |
|---|---|---|
| SOAR with ScaNN | Approximate vector indexing | The search-efficiency and recall tradeoff is the bottleneck |
| TurboQuant | Vector compression | Vector memory requirements constrain the system |
| Gemini Embedding 2 | Multimodal representations | Useful evidence lives in images, audio, video, or visually structured documents |
Google described SOAR in April 2024, so it should not be presented as a new 2026 release. It introduces controlled redundancy into ScaNN’s index to improve search efficiency. Google’s SOAR explanation
Google’s March 2026 TurboQuant announcement covers compression for vector search and model key-value caches. Cache-efficiency results are not measurements of end-to-end RAG response time. Qdrant now documents a TurboQuant implementation, providing evidence of product adoption rather than a reason to assume identical results everywhere. Google TurboQuant research, Qdrant implementation documentation
Gemini Embedding 2 became generally available in April 2026. It represents multiple media types in a shared embedding space. That makes it relevant when flattening a document into text loses evidence, but it does not replace index tuning or answer evaluation. Google’s availability announcement
When should you add reranking, multivector retrieval, or GraphRAG?
Add a reranker when relevant evidence reaches the candidate list but the ordering is weak. It evaluates candidates against the question more closely. Cost and latency depend on model choice, text length, and the number of candidates. Reranking documentation
As an illustrative experiment, retrieve 100 candidates and compare reranking 20, 50, and 100. Measure which setting supplies sufficient evidence within the response-time budget. Those counts are experiment settings, not recommended defaults for every application.
Multivector models retain more detailed representations rather than reducing a passage or page to one vector. For visually complex PDFs, ColPali-style retrieval can preserve information associated with page appearance. Qdrant provides a documented PDF-retrieval workflow using ColPali and ColQwen models. The added vectors require storage and search planning. Qdrant multivector PDF tutorial
GraphRAG is worth investigating when questions depend on relationships or themes across a collection. Microsoft documents both the research and its availability through Microsoft Discovery. It adds a different knowledge representation, rather than simply speeding up ordinary passage search. Microsoft GraphRAG project
I would reserve repeated, agent-driven searches for questions that benefit from them. Each additional search or model call needs to justify its contribution to the answer and its share of the latency budget.
A practical pipeline for benchmarking RAG at scale
For a mixed business-document collection, I would compare a hybrid pipeline against simpler lexical and dense baselines:
- Establish the user’s permitted scope and applicable document versions.
- Route exact lookups or calculations to structured queries where appropriate.
- Retrieve keyword and vector candidates for document questions.
- Merge rankings and remove redundant passages without discarding necessary evidence.
- Rerank a bounded candidate set if evaluation supports it.
- Select enough evidence to answer, retaining source and page references.
- Generate an answer that cites supporting material and acknowledges missing evidence.
Move parsing, embedding, and other reusable preparation out of the request path where possible. Measure the indexing cost and update lag separately.
Caching can reduce repeated work, but cache keys and invalidation must account for permissions, document changes, and configuration. A cached answer to last month’s policy question may be fast and wrong today.
How should you measure RAG accuracy and speed at scale?
Build a reviewed question set containing exact identifiers, paraphrases, tables, conflicting versions, access restrictions, and questions the collection cannot answer. Include questions needing evidence from several passages.
Run the same questions against each configuration and keep a separate set for final validation. For each question, identify acceptable evidence and what a supported answer must contain.
| Measure | What it tells you |
|---|---|
| Evidence recall@k | Whether the first k results contain the required evidence |
| Ranking quality | Whether useful evidence appears early |
| Answer correctness and citation support | Whether the response is right and its claims are supported |
| Appropriate abstention | Whether the assistant recognizes insufficient evidence |
| Retrieval, reranking, and generation times | Which stage consumes the budget |
| End-to-end p50 and p95 latency | Typical and slower user experiences |
| Cost per query, memory, and index size | Resource requirements at the desired quality |
| Update lag and throughput under load | Whether fresh data and concurrent users change the result |
Report corpus size, chunk count, hardware, cache state, filter patterns, and concurrency alongside results. For example, a warm-cache single-user test is not evidence of performance during a busy production period.
My preference is to choose the configuration that meets an explicit answer-quality target within the business’s response-time and cost limits. If several configurations qualify, the simpler one is easier to operate and explain.
Common questions about large-scale document retrieval
Does a larger embedding model always improve RAG accuracy?
No. It may improve representation quality, but ingestion errors, unsuitable chunking, missing filters, and incorrect context selection can remain. Compare its impact on supported answers as well as retrieval scores and operating cost.
Can increasing top-k fix missing answers?
It can expose evidence that ranked below the previous cutoff. It also increases downstream work and may introduce distracting passages. Test retrieval breadth separately from the amount of context supplied to the model.
What is the best retrieval approach for millions of documents?
There is no universal winner for RAG at scale. Exact lookups, policy questions, scanned drawings, and collection-wide analysis require different capabilities. Benchmark appropriate baselines and add complexity where the observed failures justify it.
Can fast retrieval guarantee an accurate final answer?
No. The model can still misread evidence or combine incompatible sources. Evaluate the complete response, including citation support and whether the system should have declined to answer.
If your assistant becomes slower or less dependable as the collection grows, I can help trace where evidence is lost and which stage consumes the response time. Get in touch with me at TRS.
