Handling multiple files in a RAG pipeline
August 8, 2026
A RAG demo with one PDF almost always works. The moment you point the same
pipeline at fifty documents, answers start blending content from files that
have nothing to do with each other. Here's what actually breaks, and what
fixes it.
What goes wrong with naive multi-file RAG
If you just dump every document into one vector index with no structure,
retrieval has no way to know that a chunk from refund-policy.pdf and a
chunk from 2019-audit.pdf shouldn't be blended into the same answer. Top-k
similarity search doesn't care which file a chunk came from — it just finds
the closest vectors, wherever they live.
Two documents that happen to use similar wording (a policy revised year over
year, for example) will retrieve interchangeably, and the model may quietly
mix an old policy with a new one in the same answer.
Fixes that actually help
Attach metadata to every chunk, not just text. Store source_file,
page_number, and ideally a document_date or version alongside each
embedding. Retrieval can then filter or boost by metadata before ranking —
"only search documents from this category" or "prefer the most recent
version" — instead of relying purely on semantic distance.
Chunk per document, not across the whole corpus. Chunking boundaries
should never cross a file boundary. It sounds obvious, but it's an easy bug
when documents are concatenated before splitting.
Attribute every answer to its source. Return the source filename (and
ideally page number) alongside the generated answer. This isn't just a UX
nicety — it's how you catch the model quietly blending two documents,
because the citations won't make sense together.
**Consider per-document or per-category indexes for genuinely distinct
corpora.** If you have HR policies and engineering runbooks in the same
system, a single flat index will surface irrelevant cross-domain matches.
Splitting into separate collections (or using metadata filtering as a hard
pre-filter, not just a re-ranking signal) keeps retrieval scoped to the
right domain.
Re-rank after retrieval, don't trust raw similarity alone. A cheap
cross-encoder re-ranking pass on the top ~20 candidates catches cases where
semantic similarity picked something topically close but contextually wrong.
None of this is exotic — it's the difference between a RAG demo and a RAG
system that survives more than one document in the index.