How to Improve RAG Retrieval Quality
Short answer: treat retrieval as an independently measurable system. Improve the evidence reaching the model before trying to compensate with a larger prompt.
1. Start with chunking
Chunk boundaries should preserve meaning. Fixed-size chunks are simple, but headings, paragraphs, tables, and document structure can provide better boundaries for many corpora.
2. Tune top-k using evaluation data
More retrieved chunks do not automatically mean better answers. Large k can introduce noise and consume context. Test several values against representative questions.
3. Consider reranking
Vector similarity is a useful first-stage retrieval signal. A reranker can score the smaller candidate set using a more expensive relevance model before context reaches the LLM.
4. Measure retrieval separately
Create a small evaluation set containing realistic questions and expected evidence. Track whether the correct source appears in the retrieved set, then separately measure answer quality.
5. Analyze failures
- Correct chunk not indexed.
- Chunk too small or too large.
- Query wording differs from source wording.
- Similar but incorrect chunks outrank the correct evidence.
- Relevant context is retrieved but truncated by the model context budget.
Practical loop
baseline → evaluation set → inspect failures → change one variable → re-evaluate → document result