Production RAG Architecture with FastAPI
Short answer: separate ingestion from query-time retrieval, keep retrieval observable, and make the generated answer traceable to retrieved evidence.
Architecture
documents → ingestion → parsing → chunking → embeddings → vector index
↓
question → API → query embedding → retrieval → optional reranking → LLM → cited answerIngestion should be asynchronous
Parsing and embedding a large document should not block an HTTP request. A queue or workflow can process ingestion, record document status, and retry failed steps independently.
FastAPI responsibilities
- Validate requests and responses with typed schemas.
- Authenticate and authorize access to documents.
- Expose health and readiness endpoints.
- Stream generated responses when useful.
- Attach request IDs and latency measurements to traces.
Retrieval quality matters more than prompt tricks
A polished prompt cannot recover evidence that retrieval failed to supply. Evaluate retrieval separately from generation and inspect failure cases such as missing chunks, irrelevant chunks, duplicate chunks, and insufficient context.
Production checklist
- Version embedding and chunking configuration.
- Track retrieval scores and top-k results.
- Apply tenant and document-level authorization before returning context.
- Control token budgets and context growth.
- Cache safe repeated work.
- Record enough telemetry to reproduce failures without logging sensitive document contents.
Related work
I built a RAG-based PDF Q&A project using parsing, chunking, embeddings, vector retrieval, and FastAPI.