TLDR Tech / Data
HNSW at Scale: Why Adding More Documents to Your Database Breaks RAG
When HNSW-based RAG systems exceed around 100K vectors, latency grows super linearly and recall drops, often returning highly similar but irrelevant results for rare queries. The causes include local minima traps, hubness in high dimensions, and memory pressure. Mitigations include tuning M, ef_construct, and ef_search, using hybrid two-stage retrieval, applying quantization with oversampling and rescoring, and relying on optimized engines like Qdrant.
Read the full story