When a RAG system gives a bad answer, the cause is usually one of a handful of recurring problems.
Retrieval Misses the Answer
Causes: poor parsing, chunks split at the wrong place, vocabulary mismatch, too few results retrieved, overly strict filters. Fixes: better parsing and chunking, hybrid search, query rewriting, higher k, re-ranking.
Right Document, Wrong Passage
Causes: chunks lacking context; many similar passages. Fixes: contextual chunks, metadata, re-ranking.
Answer Not Grounded
Causes: weak instructions; model filling gaps from general knowledge. Fixes: stricter grounding prompts, citations, faithfulness checks.
Conflicting Sources
Causes: multiple versions of a policy indexed. Fixes: remove superseded documents, prefer current versions, show dates.
Stale Content
Causes: index not updated or deletions missed. Fixes: incremental ingestion, deletion handling, freshness monitoring.
Out-of-Scope Questions Answered Anyway
Causes: no instruction to decline. Fixes: explicit refusal instructions and test cases.
Permission Leaks
Causes: missing or outdated access metadata. Fixes: retrieval-time filtering and automated access tests.
Diagnose Systematically
For each failure, check: was the right passage retrieved? Was it in the prompt? Did the model use it correctly? Logging retrieved chunks with every answer makes this possible.