A trustworthy RAG system declines when its sources don't contain the answer. Many systems instead improvise, which is worse than silence.
Why It Matters
An invented answer delivered with citations to loosely related documents looks authoritative. Users act on it; trust collapses when they discover the error.
Prompting for Honest Refusals
- State explicitly that answering from general knowledge is not allowed.
- Provide the exact wording for "not found" responses and a next step, such as who to contact.
- Ask the model to check that the retrieved documents actually address the question before answering.
Retrieval Signals
Low relevance scores for all retrieved chunks suggest the collection doesn't cover the question. Use this signal to trigger a refusal or a clarifying question.
Partial Answers
When sources answer part of a question, the system should answer that part and clearly say what's missing.
Measure It
Include out-of-scope and unanswerable questions in your test set, and track:
- false answers: responding when it should decline;
- false refusals: declining when the answer was available.
Both matter; an assistant that refuses too often is useless.
Learn From Refusals
Log refused questions. Frequent ones reveal content gaps worth filling.