Skip to content

Building a RAG Test Set From Real Questions

How to collect, label and maintain the evaluation questions that tell you whether a RAG system actually works.

Editorial team 1 min read

The test set is the foundation of RAG improvement. Without it, every change is guesswork.

Sources of Questions

  • Real user questions from logs, search queries or support tickets (with personal data removed).
  • Questions written by subject-matter experts, including tricky ones.
  • Generated questions from documents, reviewed by people — useful for coverage, but often easier than real questions.

What to Record for Each Question

  • The question.
  • The expected answer or key facts.
  • The documents or passages that contain the answer.
  • Whether it should be answered at all.
  • Category and difficulty.

Coverage

Include common questions, rare but important ones, multi-part questions, ambiguous questions, questions whose answers changed recently, and questions outside scope.

Size

Start with 50–100 well-chosen questions; expand as you find failures. Keep a stable core for comparisons over time.

Keep It Honest

Don't tune prompts using the exact test questions as examples. Keep a hidden subset for final checks.

Maintain It

When documents change, update expected answers. Add every production failure you fix as a new test question.

Share It

A test set agreed with content owners and stakeholders defines what "good" means for everyone.

More in RAG

All RAG guides →
RAG Guide · 1 min

RAG Architecture: The Components End to End

A map of a complete retrieval-augmented generation system, from ingestion to answer, and what each component is responsible for.

RAG 1 min read 6 Dec 2025

RAG Guide · 2 min

Document Parsing for RAG

Turning PDFs, slides, HTML and scans into clean, structured text — the unglamorous step that decides RAG quality.

RAG 2 min read 5 Dec 2025

RAG Guide · 2 min

Chunk Size and Overlap Tuning

How to choose chunk size and overlap for retrieval by testing against real questions rather than guessing.

RAG 2 min read 4 Dec 2025

RAG Guide · 2 min

Hybrid Search for RAG

Combining keyword and vector search so RAG finds both exact terms and paraphrased meaning.

RAG 2 min read 3 Dec 2025