Evaluating retrieval context quality using the Ragas framework

"Our RAG demo looks good" is not a metric — it's an anecdote from whichever queries you happened to try. Ragas exists to replace that anecdote with a repeatable score: run a fixed set of test questions through your pipeline, and get faithfulness, context precision, context recall, and answer relevancy scores you can compare across changes to your chunking, retrieval, or prompt.

Running a Ragas evaluation

1. Build a test set of questions. Ideally each has a reference ("ground truth") answer, so context recall can be measured against what should have been retrieved.

2. Run each question through your actual RAG pipeline and capture both the retrieved contexts and the generated answer — Ragas scores need all of this, not just the final answer text.

3. Score the results against the four core metrics and track them over time — a regression in context recall after a chunking change tells you exactly where to look, instead of just "the app feels worse."

# Evaluating a RAG pipeline's outputs against Ragas metrics
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision, context_recall, answer_relevancy
from datasets import Dataset

# One row per test question - your RAG pipeline produces these fields
eval_data = Dataset.from_dict({
    "question": ["What is our refund policy?"],
    "answer": [generated_answer],           # from your RAG pipeline
    "contexts": [retrieved_chunks],         # what your retriever returned
    "ground_truth": ["Refunds are processed within 14 days."],
})

results = evaluate(
    eval_data,
    metrics=[faithfulness, context_precision, context_recall, answer_relevancy],
)
print(results)

None of these four metrics tells the whole story alone — a pipeline can have perfect context recall (it retrieved everything needed) but still score poorly on faithfulness if the model ignores the retrieved context and answers from its own training data instead. Track all four together.