Reference test set
Real questions with answers written by your subject-matter experts, plus edge cases. Cases are YAML or JSONL and can require specific text, specific sources, or a refusal.
A test set from your experts that scores every assistant change before it ships.
Download the PDFA short PDF to share with your team. It covers what is in the package, how it works, and a worked example.
Your download has started.
We also emailed the link to . It works for 7 days.
Download didn't start? Get the PDF
Want to try it on your own data? Talk to us.
Prompts, models, and retrieval settings change all the time. Without a test set, quality gets judged by a few people trying a few questions, and users find the regressions first.
Real questions with answers written by your subject-matter experts, plus edge cases. Cases are YAML or JSONL and can require specific text, specific sources, or a refusal.
Accuracy, source use, refusals, leaks, personal data, safety, latency, and cost for every answer. Built-in checks need no model. An optional LLM grader runs on Azure OpenAI.
An adversarial suite we extend with cases for your own tools, data, and known risks.
Pipelines for GitHub Actions and Azure DevOps use workload identity, so no keys are stored. Each run is compared with the last, and reports come out as Markdown, JSON, and JUnit XML.
Start from real questions in logs, tickets, or a pilot, with personal data stripped out.
Accuracy, grounding, refusal, permissions, injection, and safety. Critical cases are marked.
Put unique strings in restricted documents and the system prompt. Any answer that contains one is a leak.
Pass rates overall and by category, limits on cost and latency, and the largest drop allowed against the last release.
The pipeline scores the change, publishes the report, and blocks the merge when a gate fails.
A bad answer seen in production goes into the set before it gets fixed.
Thresholds are set in the Ground step and written into the system’s Evaluation Card.
A failing critical case or a case that errors also fails the run. The report lists every case that passed last time and fails now.
Ask about the GenAI Evaluation Tests
Dex answers from our own pages and links to where it found the answer.
Send us a handful of real questions your assistant gets. We will show you how they become test cases.