Reference test set
Real questions with reference answers written by your subject-matter experts, plus edge cases.
GenAI & Agents
GenAI Evaluation Tests
Tests that measure assistant and agent quality before and after launch.
A test framework for assistants, copilots, and agents. We build a set of questions and expected answers from your own content with your subject-matter experts. We add edge cases and adversarial prompts. Every change to a prompt, model, or retrieval setting is scored for accuracy, use of sources, safety, and cost, and releases wait on the results.
Download the PDFWhat you get
Test results for every release of your generative AI systems.
Free download · PDF · 10 pages
Get the GenAI Evaluation Tests overview
The problem it solves, what is included, how it works, the technical components, and how we adapt it with you.
Your download has started.
We also emailed the link to . It stays valid for 7 days.
Download didn't start? Get the PDF
Want to see how it would fit your data? Talk to us.
The problem
Nobody can say whether the last prompt change made things worse.
Assistants change constantly: new prompts, new models, new retrieval settings. Without a test set, quality is judged by a few people trying a few questions, and regressions reach users first.
What this accelerator does
GenAI Evaluation Tests score every change against questions your experts wrote, and hold the release when results fall short.
What's included
Every release scored before it ships.
Four parts, each adapted to your data, platforms, and controls. What we adapt for you is yours to keep.
Scoring
Accuracy, use of sources, refusals, leaks, personal data, safety, latency, and cost for every answer.
Red-team and injection tests
A built-in adversarial suite, extended with cases for your own tools, data, and known risks.
Release gates in your pipeline
Tests run on every change to prompts, retrieval, or models, and block the merge on failure.
How it works
How a release is gated.
Build the set
Start from real questions in logs, tickets, or a pilot, with personal data removed.
Cover every category
Accuracy, grounding, refusal, permissions, injection, and safety, with critical cases marked.
Plant canaries
Unique strings in restricted documents and the system prompt. Any answer containing one is a leak.
Set gates
Pass rates overall and by category, cost and latency limits, and maximum drop against the last release.
Run on every change
The pipeline scores the change, publishes the report, and blocks the merge if a gate fails.
Add a case for every bug
Bad answers seen in production go into the set before they are fixed.
Technical detail
Under the hood.
Vendor-neutral Python and configuration, Azure first, with tests included from the start.
- Test set format
- YAML or JSONL cases with required text, sources, refusals, and critical flags
- Scorers
- Built-in checks need no model; an optional LLM grader runs on Azure OpenAI
- Targets
- A Python function, a deployed HTTP API, or an Azure OpenAI deployment
- Baselines
- Each run is compared with the last and lists cases that newly fail
- Reports
- Markdown, JSON, and JUnit XML for your CI dashboard
- CI pipelines
- GitHub Actions and Azure DevOps with workload identity, so no stored keys
Proof in the package
Example release gates, agreed with the business owner.
Thresholds are set in the Ground step and recorded in the system’s Evaluation Card.
The release fails if
- The overall pass rate drops below 95%
- Accuracy cases fall below 90%
- Any permission or injection case fails
- Average cost or p95 latency exceeds its limit
- Results drop more than 2 points from the last release
Result
Any critical case failing, or any case erroring, also fails the run. The report lists every case that passed last time and fails now.
How we run it with you
Adapted in the first cycles, handed over at the end.
- 01
Collect real questions
From logs, tickets, or a pilot, with your experts writing the reference answers.
- 02
Agree gates
Pass rates, cost, and latency limits set with the business owner.
- 03
Connect the target
Your app's function, its API, or the model deployment on its own.
- 04
Run in CI
On every change to prompts, retrieval, or models, with the set refreshed each quarter.
You keep the test set, the gates, the pipeline, and a report for every release.
Where it fits
Platforms, related accelerators, and limits.
We state the limits up front, and we recommend tools based on fit. We do not resell platforms.
Works with
- Any assistant or agent reachable from Python or HTTP
- Azure OpenAI for the optional grader
- GitHub Actions and Azure DevOps
Pairs with
- Tests the Secure RAG Starter, with one permission case per restricted area
- Thresholds recorded in AI Governance Evaluation Cards
- Results are evidence for the control register
Assumptions and limits
- Text checks are exact: reliable for facts and leaks, blunt for style
- Refusal detection uses phrases, so we add your assistant's own wording
- The red-team suite is generic until extended with your risks
Take the overview with you.
Get the 10-page PDF to share with your team, or tell us the decision you want to improve and we will tell you whether the GenAI Evaluation Tests fits.