---
title: "GenAI Evaluation Tests Accelerator — Datagist"
description: "A test set written with your experts that scores every prompt, model, or retrieval change for accuracy, leaks, and cost, and blocks releases that fall short."
url: https://datagist.ai/accelerators/genai-evaluation
---

[Accelerators](https://datagist.ai/accelerators) GenAI Evaluation Tests

[GenAI & Agents](https://datagist.ai/services/genai-agents) 

# GenAI Evaluation Tests

A test set from your experts that scores every assistant change before it ships.

GIST step [Iterate](https://datagist.ai/approach#iterate) [Ship](https://datagist.ai/approach#ship) [Trust](https://datagist.ai/approach#trust)

 Download the PDF 

## Get the GenAI Evaluation Tests overview

A short PDF to share with your team. It covers what is in the package, how it works, and a worked example.

On this page The problem What's included How it works Worked example Where it fits

The problem 

## Nobody can say whether the last prompt change made things worse.

Prompts, models, and retrieval settings change all the time. Without a test set, quality gets judged by a few people trying a few questions, and users find the regressions first.

What's included 

## Every release scored before it ships.

01 

### Reference test set

Real questions with answers written by your subject-matter experts, plus edge cases. Cases are YAML or JSONL and can require specific text, specific sources, or a refusal.

02 

### Scorers

Accuracy, source use, refusals, leaks, personal data, safety, latency, and cost for every answer. Built-in checks need no model. An optional LLM grader runs on Azure OpenAI.

03 

### Red-team and injection tests

An adversarial suite we extend with cases for your own tools, data, and known risks.

04 

### Release gates in CI

Pipelines for GitHub Actions and Azure DevOps use workload identity, so no keys are stored. Each run is compared with the last, and reports come out as Markdown, JSON, and JUnit XML.

How it works 

## How a release is gated.

1. ### Build the set  
Start from real questions in logs, tickets, or a pilot, with personal data stripped out.
2. ### Cover the categories  
Accuracy, grounding, refusal, permissions, injection, and safety. Critical cases are marked.
3. ### Plant canaries  
Put unique strings in restricted documents and the system prompt. Any answer that contains one is a leak.
4. ### Set gates  
Pass rates overall and by category, limits on cost and latency, and the largest drop allowed against the last release.
5. ### Run on every change  
The pipeline scores the change, publishes the report, and blocks the merge when a gate fails.
6. ### Add a case for every bug  
A bad answer seen in production goes into the set before it gets fixed.

Worked example 

## Release gates agreed with the business owner.

Thresholds are set in the Ground step and written into the system’s Evaluation Card.

The release fails if 
- The overall pass rate drops below 95%
- Accuracy cases fall below 90%
- Any permission or injection case fails
- Average cost or p95 latency goes over its limit
- Results drop more than 2 points from the last release

Result 

A failing critical case or a case that errors also fails the run. The report lists every case that passed last time and fails now.

Where it fits 

## What it runs on, and where it stops.

### Works with

- Any assistant or agent you can call from Python or HTTP, including an Azure OpenAI deployment on its own
- Azure OpenAI for the optional grader
- GitHub Actions and Azure DevOps

### Assumptions and limits

- Text checks are exact. That suits facts and leaks and is blunt for tone and style
- Refusal detection matches phrases, so we add your assistant’s own wording
- The red-team suite stays generic until we add your risks

Related accelerators 

[Secure RAG Starter Gets one permission test for each restricted area. View details of the Secure RAG Starter](https://datagist.ai/accelerators/secure-rag) [AI Governance Starter Records the thresholds in each Evaluation Card and keeps results as evidence. View details of the AI Governance Starter](https://datagist.ai/accelerators/ai-governance) [AI Agent Controls Agent behavior is tested here before release. View details of the AI Agent Controls](https://datagist.ai/accelerators/agent-controls)

## Changing prompts without a safety net?

Send us a handful of real questions your assistant gets. We will show you how they become test cases.

Download the PDF [Talk to us ](https://datagist.ai/contact)
