Evals

A scripted conversation, graded automatically, run against any agent — a way to check a prompt still behaves before a real caller finds out it doesn't.

How a turn works

An eval is a list of turns. Each turn is what the caller says, plus — optionally — what a passing reply looks like: an exact string, a regex, a rubric an AI judge checks the reply against, or that the agent should have called a particular tool with particular arguments. A turn with no check attached still runs and shows up in the transcript; it just doesn’t affect pass or fail.

Grading a reply

Exact — the reply must match a given string (case and whitespace-insensitive).
Regex — the reply must match a given pattern.
AI judge — a rubric in plain English (e.g. “offers to book a slot and asks for a name”), graded by a second model call.

A turn can also assert that a specific during_call tool was called, optionally with specific arguments — independent of, or alongside, a reply check.

Running one

Pick an agent and run — the eval replays the script against that agent’s real, current prompt and model, turn by turn, and reports pass/fail per turn plus the reason.

Run an eval

curl -X POST https://api.voicelo.ai/api/evals/6511c.../run \
  -H "x-api-key: $VOICELO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "agentId": "6511f..." }'

Careful

If a turn expects a tool call, the tool is detected and graded but not actually executed by default — the same reasoning as the in-browser agent tester: a script that fires a real CRM webhook on every run is a trap, not a feature. Set execute: true on that turn’s expected tool call to run it for real, against a staging endpoint built for that.

Note

An agent on Deepgram’s managed model (no OpenRouter model or key set) has no standalone chat endpoint to replay outside a live voice session — evals need an OpenRouter key (or a Sarvam key, for a Sarvam-engine agent) to run.

PreviousWebhooks NextAPI reference