---
title: "Evaluations"
description: "Test a change to a Raíz text agent on real WhatsApp replies your team corrected, before the change reaches patients."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.getarbol.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

An evaluation checks a change to an agent before patients see it. When your team says one of the agent's WhatsApp replies was wrong, incomplete or could be simpler, that reply becomes an evaluation case. Before you change the agent's instructions, skills or model, Raíz can replay the cases you kept with the new version, without sending anything to anyone and without running any tool, and a judge model says whether each new reply does what your team expects. Evaluations cover text agents.

## Evaluation cases

A case is one completed WhatsApp reply of an agent (`replyMessageId` in `conversationId`) and, when it was recorded, the version of the agent that wrote it (`agentVersionId`). Its `source` says where it comes from:

| `source` | Where it comes from |
| --- | --- |
| `correction` | A person of your team wrote right after the reply. |
| `authored` | Your team recorded feedback on the reply. |
| `sample` | A sampled reply. |

[`raiz.evalCases.create`](/api/raiz/eval-cases/create/) records your team's feedback on a reply. It takes the `conversationId` and the agent's `replyMessageId`, a `feedbackKind` (`incorrect`, `incomplete` or `simpler`), an `expectation`, which says what the agent should have done as a short rubric, and a `preferredResponse`: the reply or procedure your team prefers. Don't write the contact's personal data in the expectation. Recording feedback contacts nobody and changes no live instructions, and repeating it returns the existing case.

```bash
curl https://api.getarbol.com/v1/raiz/eval-cases \
  --request POST \
  --header "Authorization: Bearer $ARBOL_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "conversationId": "cnv_01jd8x8d5e6f7g8h9j0k1m2n3p",
    "replyMessageId": "msg_01jd8xek1m2n3p4q5r6s7t8v9w",
    "feedbackKind": "incomplete",
    "expectation": "Offer the next available slot at another site when the requested site has none.",
    "preferredResponse": "The North site has no appointments this week. I can book you at the South site on Thursday at 9:00. Would that work?"
  }'
```

### Review cases

A case starts `open`. [`raiz.evalCases.update`](/api/raiz/eval-cases/update/) reviews it: `kept` means every candidate version must handle it, and `dismissed` means it isn't a case, such as when a person only continued after a handoff. The review can also rewrite the `expectation`, `preferredResponse` and `feedbackKind`, with `expectedVersion`. [`raiz.evalCases.list`](/api/raiz/eval-cases/list/) lists cases, newest first, by `agentId`, `status`, `source` and `replyMessageId`; read the conversation itself to judge one.

## Evaluation runs

[`raiz.evalRuns.create`](/api/raiz/eval-runs/create/) measures a candidate version of a text agent on its kept cases: its 50 newest, or those in `caseIds` (up to 50). The `candidate` holds what changes in what the model reads, `instructions`, `skills`, `integrationSkills`, `model` and `reasoningEffort`, and anything you leave out stays as the live agent has it. A run calls paid models, so it needs `"confirmCost": true`.

```bash
curl https://api.getarbol.com/v1/raiz/eval-runs \
  --request POST \
  --header "Authorization: Bearer $ARBOL_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "agentId": "agt_01jd8x5k2m7q9r3t6v8w0y4z1b",
    "candidate": { "model": "arbolito/balanced" },
    "confirmCost": true
  }'
```

For each case, the agent writes its reply again with the candidate. No integration tool runs, since each answers what it answered in the original conversation, and nothing is sent. A judge model then compares the new reply with what your team expects. An agent with no kept case to replay, or a case in `caseIds` that isn't one of its kept cases, answers `409 EVAL_CASES_UNAVAILABLE`.

### Read the results

Follow the run with [`raiz.evalRuns.get`](/api/raiz/eval-runs/get/). Its `status` is `queued`, then `running` while cases are replayed, then `completed` once every case has a result, or `failed` if it couldn't continue. `passed`, `failed` and `errors` count the cases, and each of its `results`, which appear as they are judged, has:

| Field | What it says |
| --- | --- |
| `verdict` | `passed`: the new reply does what your team expects. `failed`: it doesn't, or the candidate wrote no reply. `error`: the case couldn't be replayed or judged, and `errorCode` says why. |
| `reply` | The candidate's reply as the contact would have read it. |
| `escalated`, `handedOff` | Whether the candidate escalated the conversation to your team or handed it to a person. |
| `toolCalls` | The tools the candidate called, in order. None of them ran. |
| `reasons` | The judge's reasons. |

## Require a passing run

With `settings.evalGate` on a text agent, an update that changes what its model reads (`instructions`, `skills`, `integrationSkills`, `model` or `reasoningEffort`) must send `evalRunId`: a completed run of exactly that content in which every case passed. Otherwise [`raiz.agents.update`](/api/raiz/agents/update/) answers `409 EVAL_GATE_UNMET`. A run's `contentSha256` is what the gate compares with the update.

## Proposals from feedback

Instead of a candidate, a run can take `feedbackCaseId`, a kept correction. Raíz prepares guidance from it, returned as `proposal` and as the complete `proposedInstructions`, and tests it on the agent's newest kept cases, always including that one. Don't combine `feedbackCaseId` with `candidate` or `caseIds`; `confirmCost` is still required. Nothing is published: review the proposed instructions, then send them to `raiz.agents.update` with the run's `id` as `evalRunId` and the agent's `expectedVersion`.

## Permissions

Evaluations need an agents scope and a conversations scope:

| Operation | Scopes |
| --- | --- |
| `raiz.evalCases.create` | `raiz:conversations:write`, and `raiz:agents:read` or `raiz:agents:write` |
| `raiz.evalCases.list`, `raiz.evalRuns.get` | `raiz:agents:read` or `raiz:agents:write`, and `raiz:conversations:read` or `raiz:conversations:write` |
| `raiz.evalCases.update`, `raiz.evalRuns.create` | `raiz:agents:write`, and `raiz:conversations:read` or `raiz:conversations:write` |

Only `raiz.evalCases.create` is offered on [MCP](/en/developers/mcp/); the others are REST only.

Source: https://docs.getarbol.com/en/developers/raiz/evaluations/index.mdx