Evaluations

An evaluation checks a change to an agent before patients see it. When your team says one of the agent’s WhatsApp replies was wrong, incomplete or could be simpler, that reply becomes an evaluation case. Before you change the agent’s instructions, skills or model, Raíz can replay the cases you kept with the new version, without sending anything to anyone and without running any tool, and a judge model says whether each new reply does what your team expects. Evaluations cover text agents.

Evaluation cases

A case is one completed WhatsApp reply of an agent (replyMessageId in conversationId) and, when it was recorded, the version of the agent that wrote it (agentVersionId). Its source says where it comes from:

raiz.evalCases.create records your team’s feedback on a reply. It takes the conversationId and the agent’s replyMessageId, a feedbackKind (incorrect, incomplete or simpler), an expectation, which says what the agent should have done as a short rubric, and a preferredResponse: the reply or procedure your team prefers. Don’t write the contact’s personal data in the expectation. Recording feedback contacts nobody and changes no live instructions, and repeating it returns the existing case.

curl https://api.getarbol.com/v1/raiz/eval-cases \
  --request POST \
  --header "Authorization: Bearer $ARBOL_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "conversationId": "cnv_01jd8x8d5e6f7g8h9j0k1m2n3p",
    "replyMessageId": "msg_01jd8xek1m2n3p4q5r6s7t8v9w",
    "feedbackKind": "incomplete",
    "expectation": "Offer the next available slot at another site when the requested site has none.",
    "preferredResponse": "The North site has no appointments this week. I can book you at the South site on Thursday at 9:00. Would that work?"
  }'

Review cases

A case starts open. raiz.evalCases.update reviews it: kept means every candidate version must handle it, and dismissed means it isn’t a case, such as when a person only continued after a handoff. The review can also rewrite the expectation, preferredResponse and feedbackKind, with expectedVersion. raiz.evalCases.list lists cases, newest first, by agentId, status, source and replyMessageId; read the conversation itself to judge one.

Evaluation runs

raiz.evalRuns.create measures a candidate version of a text agent on its kept cases: its 50 newest, or those in caseIds (up to 50). The candidate holds what changes in what the model reads, instructions, skills, integrationSkills, model and reasoningEffort, and anything you leave out stays as the live agent has it. A run calls paid models, so it needs "confirmCost": true.

curl https://api.getarbol.com/v1/raiz/eval-runs \
  --request POST \
  --header "Authorization: Bearer $ARBOL_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "agentId": "agt_01jd8x5k2m7q9r3t6v8w0y4z1b",
    "candidate": { "model": "arbolito/balanced" },
    "confirmCost": true
  }'

For each case, the agent writes its reply again with the candidate. No integration tool runs, since each answers what it answered in the original conversation, and nothing is sent. A judge model then compares the new reply with what your team expects. An agent with no kept case to replay, or a case in caseIds that isn’t one of its kept cases, answers 409 EVAL_CASES_UNAVAILABLE.

Read the results

Follow the run with raiz.evalRuns.get. Its status is queued, then running while cases are replayed, then completed once every case has a result, or failed if it couldn’t continue. passed, failed and errors count the cases, and each of its results, which appear as they are judged, has:

Require a passing run

With settings.evalGate on a text agent, an update that changes what its model reads (instructions, skills, integrationSkills, model or reasoningEffort) must send evalRunId: a completed run of exactly that content in which every case passed. Otherwise raiz.agents.update answers 409 EVAL_GATE_UNMET. A run’s contentSha256 is what the gate compares with the update.

Proposals from feedback

Instead of a candidate, a run can take feedbackCaseId, a kept correction. Raíz prepares guidance from it, returned as proposal and as the complete proposedInstructions, and tests it on the agent’s newest kept cases, always including that one. Don’t combine feedbackCaseId with candidate or caseIds; confirmCost is still required. Nothing is published: review the proposed instructions, then send them to raiz.agents.update with the run’s id as evalRunId and the agent’s expectedVersion.

Permissions

Evaluations need an agents scope and a conversations scope:

Only raiz.evalCases.create is offered on MCP; the others are REST only.