> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getarbol.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create an Agent evaluation suite

> Creates one suite under agent-evaluations:write; recorded and tool-bearing cases require their additional evidence permissions, mocked tools require exact fixtures, and AI rubrics must begin advisory. Use the human-only promotion operation after golden-set calibration. No idempotency key exists, so confirm an uncertain response before retrying.



## OpenAPI

````yaml https://api.getarbol.com/openapi.json post /agents/{agentId}/evaluation-suites
openapi: 3.1.2
info:
  contact:
    email: support@getarbol.com
    name: Arbol
    url: https://docs.getarbol.com
  description: >-
    Provider-neutral API for creating and operating Arbol agents, contacts,
    conversations, channels, knowledge, integrations, and evaluations. Every
    request is bound to one Auth0 Organization.
  title: Arbol API
  version: 1.0.0
servers:
  - description: Production
    url: https://api.getarbol.com/v1
security: []
paths:
  /agents/{agentId}/evaluation-suites:
    post:
      tags:
        - Agent evaluations
      summary: Create an Agent evaluation suite
      description: >-
        Creates one suite under agent-evaluations:write; recorded and
        tool-bearing cases require their additional evidence permissions, mocked
        tools require exact fixtures, and AI rubrics must begin advisory. Use
        the human-only promotion operation after golden-set calibration. No
        idempotency key exists, so confirm an uncertain response before
        retrying.
      operationId: createAgentEvaluationSuite
      parameters:
        - in: path
          required: true
          name: agentId
          schema:
            $ref: '#/components/schemas/ArbolResourceId'
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                cases:
                  maxItems: 50
                  type: array
                  items:
                    type: object
                    properties:
                      channelKind:
                        $ref: '#/components/schemas/RuntimeChannelKind'
                        default: web
                      fixtures:
                        default: []
                        maxItems: 100
                        type: array
                        items:
                          type: object
                          properties:
                            calls:
                              maxItems: 20
                              type: array
                              items:
                                type: object
                                properties:
                                  input:
                                    $ref: '#/components/schemas/__schema0'
                                  output:
                                    $ref: '#/components/schemas/__schema0'
                                  stateAfter:
                                    $ref: '#/components/schemas/__schema0'
                                required:
                                  - output
                                additionalProperties: false
                            connectedAccountId:
                              $ref: '#/components/schemas/ArbolResourceId'
                            initialState:
                              $ref: '#/components/schemas/__schema0'
                            toolKey:
                              $ref: '#/components/schemas/ToolKey'
                          required:
                            - calls
                            - connectedAccountId
                            - toolKey
                          additionalProperties: false
                      id:
                        type: string
                        minLength: 1
                        maxLength: 100
                        pattern: ^[a-z][a-z0-9_.-]*$
                      name:
                        type: string
                        minLength: 1
                        maxLength: 200
                      scenario:
                        oneOf:
                          - type: object
                            properties:
                              messages:
                                minItems: 1
                                maxItems: 50
                                type: array
                                items:
                                  type: object
                                  properties:
                                    content:
                                      type: string
                                      minLength: 1
                                      maxLength: 16000
                                    role:
                                      type: string
                                      enum:
                                        - assistant
                                        - user
                                  required:
                                    - content
                                    - role
                                  additionalProperties: false
                              type:
                                type: string
                                const: nextReply
                            required:
                              - messages
                              - type
                            additionalProperties: false
                          - type: object
                            properties:
                              messages:
                                minItems: 1
                                maxItems: 50
                                type: array
                                items:
                                  type: object
                                  properties:
                                    content:
                                      type: string
                                      minLength: 1
                                      maxLength: 16000
                                    role:
                                      type: string
                                      enum:
                                        - assistant
                                        - user
                                  required:
                                    - content
                                    - role
                                  additionalProperties: false
                              type:
                                type: string
                                const: toolCall
                            required:
                              - messages
                              - type
                            additionalProperties: false
                          - type: object
                            properties:
                              initialMessages:
                                default: []
                                maxItems: 20
                                type: array
                                items:
                                  type: object
                                  properties:
                                    content:
                                      type: string
                                      minLength: 1
                                      maxLength: 16000
                                    role:
                                      type: string
                                      enum:
                                        - assistant
                                        - user
                                  required:
                                    - content
                                    - role
                                  additionalProperties: false
                              maxTurns:
                                default: 5
                                type: integer
                                minimum: 1
                                maximum: 20
                              type:
                                type: string
                                const: simulation
                              userGoal:
                                type: string
                                minLength: 1
                                maxLength: 4000
                            required:
                              - type
                              - userGoal
                            additionalProperties: false
                          - type: object
                            properties:
                              conversationId:
                                $ref: '#/components/schemas/ArbolResourceId'
                              type:
                                type: string
                                const: recordedConversation
                            required:
                              - conversationId
                              - type
                            additionalProperties: false
                      scorers:
                        minItems: 1
                        maxItems: 30
                        type: array
                        items:
                          oneOf:
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                caseSensitive:
                                  default: true
                                  type: boolean
                                expected:
                                  type: string
                                  maxLength: 16000
                                type:
                                  type: string
                                  const: textEquals
                              required:
                                - key
                                - expected
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                caseSensitive:
                                  default: false
                                  type: boolean
                                expected:
                                  type: string
                                  minLength: 1
                                  maxLength: 4000
                                type:
                                  type: string
                                  const: textContains
                              required:
                                - key
                                - expected
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                expected:
                                  $ref: '#/components/schemas/__schema0'
                                type:
                                  type: string
                                  const: jsonEquals
                              required:
                                - key
                                - expected
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                expected:
                                  $ref: '#/components/schemas/__schema0'
                                type:
                                  type: string
                                  const: jsonSubset
                              required:
                                - key
                                - expected
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                arguments:
                                  $ref: '#/components/schemas/__schema0'
                                argumentsMatch:
                                  default: subset
                                  type: string
                                  enum:
                                    - exact
                                    - subset
                                connectedAccountId:
                                  $ref: '#/components/schemas/ArbolResourceId'
                                toolKey:
                                  $ref: '#/components/schemas/ToolKey'
                                type:
                                  type: string
                                  const: toolCalled
                              required:
                                - key
                                - connectedAccountId
                                - toolKey
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                connectedAccountId:
                                  $ref: '#/components/schemas/ArbolResourceId'
                                toolKey:
                                  $ref: '#/components/schemas/ToolKey'
                                type:
                                  type: string
                                  const: toolNotCalled
                              required:
                                - key
                                - connectedAccountId
                                - toolKey
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                connectedAccountId:
                                  $ref: '#/components/schemas/ArbolResourceId'
                                expected:
                                  $ref: '#/components/schemas/__schema0'
                                toolKey:
                                  $ref: '#/components/schemas/ToolKey'
                                type:
                                  type: string
                                  const: stateEquals
                              required:
                                - key
                                - connectedAccountId
                                - expected
                                - toolKey
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                required:
                                  default: true
                                  type: boolean
                                maxMilliseconds:
                                  type: integer
                                  exclusiveMinimum: 0
                                  maximum: 300000
                                type:
                                  type: string
                                  const: maxLatency
                              required:
                                - key
                                - maxMilliseconds
                                - type
                              additionalProperties: false
                            - type: object
                              properties:
                                criteria:
                                  type: string
                                  minLength: 1
                                  maxLength: 4000
                                enforcement:
                                  oneOf:
                                    - type: object
                                      properties:
                                        calibration:
                                          oneOf:
                                            - type: object
                                              properties:
                                                state:
                                                  type: string
                                                  const: uncalibrated
                                              required:
                                                - state
                                              additionalProperties: false
                                            - type: object
                                              properties:
                                                examples:
                                                  minItems: 5
                                                  maxItems: 100
                                                  type: array
                                                  items:
                                                    type: object
                                                    properties:
                                                      exampleId:
                                                        type: string
                                                        minLength: 1
                                                        maxLength: 100
                                                        pattern: ^[a-z][a-z0-9_.-]*$
                                                      humanLabel:
                                                        type: string
                                                        enum:
                                                          - pass
                                                          - fail
                                                      judgeResult:
                                                        type: string
                                                        enum:
                                                          - pass
                                                          - fail
                                                          - unknown
                                                    required:
                                                      - exampleId
                                                      - humanLabel
                                                      - judgeResult
                                                    additionalProperties: false
                                                goldenSetVersion:
                                                  $ref: '#/components/schemas/Sha256Hex'
                                                calibrationVersion:
                                                  $ref: '#/components/schemas/Sha256Hex'
                                                judgeConfigHash:
                                                  $ref: '#/components/schemas/Sha256Hex'
                                                reviewedAt:
                                                  $ref: '#/components/schemas/IsoDatetime'
                                                reviewedBy:
                                                  $ref: '#/components/schemas/Auth0Subject'
                                                rubricDefinitionHash:
                                                  $ref: '#/components/schemas/Sha256Hex'
                                                state:
                                                  type: string
                                                  const: calibrated
                                              required:
                                                - examples
                                                - goldenSetVersion
                                                - calibrationVersion
                                                - judgeConfigHash
                                                - reviewedAt
                                                - reviewedBy
                                                - rubricDefinitionHash
                                                - state
                                              additionalProperties: false
                                        state:
                                          type: string
                                          const: advisory
                                      required:
                                        - calibration
                                        - state
                                      additionalProperties: false
                                    - type: object
                                      properties:
                                        calibration:
                                          type: object
                                          properties:
                                            examples:
                                              minItems: 5
                                              maxItems: 100
                                              type: array
                                              items:
                                                type: object
                                                properties:
                                                  exampleId:
                                                    type: string
                                                    minLength: 1
                                                    maxLength: 100
                                                    pattern: ^[a-z][a-z0-9_.-]*$
                                                  humanLabel:
                                                    type: string
                                                    enum:
                                                      - pass
                                                      - fail
                                                  judgeResult:
                                                    type: string
                                                    enum:
                                                      - pass
                                                      - fail
                                                      - unknown
                                                required:
                                                  - exampleId
                                                  - humanLabel
                                                  - judgeResult
                                                additionalProperties: false
                                            goldenSetVersion:
                                              $ref: '#/components/schemas/Sha256Hex'
                                            calibrationVersion:
                                              $ref: '#/components/schemas/Sha256Hex'
                                            judgeConfigHash:
                                              $ref: '#/components/schemas/Sha256Hex'
                                            reviewedAt:
                                              $ref: '#/components/schemas/IsoDatetime'
                                            reviewedBy:
                                              $ref: '#/components/schemas/Auth0Subject'
                                            rubricDefinitionHash:
                                              $ref: '#/components/schemas/Sha256Hex'
                                            state:
                                              type: string
                                              const: calibrated
                                          required:
                                            - examples
                                            - goldenSetVersion
                                            - calibrationVersion
                                            - judgeConfigHash
                                            - reviewedAt
                                            - reviewedBy
                                            - rubricDefinitionHash
                                            - state
                                          additionalProperties: false
                                        promotedAt:
                                          $ref: '#/components/schemas/IsoDatetime'
                                        promotedBy:
                                          $ref: '#/components/schemas/Auth0Subject'
                                        state:
                                          type: string
                                          const: blocking
                                      required:
                                        - calibration
                                        - promotedAt
                                        - promotedBy
                                        - state
                                      additionalProperties: false
                                key:
                                  type: string
                                  minLength: 1
                                  maxLength: 100
                                  pattern: ^[a-z][a-z0-9_.-]*$
                                type:
                                  type: string
                                  const: llmRubric
                              required:
                                - criteria
                                - enforcement
                                - key
                                - type
                              additionalProperties: false
                      tags:
                        default: []
                        maxItems: 20
                        type: array
                        items:
                          type: string
                          minLength: 1
                          maxLength: 80
                      timeoutSeconds:
                        default: 90
                        type: integer
                        minimum: 5
                        maximum: 300
                      toolMode:
                        default: mocked
                        type: string
                        enum:
                          - none
                          - mocked
                    required:
                      - id
                      - name
                      - scenario
                      - scorers
                    additionalProperties: false
                description:
                  anyOf:
                    - type: string
                      maxLength: 2000
                    - type: 'null'
                name:
                  type: string
                  minLength: 1
                  maxLength: 200
                status:
                  default: draft
                  type: string
                  enum:
                    - draft
                    - active
                    - archived
              required:
                - cases
                - name
      responses:
        '201':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/AgentEvaluationSuite'
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
        '400':
          description: >-
            Inferable validation failure or a non-inferable malformed transport
            request.
          content:
            application/json:
              schema:
                oneOf:
                  - oneOf:
                      - $ref: '#/components/schemas/BadRequest'
                      - $ref: '#/components/schemas/UndefinedError'
                  - $ref: '#/components/schemas/TransportBadRequestError'
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
        '401':
          description: Authentication is required.
          content:
            application/json:
              schema:
                oneOf:
                  - $ref: '#/components/schemas/Unauthorized'
                  - $ref: '#/components/schemas/UndefinedError'
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
        '403':
          description: The caller is not allowed to perform this operation.
          content:
            application/json:
              schema:
                oneOf:
                  - $ref: '#/components/schemas/Forbidden'
                  - $ref: '#/components/schemas/UndefinedError'
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
        '404':
          description: Agent not found
          content:
            application/json:
              schema:
                oneOf:
                  - $ref: '#/components/schemas/NotFound'
                  - $ref: '#/components/schemas/UndefinedError'
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
        '413':
          $ref: '#/components/responses/TransportPayloadTooLarge'
        '415':
          $ref: '#/components/responses/TransportUnsupportedMediaType'
        '422':
          description: Evaluation cases reference unavailable resources
          content:
            application/json:
              schema:
                oneOf:
                  - $ref: '#/components/schemas/UnprocessableContent'
                  - $ref: '#/components/schemas/UndefinedError'
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
      security:
        - userOAuth:
            - org:use
            - agent-evaluations:write
        - machineOAuth:
            - agent-evaluations:write
components:
  schemas:
    ArbolResourceId:
      type: string
      maxLength: 64
      pattern: >-
        ^[a-z][a-z0-9]{1,15}_[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$
      description: >-
        Opaque Arbol resource identifier; the endpoint and tenant-scoped lookup
        determine its resource type.
      examples:
        - agt_018f47a2-4f4d-7d61-9e2c-1d7b8e9a0c31
      title: Arbol resource ID
    RuntimeChannelKind:
      type: string
      enum:
        - web
        - whatsapp
        - voice
      description: Channel family supported by the Agent conversational runtime.
      examples:
        - web
      title: Runtime channel kind
    __schema0:
      anyOf:
        - type: 'null'
        - type: boolean
        - type: number
        - type: string
          maxLength: 65536
        - maxItems: 100
          type: array
          items:
            $ref: '#/components/schemas/__schema0'
        - type: object
          propertyNames:
            type: string
            minLength: 1
            maxLength: 200
          additionalProperties:
            $ref: '#/components/schemas/__schema0'
    ToolKey:
      type: string
      minLength: 1
      maxLength: 200
      pattern: ^[a-z][a-z0-9._:-]*$
      description: Stable provider-neutral action key derived from reviewed live discovery.
      examples:
        - google-calendar.create-event
      title: Tool key
    Sha256Hex:
      type: string
      minLength: 64
      maxLength: 64
      pattern: ^[0-9a-f]+$
      description: Lowercase hexadecimal SHA-256 digest.
      examples:
        - b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
      title: SHA-256 digest
    IsoDatetime:
      type: string
      format: date-time
      pattern: >-
        ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?(?:Z|([+-](?:[01]\d|2[0-3]):[0-5]\d)))$
      description: ISO 8601 timestamp with an explicit UTC offset.
      examples:
        - '2026-08-29T14:30:00Z'
      title: ISO timestamp
    Auth0Subject:
      $ref: '#/components/schemas/ExternalIdentityId'
      type: string
      minLength: 1
      maxLength: 255
      description: Opaque Auth0 subject for a person or organization-bound machine client.
      title: Auth0 subject
      examples:
        - auth0|0123456789abcdef
    AgentEvaluationSuite:
      type: object
      properties:
        agentId:
          $ref: '#/components/schemas/ArbolResourceId'
        cases:
          maxItems: 50
          type: array
          items:
            type: object
            properties:
              channelKind:
                $ref: '#/components/schemas/RuntimeChannelKind'
                default: web
              fixtures:
                default: []
                maxItems: 100
                type: array
                items:
                  type: object
                  properties:
                    calls:
                      maxItems: 20
                      type: array
                      items:
                        type: object
                        properties:
                          input:
                            $ref: '#/components/schemas/__schema0'
                          output:
                            $ref: '#/components/schemas/__schema0'
                          stateAfter:
                            $ref: '#/components/schemas/__schema0'
                        required:
                          - output
                        additionalProperties: false
                    connectedAccountId:
                      $ref: '#/components/schemas/ArbolResourceId'
                    initialState:
                      $ref: '#/components/schemas/__schema0'
                    toolKey:
                      $ref: '#/components/schemas/ToolKey'
                  required:
                    - calls
                    - connectedAccountId
                    - toolKey
                  additionalProperties: false
              id:
                type: string
                minLength: 1
                maxLength: 100
                pattern: ^[a-z][a-z0-9_.-]*$
              name:
                type: string
                minLength: 1
                maxLength: 200
              scenario:
                oneOf:
                  - type: object
                    properties:
                      messages:
                        minItems: 1
                        maxItems: 50
                        type: array
                        items:
                          type: object
                          properties:
                            content:
                              type: string
                              minLength: 1
                              maxLength: 16000
                            role:
                              type: string
                              enum:
                                - assistant
                                - user
                          required:
                            - content
                            - role
                          additionalProperties: false
                      type:
                        type: string
                        const: nextReply
                    required:
                      - messages
                      - type
                    additionalProperties: false
                  - type: object
                    properties:
                      messages:
                        minItems: 1
                        maxItems: 50
                        type: array
                        items:
                          type: object
                          properties:
                            content:
                              type: string
                              minLength: 1
                              maxLength: 16000
                            role:
                              type: string
                              enum:
                                - assistant
                                - user
                          required:
                            - content
                            - role
                          additionalProperties: false
                      type:
                        type: string
                        const: toolCall
                    required:
                      - messages
                      - type
                    additionalProperties: false
                  - type: object
                    properties:
                      initialMessages:
                        default: []
                        maxItems: 20
                        type: array
                        items:
                          type: object
                          properties:
                            content:
                              type: string
                              minLength: 1
                              maxLength: 16000
                            role:
                              type: string
                              enum:
                                - assistant
                                - user
                          required:
                            - content
                            - role
                          additionalProperties: false
                      maxTurns:
                        default: 5
                        type: integer
                        minimum: 1
                        maximum: 20
                      type:
                        type: string
                        const: simulation
                      userGoal:
                        type: string
                        minLength: 1
                        maxLength: 4000
                    required:
                      - initialMessages
                      - maxTurns
                      - type
                      - userGoal
                    additionalProperties: false
                  - type: object
                    properties:
                      conversationId:
                        $ref: '#/components/schemas/ArbolResourceId'
                      type:
                        type: string
                        const: recordedConversation
                    required:
                      - conversationId
                      - type
                    additionalProperties: false
              scorers:
                minItems: 1
                maxItems: 30
                type: array
                items:
                  oneOf:
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        caseSensitive:
                          default: true
                          type: boolean
                        expected:
                          type: string
                          maxLength: 16000
                        type:
                          type: string
                          const: textEquals
                      required:
                        - key
                        - required
                        - caseSensitive
                        - expected
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        caseSensitive:
                          default: false
                          type: boolean
                        expected:
                          type: string
                          minLength: 1
                          maxLength: 4000
                        type:
                          type: string
                          const: textContains
                      required:
                        - key
                        - required
                        - caseSensitive
                        - expected
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        expected:
                          $ref: '#/components/schemas/__schema0'
                        type:
                          type: string
                          const: jsonEquals
                      required:
                        - key
                        - required
                        - expected
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        expected:
                          $ref: '#/components/schemas/__schema0'
                        type:
                          type: string
                          const: jsonSubset
                      required:
                        - key
                        - required
                        - expected
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        arguments:
                          $ref: '#/components/schemas/__schema0'
                        argumentsMatch:
                          default: subset
                          type: string
                          enum:
                            - exact
                            - subset
                        connectedAccountId:
                          $ref: '#/components/schemas/ArbolResourceId'
                        toolKey:
                          $ref: '#/components/schemas/ToolKey'
                        type:
                          type: string
                          const: toolCalled
                      required:
                        - key
                        - required
                        - argumentsMatch
                        - connectedAccountId
                        - toolKey
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        connectedAccountId:
                          $ref: '#/components/schemas/ArbolResourceId'
                        toolKey:
                          $ref: '#/components/schemas/ToolKey'
                        type:
                          type: string
                          const: toolNotCalled
                      required:
                        - key
                        - required
                        - connectedAccountId
                        - toolKey
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        connectedAccountId:
                          $ref: '#/components/schemas/ArbolResourceId'
                        expected:
                          $ref: '#/components/schemas/__schema0'
                        toolKey:
                          $ref: '#/components/schemas/ToolKey'
                        type:
                          type: string
                          const: stateEquals
                      required:
                        - key
                        - required
                        - connectedAccountId
                        - expected
                        - toolKey
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        required:
                          default: true
                          type: boolean
                        maxMilliseconds:
                          type: integer
                          exclusiveMinimum: 0
                          maximum: 300000
                        type:
                          type: string
                          const: maxLatency
                      required:
                        - key
                        - required
                        - maxMilliseconds
                        - type
                      additionalProperties: false
                    - type: object
                      properties:
                        criteria:
                          type: string
                          minLength: 1
                          maxLength: 4000
                        enforcement:
                          oneOf:
                            - type: object
                              properties:
                                calibration:
                                  oneOf:
                                    - type: object
                                      properties:
                                        state:
                                          type: string
                                          const: uncalibrated
                                      required:
                                        - state
                                      additionalProperties: false
                                    - type: object
                                      properties:
                                        examples:
                                          minItems: 5
                                          maxItems: 100
                                          type: array
                                          items:
                                            type: object
                                            properties:
                                              exampleId:
                                                type: string
                                                minLength: 1
                                                maxLength: 100
                                                pattern: ^[a-z][a-z0-9_.-]*$
                                              humanLabel:
                                                type: string
                                                enum:
                                                  - pass
                                                  - fail
                                              judgeResult:
                                                type: string
                                                enum:
                                                  - pass
                                                  - fail
                                                  - unknown
                                            required:
                                              - exampleId
                                              - humanLabel
                                              - judgeResult
                                            additionalProperties: false
                                        goldenSetVersion:
                                          $ref: '#/components/schemas/Sha256Hex'
                                        calibrationVersion:
                                          $ref: '#/components/schemas/Sha256Hex'
                                        judgeConfigHash:
                                          $ref: '#/components/schemas/Sha256Hex'
                                        reviewedAt:
                                          $ref: '#/components/schemas/IsoDatetime'
                                        reviewedBy:
                                          $ref: '#/components/schemas/Auth0Subject'
                                        rubricDefinitionHash:
                                          $ref: '#/components/schemas/Sha256Hex'
                                        state:
                                          type: string
                                          const: calibrated
                                      required:
                                        - examples
                                        - goldenSetVersion
                                        - calibrationVersion
                                        - judgeConfigHash
                                        - reviewedAt
                                        - reviewedBy
                                        - rubricDefinitionHash
                                        - state
                                      additionalProperties: false
                                state:
                                  type: string
                                  const: advisory
                              required:
                                - calibration
                                - state
                              additionalProperties: false
                            - type: object
                              properties:
                                calibration:
                                  type: object
                                  properties:
                                    examples:
                                      minItems: 5
                                      maxItems: 100
                                      type: array
                                      items:
                                        type: object
                                        properties:
                                          exampleId:
                                            type: string
                                            minLength: 1
                                            maxLength: 100
                                            pattern: ^[a-z][a-z0-9_.-]*$
                                          humanLabel:
                                            type: string
                                            enum:
                                              - pass
                                              - fail
                                          judgeResult:
                                            type: string
                                            enum:
                                              - pass
                                              - fail
                                              - unknown
                                        required:
                                          - exampleId
                                          - humanLabel
                                          - judgeResult
                                        additionalProperties: false
                                    goldenSetVersion:
                                      $ref: '#/components/schemas/Sha256Hex'
                                    calibrationVersion:
                                      $ref: '#/components/schemas/Sha256Hex'
                                    judgeConfigHash:
                                      $ref: '#/components/schemas/Sha256Hex'
                                    reviewedAt:
                                      $ref: '#/components/schemas/IsoDatetime'
                                    reviewedBy:
                                      $ref: '#/components/schemas/Auth0Subject'
                                    rubricDefinitionHash:
                                      $ref: '#/components/schemas/Sha256Hex'
                                    state:
                                      type: string
                                      const: calibrated
                                  required:
                                    - examples
                                    - goldenSetVersion
                                    - calibrationVersion
                                    - judgeConfigHash
                                    - reviewedAt
                                    - reviewedBy
                                    - rubricDefinitionHash
                                    - state
                                  additionalProperties: false
                                promotedAt:
                                  $ref: '#/components/schemas/IsoDatetime'
                                promotedBy:
                                  $ref: '#/components/schemas/Auth0Subject'
                                state:
                                  type: string
                                  const: blocking
                              required:
                                - calibration
                                - promotedAt
                                - promotedBy
                                - state
                              additionalProperties: false
                        key:
                          type: string
                          minLength: 1
                          maxLength: 100
                          pattern: ^[a-z][a-z0-9_.-]*$
                        type:
                          type: string
                          const: llmRubric
                      required:
                        - criteria
                        - enforcement
                        - key
                        - type
                      additionalProperties: false
              tags:
                default: []
                maxItems: 20
                type: array
                items:
                  type: string
                  minLength: 1
                  maxLength: 80
              timeoutSeconds:
                default: 90
                type: integer
                minimum: 5
                maximum: 300
              toolMode:
                default: mocked
                type: string
                enum:
                  - none
                  - mocked
            required:
              - channelKind
              - fixtures
              - id
              - name
              - scenario
              - scorers
              - tags
              - timeoutSeconds
              - toolMode
            additionalProperties: false
        createdAt:
          $ref: '#/components/schemas/IsoDatetime'
        createdBy:
          type: string
          minLength: 1
          maxLength: 255
        description:
          anyOf:
            - type: string
              maxLength: 2000
            - type: 'null'
        id:
          $ref: '#/components/schemas/ArbolResourceId'
        name:
          type: string
          minLength: 1
          maxLength: 200
        revision:
          type: integer
          exclusiveMinimum: 0
          maximum: 9007199254740991
        status:
          type: string
          enum:
            - draft
            - active
            - archived
        updatedAt:
          $ref: '#/components/schemas/IsoDatetime'
        updatedBy:
          type: string
          minLength: 1
          maxLength: 255
      required:
        - agentId
        - cases
        - createdAt
        - createdBy
        - description
        - id
        - name
        - revision
        - status
        - updatedAt
        - updatedBy
      additionalProperties: false
      description: >-
        Revisioned evaluation case suite owned by one Agent and frozen into each
        run snapshot.
      title: Agent evaluation suite
    BadRequest:
      type: object
      properties:
        defined:
          const: true
        inferable:
          type: boolean
        code:
          const: BAD_REQUEST
        status:
          const: 400
        message:
          type: string
        data:
          type: object
          properties:
            issues:
              maxItems: 20
              type: array
              items:
                type: object
                properties:
                  message:
                    type: string
                    maxLength: 500
                  path:
                    maxItems: 16
                    type: array
                    items:
                      anyOf:
                        - type: string
                          maxLength: 120
                        - type: integer
                          minimum: -9007199254740991
                          maximum: 9007199254740991
                required:
                  - message
                additionalProperties: false
            truncated:
              type: boolean
          required:
            - issues
            - truncated
          additionalProperties: false
      required:
        - defined
        - inferable
        - code
        - status
        - message
        - data
    UndefinedError:
      type: object
      properties:
        defined:
          const: false
        inferable:
          type: boolean
        code:
          type: string
        status:
          type: number
        message:
          type: string
        data: {}
      required:
        - defined
        - inferable
        - code
        - status
        - message
    TransportBadRequestError:
      additionalProperties: false
      properties:
        code:
          const: BAD_REQUEST
        data: {}
        defined:
          const: false
        inferable:
          const: false
        message:
          type: string
      required:
        - defined
        - inferable
        - code
        - message
      type: object
    Unauthorized:
      type: object
      properties:
        defined:
          const: true
        inferable:
          type: boolean
        code:
          const: UNAUTHORIZED
        status:
          const: 401
        message:
          type: string
        data: {}
      required:
        - defined
        - inferable
        - code
        - status
        - message
    Forbidden:
      type: object
      properties:
        defined:
          const: true
        inferable:
          type: boolean
        code:
          const: FORBIDDEN
        status:
          const: 403
        message:
          type: string
        data:
          type: object
          properties:
            missingPermissions:
              maxItems: 10
              type: array
              items:
                $ref: '#/components/schemas/AuthPermission'
          required:
            - missingPermissions
          additionalProperties: false
      required:
        - defined
        - inferable
        - code
        - status
        - message
    NotFound:
      type: object
      properties:
        defined:
          const: true
        inferable:
          type: boolean
        code:
          const: NOT_FOUND
        status:
          const: 404
        message:
          type: string
        data: {}
      required:
        - defined
        - inferable
        - code
        - status
        - message
    UnprocessableContent:
      type: object
      properties:
        defined:
          const: true
        inferable:
          type: boolean
        code:
          const: UNPROCESSABLE_CONTENT
        status:
          const: 422
        message:
          type: string
        data: {}
      required:
        - defined
        - inferable
        - code
        - status
        - message
    ExternalIdentityId:
      type: string
      minLength: 1
      maxLength: 512
      description: >-
        Opaque identifier issued by an external authority; consumers compare it
        but do not interpret it.
      title: External identity ID
    AuthPermission:
      type: string
      enum:
        - agent-evaluations:read
        - agent-evaluations:run
        - agent-evaluations:write
        - agents:read
        - agents:write
        - analytics:read
        - arbi:use
        - channels:read
        - channels:write
        - contacts:read
        - contacts:write
        - conversations:read
        - conversations:transcript
        - conversations:write
        - definitions:read
        - definitions:write
        - facts:read
        - facts:write
        - integrations:read
        - integrations:write
        - knowledge:read
        - knowledge:write
        - org:manage
        - org:use
      description: >-
        One permission from the closed Arbol API vocabulary accepted from a
        verified Auth0 token.
      examples:
        - agents:read
      title: API permission
    TransportPayloadTooLargeError:
      additionalProperties: false
      properties:
        code:
          const: PAYLOAD_TOO_LARGE
        data: {}
        defined:
          const: false
        inferable:
          const: false
        message:
          type: string
      required:
        - defined
        - inferable
        - code
        - message
      type: object
    TransportUnsupportedMediaTypeError:
      additionalProperties: false
      properties:
        code:
          const: UNSUPPORTED_MEDIA_TYPE
        data: {}
        defined:
          const: false
        inferable:
          const: false
        message:
          type: string
      required:
        - defined
        - inferable
        - code
        - message
      type: object
  headers:
    RequestId:
      description: >-
        Opaque request correlation identifier. Include it when contacting Arbol
        support.
      schema:
        type: string
        maxLength: 128
  responses:
    TransportPayloadTooLarge:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/TransportPayloadTooLargeError'
      description: The request body exceeds the API transport limit.
      headers:
        X-Request-Id:
          $ref: '#/components/headers/RequestId'
    TransportUnsupportedMediaType:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/TransportUnsupportedMediaTypeError'
      description: The request media type is not supported.
      headers:
        X-Request-Id:
          $ref: '#/components/headers/RequestId'
  securitySchemes:
    userOAuth:
      description: >-
        Organization-scoped Auth0 user access token issued to an approved
        first-party client or exchanged by a trusted OBO resource server.
      flows:
        authorizationCode:
          authorizationUrl: https://auth.getarbol.com/authorize
          scopes:
            agent-evaluations:read: Read Agent evaluation suites, runs, and results.
            agent-evaluations:run: Execute and cancel Agent evaluation runs.
            agent-evaluations:write: Create and update Agent evaluation suites.
            agents:read: Read Agents and their explicitly requested expansions.
            agents:write: Create, update, and delete Agents.
            analytics:read: Read organization analytics.
            arbi:use: Use the personal Arbi workspace.
            channels:read: Read Agent channels.
            channels:write: Create, update, authorize, and delete Agent channels.
            contacts:read: Read Contacts and their explicitly requested expansions.
            contacts:write: Create, update, merge, import, and delete Contacts.
            conversations:read: Read Conversation metadata.
            conversations:transcript: Read Conversation messages and media evidence.
            conversations:write: Create and process Conversations.
            definitions:read: Read Definitions and Definition usage.
            definitions:write: Create, update, and retire Definitions.
            facts:read: Read current and historical Facts.
            facts:write: Assert typed Facts.
            integrations:read: Discover integrations, tools, accounts, and tool-bearing evidence.
            integrations:write: Authorize accounts and assign exact tool access.
            knowledge:read: Read Knowledge documents and search indexed content.
            knowledge:write: Create, index, update, and delete Knowledge documents.
            org:manage: Update the Organization and administer its members.
            org:use: Use Arbol as a member of the selected Auth0 Organization.
          tokenUrl: https://auth.getarbol.com/oauth/token
      type: oauth2
    machineOAuth:
      description: >-
        Organization-bound Auth0 client-credentials token. The client grant must
        contain every operation scope.
      flows:
        clientCredentials:
          scopes:
            agent-evaluations:read: Read Agent evaluation suites, runs, and results.
            agent-evaluations:run: Execute and cancel Agent evaluation runs.
            agent-evaluations:write: Create and update Agent evaluation suites.
            agents:read: Read Agents and their explicitly requested expansions.
            agents:write: Create, update, and delete Agents.
            analytics:read: Read organization analytics.
            channels:read: Read Agent channels.
            channels:write: Create, update, authorize, and delete Agent channels.
            contacts:read: Read Contacts and their explicitly requested expansions.
            contacts:write: Create, update, merge, import, and delete Contacts.
            conversations:read: Read Conversation metadata.
            conversations:transcript: Read Conversation messages and media evidence.
            conversations:write: Create and process Conversations.
            definitions:read: Read Definitions and Definition usage.
            definitions:write: Create, update, and retire Definitions.
            facts:read: Read current and historical Facts.
            facts:write: Assert typed Facts.
            integrations:read: Discover integrations, tools, accounts, and tool-bearing evidence.
            integrations:write: Authorize accounts and assign exact tool access.
            knowledge:read: Read Knowledge documents and search indexed content.
            knowledge:write: Create, index, update, and delete Knowledge documents.
            org:manage: Update the Organization and administer its members.
          tokenUrl: https://auth.getarbol.com/oauth/token
      type: oauth2

````