Skip to main content

Comparisons

List per-document field comparisons for N-Shot evaluation across job runs. Returns all comparisons for a specific job run.

Retrieve all per-document field comparisons for a job run. Each comparison shows the values produced by each shot, the agreement status (green/yellow/red), the majority value, and any override or judge decision that has been applied. Use this to drill into specific fields and understand where extraction diverges across shots.

GET/v1/jobs/runs/{runId}/nshot/comparisons

Response

Response fields

dataarrayArray of comparison objects.
data[].idstringComparison UUID.
data[].run_idstringJob run UUID.
data[].document_idstringDocument UUID.
data[].field_namestringField name being compared.
data[].statusstringAgreement status: green, yellow, or red.
data[].agreement_scorenumberNumeric agreement score between 0 and 1.
data[].majority_valueanyThe value agreed upon by the majority of shots.
data[].comparison_methodstringMethod used to compare values (e.g. exact, semantic).
data[].valuesarrayPer-shot values with shot_number and value.
data[].overrideobject | nullOverride record if a value was manually selected.
data[].judgementobject | nullLLM judge recommendation with accepted flag.
data[].created_atstringISO 8601 creation timestamp.
data[].links.selfstringURL to retrieve this specific comparison.

Response

{
  "data": [
    {
      "id": "cmp-uuid-1",
      "run_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "document_id": "doc-uuid-1",
      "field_name": "invoice_number",
      "status": "green",
      "agreement_score": 1.0,
      "majority_value": "INV-2024-001",
      "comparison_method": "exact",
      "values": [
        { "shot_number": 1, "value": "INV-2024-001" },
        { "shot_number": 2, "value": "INV-2024-001" },
        { "shot_number": 3, "value": "INV-2024-001" }
      ],
      "override": null,
      "judgement": null,
      "created_at": "2024-09-14T10:32:00.000Z",
      "links": {
        "self": "/v1/jobs/runs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/nshot/comparison?document_id=doc-uuid-1&field_name=invoice_number"
      }
    }
  ]
}

Most integrations start with GET /v1/jobs/runs/{runId}/nshot/summary to check the overall agreement_rate, then drill into this endpoint to find status: "red" and status: "yellow" comparisons that need attention. Use the override and judgement fields to track which comparisons have already been reviewed and which still need a decision.

Errors

Error responses

401unauthorizedMissing or invalid API key.
404not_foundNo job run with this ID exists for your organization.
429rate_limitedToo many requests. Retry after the period indicated in the Retry-After header.

Related

Single Comparison
Summary