Skip to main content

Judge Decision

Submit a judge decision (human or AI) for an N-Shot comparison to record which candidate produced the correct extraction result.

Submit a decision to accept or decline the LLM judge's recommendation for a specific comparison. When accepted is true, the recommended shot value is automatically applied as an override. When false, the recommendation is recorded as declined and no override is applied. Use this to efficiently review LLM suggestions at scale.

Accepting a judge decision automatically creates an override with actor_id: "judge". You can still manually override the value afterwards using the override endpoint.
POST/v1/jobs/runs/{runId}/nshot/judge-decision

Response

Response fields

idstringComparison UUID.
run_idstringJob run UUID.
document_idstringDocument UUID.
field_namestringField name.
statusstringAgreement status.
majority_valueanyOriginal majority value.
valuesarrayPer-shot values.
overrideobject | nullOverride record applied if accepted is true.
judgementobjectUpdated judgement record.
judgement.recommended_shotintegerShot number the LLM recommended.
judgement.recommended_valueanyValue the LLM recommended.
judgement.acceptedbooleanWhether the recommendation was accepted.
judgement.decided_bystringAlways "api" when decided via this endpoint.
judgement.decided_atstringISO 8601 timestamp of the decision.

Response

{
  "id": "cmp-uuid-1",
  "run_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "document_id": "doc-uuid-1",
  "field_name": "total_amount",
  "status": "yellow",
  "agreement_score": 0.6667,
  "majority_value": "12450.00",
  "comparison_method": "exact",
  "values": [
    { "shot_number": 1, "value": "12450.00" },
    { "shot_number": 2, "value": "12450.00" },
    { "shot_number": 3, "value": "12,450.00" }
  ],
  "override": {
    "selected_shot": 1,
    "actor_id": "judge",
    "overridden_at": "2024-09-14T11:05:00.000Z",
    "from_value": "12450.00",
    "to_value": "12450.00"
  },
  "judgement": {
    "recommended_shot": 1,
    "recommended_value": "12450.00",
    "accepted": true,
    "decided_by": "api",
    "decided_at": "2024-09-14T11:05:00.000Z"
  },
  "created_at": "2024-09-14T10:32:00.000Z",
  "links": {
    "self": "/v1/jobs/runs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/nshot/comparison?document_id=doc-uuid-1&field_name=total_amount"
  }
}

This endpoint fits into a review workflow where you batch-process LLM judge recommendations. Retrieve comparisons with judgement.accepted: null from the comparisons list, then iterate through them calling this endpoint with accepted: true or false. Accepted decisions automatically create an override with actor_id: "judge", so no separate override call is needed.

Errors

Error responses

401unauthorizedMissing or invalid API key.
404not_foundNo job run with this ID exists for your organization, or no comparison found for the given document_id and field_name.
429rate_limitedToo many requests. Retry after the period indicated in the Retry-After header.

Related

Summary
Override