Skip to main content

Judge Decision

Accept or decline the LLM judge's recommendation for an N-Shot comparison. Accepted decisions automatically apply the recommended shot value as an override.

POST /v1/jobs/runs/{runId}/nshot/judge-decision accepts or declines the LLM judge's recommendation for a specific N-Shot comparison. When accepted is true, the recommended shot value is automatically applied as an override. When false, the recommendation is recorded as declined and no override is applied. Use this to efficiently review LLM suggestions at scale.

Accepting a judge decision automatically creates an override with actor_id: "judge". You can still manually override the value afterwards using the override endpoint.
POST/v1/jobs/runs/{runId}/nshot/judge-decision

Body parameters

document_id*uuidDocument ID.
field_name*stringField name.
accepted*booleanWhether to accept the LLM judge recommendation.

Request

Response

Response fields

idstringComparison UUID.
run_idstringJob run UUID.
document_idstringDocument UUID.
field_namestringField name.
statusstringAgreement status.
majority_valueanyOriginal majority value.
valuesarrayPer-shot values.
overrideobject | nullOverride record applied if accepted is true.
judgementobjectUpdated judgement record.
judgement.recommended_shotintegerShot number the LLM recommended.
judgement.recommended_valueanyValue the LLM recommended.
judgement.acceptedbooleanWhether the recommendation was accepted.
judgement.decided_bystringAlways "api" when decided via this endpoint.
judgement.decided_atstringISO 8601 timestamp of the decision.

Response

{
  "id": "cmp-uuid-1",
  "run_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "document_id": "doc-uuid-1",
  "field_name": "total_amount",
  "status": "yellow",
  "agreement_score": 0.6667,
  "majority_value": "12450.00",
  "comparison_method": "exact",
  "values": [
    { "shot_number": 1, "value": "12450.00" },
    { "shot_number": 2, "value": "12450.00" },
    { "shot_number": 3, "value": "12,450.00" }
  ],
  "override": {
    "selected_shot": 1,
    "actor_id": "judge",
    "overridden_at": "2024-09-14T11:05:00.000Z",
    "from_value": "12450.00",
    "to_value": "12450.00"
  },
  "judgement": {
    "recommended_shot": 1,
    "recommended_value": "12450.00",
    "accepted": true,
    "decided_by": "api",
    "decided_at": "2024-09-14T11:05:00.000Z"
  },
  "created_at": "2024-09-14T10:32:00.000Z",
  "links": {
    "self": "/v1/jobs/runs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/nshot/comparison?document_id=doc-uuid-1&field_name=total_amount"
  }
}

This endpoint fits into a review workflow where you batch-process LLM judge recommendations. Retrieve comparisons with judgement.accepted: null from the comparisons list, then iterate through them calling this endpoint with accepted: true or false. Accepted decisions automatically create an override with actor_id: "judge", so no separate override call is needed.

Errors

Error responses

400validation_errorThe request body failed validation, e.g. document_id is not a UUID or accepted is not a boolean.
401unauthorizedMissing or invalid API key.
403forbiddenThe API key lacks the write scope required by this endpoint.
404not_foundNo job run with this ID exists for your organization, or no comparison found for the given document_id and field_name.
429rate_limitedToo many requests. Retry after the period indicated in the Retry-After header.

Related

Summary
Override
Single Comparison