Skip to main content

Benchmark Results

Get per-document benchmark results showing matched and diverged fields, or compare two benchmark runs side by side to track extraction accuracy trends.

Benchmark results are the per-document accuracy records of a completed benchmark run: for every document, they show which fields matched the ground truth and which diverged. Each result includes the extracted value, the expected value, and a correct flag per field. A companion compare endpoint puts two runs side by side to quantify accuracy change.

Use the results endpoint after a run reaches completed to find where extraction fails: scan field_results for correct: false entries and inspect the extracted-versus-expected pairs. This tells you whether a low accuracy_overall comes from one systematically wrong field or from scattered document-level issues.

Results are only meaningful for completed runs. A run still in queued or running status has no results to return yet; poll GET /v1/quality/benchmarks/:id until status is completed.
GET/v1/quality/benchmarks/:id/results

Response (Results)

Response fields

dataarrayArray of per-document result objects.
data[].idstringResult UUID.
data[].document_idstringDocument evaluated.
data[].ground_truth_entry_idstringGround truth entry compared against.
data[].accuracynumberAccuracy score for this document (0–1).
data[].field_resultsobjectPer-field accuracy breakdown.
data[].created_atstringISO 8601 timestamp.

Response

{
  "data": [
    {
      "id": "d4e5f6a7-b8c9-0123-defa-234567890123",
      "document_id": "doc_abc123",
      "ground_truth_entry_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
      "accuracy": 0.93,
      "field_results": {
        "vendor_name": { "correct": true, "extracted": "Acme Corp", "expected": "Acme Corp" },
        "total_amount": { "correct": false, "extracted": "14000.00", "expected": "14250.00" },
        "invoice_number": { "correct": true, "extracted": "INV-2024-0847", "expected": "INV-2024-0847" }
      },
      "created_at": "2024-09-25T12:00:04.200Z"
    }
  ]
}

To track accuracy trends over time, compare two benchmark runs side by side. The accuracy_delta shows the difference in overall accuracy between the two runs.

GET/v1/quality/benchmarks/compare

Query parameters

run_a*stringFirst benchmark run ID.
run_b*stringSecond benchmark run ID to compare against.

Response (Compare)

Response fields

run_aobjectFull benchmark object for the first run.
run_bobjectFull benchmark object for the second run.
accuracy_deltanumber | nullDifference in overall accuracy (run_a minus run_b). Null if either run has no accuracy score yet.

Response (Compare)

{
  "run_a": {
    "id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
    "name": "Benchmark 2024-09-25",
    "status": "completed",
    "accuracy_overall": 0.93,
    "documents_total": 50,
    "created_at": "2024-09-25T12:00:00.000Z"
  },
  "run_b": {
    "id": "d4e5f6a7-b8c9-0123-defa-234567890123",
    "name": "Benchmark 2024-10-01",
    "status": "completed",
    "accuracy_overall": 0.96,
    "documents_total": 50,
    "created_at": "2024-10-01T09:00:00.000Z"
  },
  "accuracy_delta": -0.03
}

Errors

Error responses

400bad_requestBoth run_a and run_b query parameters are required for the compare endpoint.
401unauthorizedMissing or invalid API key.
404not_foundOne or both benchmark run IDs not found for your workspace.
429rate_limitedToo many requests. Retry after the period indicated in the Retry-After header.

Related

Start a Benchmark Run
List Benchmark Runs
List Ground Truth Datasets