List Benchmarks

List benchmark runs with GET /v1/quality/benchmarks. Each run scores extraction output against a ground truth dataset with per-field accuracy metrics.

A benchmark run compares your extraction output against a ground truth dataset to produce per-field accuracy scores. GET /v1/quality/benchmarks lists every run in your workspace: each evaluates all documents in its dataset and reports an accuracy_overall score with per-field breakdowns. Use benchmarks to track extraction quality over time and measure the impact of schema or extraction changes.

Use this endpoint to see all benchmark runs and their accuracy scores. A typical workflow is to list benchmarks after making schema or extraction changes, then compare the latest run against previous ones using GET /v1/quality/benchmarks/compare to measure improvement or detect regressions.

Each benchmark includes status (queued, running, completed, or failed), accuracy_overall (0-1 score, null while running), accuracy_by_field (per-field breakdown), and documents_processed/documents_total for progress tracking. The accuracy_delta and compared_to_run_id fields support cross-run comparisons.

Run benchmarks regularly after extraction changes. Pair with GET /v1/quality/benchmarks/:id/results for per-document drill-down showing which fields matched and which diverged. Use the compare endpoint to track accuracy trends across multiple runs.

Accuracy fields (accuracy_overall, accuracy_by_field) stay null while a run is queued or running. Poll GET /v1/quality/benchmarks/:id and read the scores once status reaches completed.

GET/v1/quality/benchmarks

Query parameters

limitintegerMaximum number of results to return (1–100). Default: 20

cursorstringPagination cursor from a previous response.

orderstringSort direction by created_at. Default: desc

Response

Response fields

dataarrayArray of benchmark run objects.

data[].idstringBenchmark run UUID.

data[].namestringBenchmark run name.

data[].dataset_idstringGround truth dataset ID used for this run.

data[].user_schema_idstring | nullUser schema scoping this benchmark, if any.

data[].statusstringRun status: queued, running, completed, or failed.

data[].accuracy_overallnumber | nullOverall accuracy score (0–1). Null while running.

data[].accuracy_by_fieldobject | nullPer-field accuracy scores. Null while running.

data[].documents_processedintegerNumber of documents evaluated so far.

data[].documents_totalintegerTotal documents to evaluate.

data[].duration_msinteger | nullTotal run duration in milliseconds.

data[].created_atstringISO 8601 creation timestamp.

data[].completed_atstring | nullISO 8601 completion timestamp.

data[].links.selfstringURL to this benchmark run.

data[].links.resultsstringURL to the per-document results.

pagination.totalintegerTotal number of benchmark runs.

pagination.limitintegerMaximum results per page.

pagination.has_morebooleanWhether more results exist beyond this page.

pagination.next_cursorstring | nullCursor to fetch the next page.

Response

{
  "data": [
    {
      "id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
      "name": "Benchmark 2024-09-25",
      "dataset_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "user_schema_id": null,
      "status": "completed",
      "accuracy_overall": 0.93,
      "accuracy_by_field": {
        "vendor_name": 0.98,
        "total_amount": 0.90,
        "invoice_number": 0.92
      },
      "documents_processed": 50,
      "documents_total": 50,
      "duration_ms": 4200,
      "accuracy_delta": null,
      "compared_to_run_id": null,
      "created_at": "2024-09-25T12:00:00.000Z",
      "completed_at": "2024-09-25T12:00:04.200Z",
      "links": {
        "self": "/v1/quality/benchmarks/c3d4e5f6-a7b8-9012-cdef-123456789012",
        "results": "/v1/quality/benchmarks/c3d4e5f6-a7b8-9012-cdef-123456789012/results"
      }
    }
  ],
  "pagination": {
    "total": 5,
    "limit": 20,
    "has_more": false,
    "next_cursor": null
  }
}

Errors

Error responses

401unauthorizedMissing or invalid API key.

429rate_limitedToo many requests. Retry after the period indicated in the Retry-After header.

Start a Benchmark Run

Benchmark Results and Comparison

List Ground Truth Datasets

List Benchmarks

Response

Errors

Related