Skip to main content

List Benchmarks

List benchmark runs with GET /v1/quality/benchmarks. Each run scores extraction output against a ground truth dataset with per-field accuracy metrics.

A benchmark run compares your extraction output against a ground truth dataset to produce per-field accuracy scores. GET /v1/quality/benchmarks lists every run in your workspace: each evaluates all documents in its dataset and reports an accuracy_overall score with per-field breakdowns. Use benchmarks to track extraction quality over time and measure the impact of schema or extraction changes.

Use this endpoint to see all benchmark runs and their accuracy scores. A typical workflow is to list benchmarks after making schema or extraction changes, then compare the latest run against previous ones using GET /v1/quality/benchmarks/compare to measure improvement or detect regressions.

Each benchmark includes status (queued, running, or complete — note the terminal value is complete, not completed), accuracy_overall (0-1 score, null until the run finishes), accuracy_by_field (per-field breakdown), and documents_processed/documents_total for progress tracking. accuracy_delta and compared_to_run_id are populated only when a run was started with a comparison target in the platform; for ad-hoc deltas use the compare endpoint.

Run benchmarks regularly after extraction changes. Pair with GET /v1/quality/benchmarks/:id/results for per-document drill-down showing which fields matched and which diverged. Use the compare endpoint to track accuracy trends across multiple runs.

Accuracy fields (accuracy_overall, accuracy_by_field) stay null while a run is queued or running. Poll GET /v1/quality/benchmarks/:id and read the scores once status reaches complete. Runs created through the API execute in the request and come back complete.
GET/v1/quality/benchmarks

Query parameters

limitintegerMaximum number of results to return (1–100). Default: 20
cursorstringPagination cursor from a previous response.
orderstringSort direction by created_at. Default: desc

Request

curl "https://api.talonic.com/v1/quality/benchmarks?limit=20&order=desc" \
  -H "Authorization: Bearer tlnc_..."

The detail route GET /v1/quality/benchmarks/:id returns the same run object with an embedded results array (per-document accuracy records), so a single poll can both check status and read results once the run completes. The list route uses the same keyset cursor pagination as the dataset list: pass limit (1–100), cursor from pagination.next_cursor, and order.

Response

Response fields

dataarrayArray of benchmark run objects.
data[].idstringBenchmark run UUID.
data[].namestringBenchmark run name.
data[].dataset_idstringGround truth dataset ID used for this run.
data[].user_schema_idstring | nullUser schema scoping this benchmark, if any.
data[].statusstringRun status: queued, running, or complete. Runs created through the API return already complete.
data[].accuracy_overallnumber | nullOverall accuracy score (0–1). Null while running.
data[].accuracy_by_fieldobject | nullPer-field accuracy scores. Null while running.
data[].documents_processedintegerNumber of documents evaluated so far.
data[].documents_totalintegerTotal documents to evaluate.
data[].duration_msinteger | nullTotal run duration in milliseconds.
data[].created_atstringISO 8601 creation timestamp.
data[].completed_atstring | nullISO 8601 completion timestamp.
data[].links.selfstringURL to this benchmark run.
data[].links.resultsstringURL to the per-document results.
pagination.totalintegerTotal number of benchmark runs.
pagination.limitintegerMaximum results per page.
pagination.has_morebooleanWhether more results exist beyond this page.
pagination.next_cursorstring | nullCursor to fetch the next page.

Response

{
  "data": [
    {
      "id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
      "name": "Benchmark 2024-09-25",
      "dataset_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "user_schema_id": null,
      "status": "complete",
      "accuracy_overall": 0.93,
      "accuracy_by_field": {
        "vendor_name": 0.98,
        "total_amount": 0.90,
        "invoice_number": 0.92
      },
      "documents_processed": 50,
      "documents_total": 50,
      "duration_ms": 4200,
      "accuracy_delta": null,
      "compared_to_run_id": null,
      "created_at": "2024-09-25T12:00:00.000Z",
      "completed_at": "2024-09-25T12:00:04.200Z",
      "links": {
        "self": "/v1/quality/benchmarks/c3d4e5f6-a7b8-9012-cdef-123456789012",
        "results": "/v1/quality/benchmarks/c3d4e5f6-a7b8-9012-cdef-123456789012/results"
      }
    }
  ],
  "pagination": {
    "total": 5,
    "limit": 20,
    "has_more": false,
    "next_cursor": null
  }
}

Errors

Error responses

401unauthorizedMissing or invalid API key.
429rate_limitedToo many requests. Retry after the period indicated in the Retry-After header.

Frequently asked questions

What benchmark statuses are possible?+
queued, running, and complete (note: complete, not completed). Runs created through the API execute in the request and are returned already complete. Match on the exact string when polling runs launched elsewhere.
Why is accuracy_overall null?+
Accuracy scores are only computed when the benchmark run finishes. While the status is queued or running, accuracy fields are null.
How do I track the progress of a running benchmark?+
Poll GET /v1/quality/benchmarks/:id and compare documents_processed against documents_total. Once status reaches complete, duration_ms records the total evaluation time and the accuracy fields are populated.
How do I compare two benchmark runs?+
Call `GET /v1/quality/benchmarks/compare?run_a=<id>&run_b=<id>`. The response returns both runs side by side plus an `accuracy_delta` (run_a minus run_b) so you can quantify improvement or regression between runs.

Related

Start a Benchmark Run
Benchmark Results and Comparison
List Benchmark Datasets