Skip to main content

Validation Checks

Validation checks are automated quality rules that verify extracted data in Talonic before it reaches downstream systems. They run as positional checkpoints in the pipeline, combining deterministic rules with LLM-based judging, and are backed by two companion layers: Benchmarks, which measure accuracy against Ground Truth datasets, and a central field-review queue that collects failing fields for human resolution.

A note on naming, because the two layers are easy to conflate: Validation always means these in-pipeline checks and gates that run on live documents. Accuracy measurement is a separate surface called Benchmarks, which scores extraction results against Ground Truth datasets (formerly called "golden samples") of known-correct values. This page covers the in-pipeline checks first, then the Benchmarks API.

Validation runs as validation checkpoints placed positionally in the Spec rail. A checkpoint validates the output of the stage immediately before it, so several checkpoints can sit at different positions along the pipeline. Each checkpoint runs one or more mechanisms: evidence (a suite of 23 deterministic rules), business-rules (evaluated by a Haiku model), n-shot (few-shot example matching), and llm-judge (a model scoring each field against the schema and context).

The evidence mechanism covers structural and format checks: values matching expected patterns, numeric or date ranges, and cross-field consistency such as a start date preceding an end date. business-rules expresses domain constraints in natural language and lets a Haiku model judge each record against them. n-shot compares a field against labelled examples, and llm-judge scores fields a deterministic rule cannot express.

A checkpoint can be blocking or non-blocking. A non-blocking checkpoint records pass/fail for inspection without halting the run. A blocking checkpoint routes failing fields to the central field-review queue (see Review Gates and Field Review), so a human resolves them before the field continues downstream.

For best results, place a checkpoint after the stage whose output you most need to trust, and start with the evidence mechanism on critical identifiers (invoice numbers, dates, amounts) before adding business-rules or llm-judge. Benchmarks against Ground Truth (covered below) measure accuracy across runs independently of the in-pipeline checkpoints.

To set up validation, add a Validation node to the Spec rail at the position you want to validate. Configure which mechanisms it runs and whether it blocks. See the Pipelines guide on validation checkpoints (slug validation-checkpoints) for the full setup walkthrough. On each pipeline run, the checkpoint evaluates the preceding stage output and, if blocking, sends failing fields to the field-review queue.

  • Validation runs as positional checkpoints in the Spec rail, validating the preceding stage output
  • Evidence: a suite of 23 deterministic rules (format, range, cross-field consistency)
  • Business-rules: domain constraints in natural language, judged by a Haiku model
  • N-shot: few-shot example matching; llm-judge: model scoring against schema and context
  • Blocking checkpoints route failing fields to the central field-review queue
  • Benchmarks against Ground Truth measure accuracy independently of in-pipeline validation

Benchmarks via API

Benchmark runs are served by the /v1/validation/runs and /v1/quality/benchmarks namespaces. The legacy vocabulary survives only in the API contract: the paths keep the validation and quality names, and the request body still takes a golden_sample_id field, which is the ID of a Ground Truth dataset.

Start a benchmark run against a Ground Truth dataset
curl -X POST https://api.talonic.com/v1/validation/runs \
  -H "Authorization: Bearer $TALONIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "golden_sample_id": "e5f6a7b8-c9d0-1234-efab-345678901234",
    "schema_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901"
  }'

# Response (201): the created validation run.
# Poll GET /v1/validation/runs/{id} for status, then fetch
# GET /v1/validation/runs/{id}/results for per-document and
# per-field accuracy.
Get a benchmark run with per-field accuracy
curl -s https://api.talonic.com/v1/quality/benchmarks/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  -H "Authorization: Bearer $TALONIC_API_KEY"

# Response:
# {
#   "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
#   "status": "completed",
#   "accuracy_overall": 0.91,
#   "accuracy_by_field": {
#     "invoice_number": 1.0,
#     "total_amount": 0.95,
#     "payment_terms": 0.67
#   },
#   "documents_processed": 3,
#   "documents_total": 3
# }

Validation checkpoints form the in-pipeline line of defense in your data quality strategy. They catch issues at the point in the pipeline where they matter before data reaches downstream systems. The key to effective validation is placing checkpoints deliberately and starting simple. Begin with the evidence mechanism on your most critical identifiers (invoice numbers, dates, and monetary amounts) where format violations have the highest business impact. Then add business-rules or llm-judge as you learn your data patterns. Benchmarks against Ground Truth complement this by measuring accuracy across runs.

Validation checkpoints are positional. A checkpoint validates only the output of the stage immediately before it, so place several along the rail to validate different pipeline stages. A blocking checkpoint routes failing fields to the central field-review queue; a non-blocking one records pass/fail for inspection without halting the run.

Frequently asked questions

What validation mechanisms does Talonic support?+
Validation checkpoints run four mechanisms: evidence (a suite of 23 deterministic rules covering format, range, and cross-field consistency), business-rules (domain constraints judged by a Haiku model), n-shot (few-shot example matching), and llm-judge (a model scoring fields against the schema and context). Checkpoints sit positionally in the Spec rail.
Where do validation checks run?+
Validation runs as positional checkpoints in the Spec rail, not during a fixed phase of every job. Each checkpoint validates the output of the stage immediately before it, so you can place several checkpoints to validate different pipeline stages.
Do validation failures block the pipeline?+
Only if the checkpoint is blocking. A blocking checkpoint routes failing fields to the central field-review queue for human resolution before the field continues downstream. A non-blocking checkpoint records pass/fail for inspection without halting the run.
How do I measure extraction quality over time?+
Create Ground Truth datasets (5-10 documents per schema) and run Benchmarks after schema changes or model upgrades. A benchmark run compares extraction results field-by-field against your known-correct values, producing per-field accuracy scores with AI judge verdicts. Track these scores over time to measure the impact of schema improvements, instruction refinements, and reference table updates.

Related

Ground Truth
Review Gates
Pipelines: Validation Checkpoints