Running a Pipeline

Running a pipeline applies a Spec to a set of documents. Each document is enqueued as its own job and processed in parallel; within a document the phases run in strict sequence. The run is observable while it executes: the progress endpoint reports per-phase counts (pending, running, completed, errors), so you can watch transfer drain into extraction, extraction into resolution, and so on. Results appear progressively, the same way they do for a Job.

A document that clears every phase finishes as complete. A document with fields still blocked at a validation gate, held for declarative review, or waiting on a dependency finishes as partial: its clean fields are done, and the flagged fields wait in the review queue. This is by design. The pipeline never silently ships a value that failed a gate; it holds that cell and surfaces it for a human decision while the rest of the document proceeds.

Once a pipeline is finished you can turn it into a data product: an assembled, deliverable dataset that reads the latest canonical value per cell. The data product owns its own review surface, so creating one replays the pipeline gates over the stored verdicts to reconstruct the review queue, and held cells surface with status only until a reviewer resolves them. From there the product flows into delivery the same as any other.

Finding runs and values

The Pipelines overview is built for workspaces with many runs. A Filter panel narrows the list by run status (active, finalizing, completed, paused, archived, failed), by Spec, by creation date range, by document count bounds, and by two identifiers your integration already holds: Batch ID (the exact caller batch id attached at ingest) and Run ID (the submission run UUID that a /v1/run response and every dispatch payload carry). A Batch ID counts as active once it is non-blank; a Run ID only once it is a complete UUID, because a partial one would be ignored. The same filters are available on GET /pipelines as batch_id and run_id. The button shows the active-filter count and offers a one-click clear. Beside the name search sits a "Find a value…" search that answers a different question: which pipelines contain this extracted value anywhere in their cells. Type a needle and press Enter, and the results replace the list, each hit showing how many documents in that pipeline carry the value.

Value search takes an optional field scope. Once you type a needle, a field selector appears (defaulting to Any field) listing the Spec fields available to scope by; an unscoped search additionally annotates each hit with a Matched in list of the fields the value was found under, and clicking a scopable field there re-runs the search scoped to it. This is the fastest way to answer "which run processed invoice INV-2026-0472" or "where did this vendor name end up" without opening pipelines one by one.

Opening a pipeline lands on the Pipeline workspace, the default view of every run: the results grid sits under an evidence inspector, so selecting a cell shows its source evidence, verdicts, and history without leaving the grid. Clicking a document name opens the per-document blade with processing status and model usage; the blade ends with a Source preview of the file itself (a Document / Markdown toggle that also renders spreadsheets) and an Open document link to the full document page.

Inside a pipeline, every cell opens into a detail blade with a per-cell audit trail: the value versions across phases with their sources and confidence, the validation verdicts that judged the cell, any hold status, and the review decision that resolved it. This is the same trail component the review workbench and the data product review surface render, so what you see while debugging a run is exactly what a reviewer saw when deciding it. Spec-version provenance is also recorded server-side — each extraction cell stamps the schema version that produced it — anchoring every value to the exact Spec version it came from.

Polling progress

Poll the progress endpoint to track a run. It derives its phase rows from the Spec rail, so the phases you see match what you composed, and it returns an errors array with per-document messages for anything that failed.

curl https://api.talonic.com/v1/pipelines/pl_x8k2m9/progress \
  -H "Authorization: Bearer $TALONIC_API_KEY"

{
  "pipelineId": "pl_x8k2m9",
  "status": "active",
  "totalDocuments": 2,
  "completedDocuments": 1,
  "errorDocuments": 0,
  "phases": [
    { "phaseId": "transfer", "name": "Field Registry", "type": "transfer",
      "completed": 2, "running": 0, "pending": 0, "errors": 0 },
    { "phaseId": "extraction", "name": "Extraction", "type": "extraction",
      "completed": 1, "running": 1, "pending": 0, "errors": 0 }
  ],
  "errors": []
}

Re-run and data product

A finished pipeline can be re-run from a phase onward without redoing the work before it: pass the phase type to start from, and earlier phases reuse their stored cells. When the output looks right, create a data product from the run to make it deliverable.

A run keeps the plan it was compiled with, so a Spec republished after the run started does not change it silently. In the workspace, ↻ Rerun stage first checks whether the Spec changed since this run was planned. If nothing changed, the stage reruns. If the plan is stale, a dialog lists what would rerun with the changes and what is kept from this run, and offers two choices: Apply Spec changes and rerun starts from the earliest affected stage (never later than the stage you clicked), while Rerun as planned ignores the changes and reruns only what you clicked. When Extraction is among the affected stages, the dialog warns that every document goes through the model again.

curl -X POST https://api.talonic.com/v1/pipelines/pl_x8k2m9/rerun \
  -H "Authorization: Bearer $TALONIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "from_phase": "resolution" }'

curl -X POST https://api.talonic.com/v1/pipelines/pl_x8k2m9/data-product \
  -H "Authorization: Bearer $TALONIC_API_KEY"