Skip to main content

Running a Pipeline

Running a pipeline applies a Spec to a set of documents. Each document is enqueued as its own job and processed in parallel; within a document the phases run in strict sequence. The run is observable while it executes: the progress endpoint reports per-phase counts (pending, running, completed, errors), so you can watch transfer drain into extraction, extraction into resolution, and so on. Results appear progressively, the same way they do for a Job.

A document that clears every phase finishes as complete. A document with fields still blocked at a validation gate, held for declarative review, or waiting on a dependency finishes as partial: its clean fields are done, and the flagged fields wait in the review queue. This is by design. The pipeline never silently ships a value that failed a gate; it holds that cell and surfaces it for a human decision while the rest of the document proceeds.

Once a pipeline is finished you can turn it into a data product: an assembled, deliverable dataset that reads the latest canonical value per cell. The data product owns its own review surface, so creating one replays the pipeline gates over the stored verdicts to reconstruct the review queue, and held cells surface with status only until a reviewer resolves them. From there the product flows into delivery the same as any other.

Finding runs and values

The Pipelines overview is built for workspaces with many runs. A Filter panel narrows the list by run status (active, finalizing, completed, paused, archived, failed), by Spec, by creation date range, and by document count bounds, with an active-filter count on the button and a one-click clear. Beside the name search sits a "Find a value…" search that answers a different question: which pipelines contain this extracted value anywhere in their cells. Type a needle and press Enter, and the results replace the list, each hit showing how many documents in that pipeline carry the value.

Value search takes an optional field scope. Once you type a needle, a field selector appears (defaulting to Any field) listing the Spec fields available to scope by; an unscoped search additionally annotates each hit with a Matched in list of the fields the value was found under, and clicking a scopable field there re-runs the search scoped to it. This is the fastest way to answer "which run processed invoice INV-2026-0472" or "where did this vendor name end up" without opening pipelines one by one.

Inside a pipeline, every cell opens into a detail blade with a per-cell audit trail: the value versions across phases with their sources and confidence, the validation verdicts that judged the cell, any hold status, and the review decision that resolved it. This is the same trail component the review workbench and the data product review surface render, so what you see while debugging a run is exactly what a reviewer saw when deciding it. Spec-version provenance is also recorded server-side — each extraction cell stamps the schema version that produced it — anchoring every value to the exact Spec version it came from.

Polling progress

Poll the progress endpoint to track a run. It derives its phase rows from the Spec rail, so the phases you see match what you composed, and it returns an errors array with per-document messages for anything that failed.

Check pipeline progress
curl https://api.talonic.com/v1/pipelines/pl_x8k2m9/progress \
  -H "Authorization: Bearer $TALONIC_API_KEY"
Response
{
  "pipelineId": "pl_x8k2m9",
  "status": "active",
  "totalDocuments": 2,
  "completedDocuments": 1,
  "errorDocuments": 0,
  "phases": [
    { "phaseId": "transfer", "name": "Field Registry", "type": "transfer",
      "completed": 2, "running": 0, "pending": 0, "errors": 0 },
    { "phaseId": "extraction", "name": "Extraction", "type": "extraction",
      "completed": 1, "running": 1, "pending": 0, "errors": 0 }
  ],
  "errors": []
}

Re-run and data product

A finished pipeline can be re-run from a phase onward without redoing the work before it: pass the phase type to start from, and earlier phases reuse their stored cells. When the output looks right, create a data product from the run to make it deliverable.

Re-run from resolution onward
curl -X POST https://api.talonic.com/v1/pipelines/pl_x8k2m9/rerun \
  -H "Authorization: Bearer $TALONIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "from_phase": "resolution" }'
Create a data product from the run
curl -X POST https://api.talonic.com/v1/pipelines/pl_x8k2m9/data-product \
  -H "Authorization: Bearer $TALONIC_API_KEY"
Held and blocked cells surface with status only and a null value in data-product reads, the public share page, and CSV exports, until a reviewer resolves them. This holdback is enforced at read time, so an in-review value can never leak into delivered output.

Frequently asked questions

How do I track a running pipeline?+
Poll GET /v1/pipelines/{id}/progress. It reports overall status and per-phase counts (pending, running, completed, errors) derived from the Spec rail, plus an errors array with per-document messages. Results are progressive, so you can begin reviewing completed documents while others are still processing.
Why did a document finish as "partial"?+
A document finishes partial when it has fields blocked at a validation gate, held for declarative review, or waiting on a dependency. Its clean fields are complete; the flagged fields wait in the review queue for a human decision. The pipeline never ships a value that failed a blocking gate.
Can I re-run only part of a pipeline?+
Yes. POST /v1/pipelines/{id}/rerun with from_phase re-runs from that phase onward and reuses the stored cells from earlier phases. This is useful after changing a Data Policy or a gate: re-run from resolution or validation without repeating transfer and extraction.
How does a pipeline become deliverable?+
Create a data product from the finished run with POST /v1/pipelines/{id}/data-product. The data product reads the latest canonical value per cell, owns its own review surface, and feeds delivery. Cells still in review surface with status only until resolved, so delivered output is always reviewed.
How do I find which pipeline processed a specific value?+
Use the "Find a value…" search on the Pipelines overview. Type the value and press Enter to see every pipeline whose cells contain it, with per-pipeline document counts; scope the search to a specific Spec field with the field selector, or use the Matched in annotations on unscoped hits to discover which field carried it.

Related

Review Queue
Assembly
Data Products