Pipeline Progress

Poll a document processing pipeline's progress: per-phase document counts (completed, running, pending, errors) plus per-document error detail for each run.

GET /v1/pipelines/{id}/progress reports a pipeline run's phase-by-phase progress: overall document counts plus one row per compiled phase, each carrying how many documents have completed, are running, are pending, or errored in that phase. Phase rows are derived from the run's actual compiled phase config, not a hardcoded list, so a run with no transfer phase never shows a phantom "transfer" row stuck at pending.

Each document moves through the per-document phases sequentially: transfer, extraction, resolution, then validation checkpoints. Pipeline-scoped phases such as assembly run once after every document is terminal, so they report a single status and scope: "pipeline" instead of per-document counters. The errors array surfaces per-document failures, including documents that errored before reaching any phase, so no failure is invisible.

This endpoint is the polling target after POST /v1/pipelines. Poll every few seconds and watch the per-phase counters advance as documents flow through the rail. When completedDocuments plus errorDocuments reaches totalDocuments, the per-document work is done — the pipeline then enters finalizing while the pipeline-scoped tail (assembly, post-assembly gates, review triage) runs, and only reaches completed when the tail is done. Wait for status: "completed" before producing a data product: POST /v1/pipelines/{id}/data-product rejects a still-finalizing run with 400. This endpoint requires an API key with the read scope.

While status is finalizing, three extra signals report tail health. finalizationPending names the tail units not yet done, validatedDocuments counts documents that already carry validation verdicts (it climbs during the window, so the page never looks idle), and tailStalled turns true when nothing has written to the run for two missed heartbeats — the run is wedged and only the recovery sweep will move it. A pinned per-phase counter alone is NOT evidence of a wedge; tailStalled is the authoritative signal.

A Spec can place Resolution or Validation after Assembly in the rail. These compile to pipeline-scoped tail phases that run once over the composed record: a post-assembly resolution normalizes the assembled row (and mirrors the resolved values back so the data product ships them), and a post-assembly validation gate checks the full composed row, resolution first. Like assembly, these tail phases run after every document is terminal and are not surfaced as per-document progress rows, so do not wait on them in the counters — the run is terminal once completedDocuments plus errorDocuments reaches totalDocuments.

curl https://api.talonic.com/v1/pipelines/a1b2c3d4-e5f6-7890-abcd-ef1234567890/progress \
  -H "Authorization: Bearer $TALONIC_API_KEY"

Response

{
  "pipelineId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "active",
  "totalDocuments": 24,
  "completedDocuments": 15,
  "errorDocuments": 1,
  "phases": [
    { "phaseId": "transfer", "name": "Transfer", "type": "transfer",
      "completed": 24, "running": 0, "pending": 0, "errors": 0 },
    { "phaseId": "extraction", "name": "Extraction", "type": "extraction",
      "completed": 23, "running": 6, "pending": 0, "errors": 1 },
    { "phaseId": "resolution", "name": "Resolution", "type": "resolution",
      "completed": 16, "running": 7, "pending": 1, "errors": 0 },
    { "phaseId": "validation", "name": "Validation", "type": "validation",
      "completed": 15, "running": 2, "pending": 7, "errors": 0 },
    { "phaseId": "assembly", "name": "Assembly", "type": "assembly",
      "scope": "pipeline", "status": null,
      "completed": 0, "running": 0, "pending": 24, "errors": 0 }
  ],
  "errors": [
    {
      "pipelineDocumentId": "pd_uuid_9",
      "documentId": "doc_uuid_9",
      "phase": "extraction",
      "message": "Extraction failed after 3 attempts."
    }
  ]
}

PIPELINE_ID="a1b2c3d4-e5f6-7890-abcd-ef1234567890"
while true; do
  STATUS=$(curl -s "https://api.talonic.com/v1/pipelines/$PIPELINE_ID/progress" \
    -H "Authorization: Bearer $TALONIC_API_KEY" | jq -r '.status')
  echo "status: $STATUS"
  [ "$STATUS" = "completed" ] && break
  sleep 3
done

# now safe to create the data product
curl -s -X POST "https://api.talonic.com/v1/pipelines/$PIPELINE_ID/data-product" \
  -H "Authorization: Bearer $TALONIC_API_KEY"

Errors