Pipeline Progress

Poll a document processing pipeline's progress: per-phase document counts (completed, running, pending, errors) plus per-document error detail for each run.

GET /v1/pipelines/{id}/progress reports a pipeline run's phase-by-phase progress: overall document counts plus one row per compiled phase, each carrying how many documents have completed, are running, are pending, or errored in that phase. Phase rows are derived from the run's actual compiled phase config, not a hardcoded list, so a run with no transfer phase never shows a phantom "transfer" row stuck at pending.

Each document moves through the per-document phases sequentially: transfer, extraction, resolution, then validation checkpoints. Pipeline-scoped phases such as assembly run once after every document is terminal, so they report a single status and scope: "pipeline" instead of per-document counters. The errors array surfaces per-document failures, including documents that errored before reaching any phase, so no failure is invisible.

This endpoint is the polling target after POST /v1/pipelines. Poll every few seconds and watch the per-phase counters advance as documents flow through the rail. When completedDocuments plus errorDocuments reaches totalDocuments, the run is terminal and you can produce a data product. This endpoint requires an API key with the read scope.

A Spec can place Resolution or Validation after Assembly in the rail. These compile to pipeline-scoped tail phases that run once over the composed record: a post-assembly resolution normalizes the assembled row (and mirrors the resolved values back so the data product ships them), and a post-assembly validation gate checks the full composed row, resolution first. Like assembly, these tail phases run after every document is terminal and are not surfaced as per-document progress rows, so do not wait on them in the counters — the run is terminal once completedDocuments plus errorDocuments reaches totalDocuments.

Response

{
  "pipelineId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "active",
  "totalDocuments": 24,
  "completedDocuments": 15,
  "errorDocuments": 1,
  "phases": [
    { "phaseId": "transfer", "name": "Transfer", "type": "transfer",
      "completed": 24, "running": 0, "pending": 0, "errors": 0 },
    { "phaseId": "extraction", "name": "Extraction", "type": "extraction",
      "completed": 23, "running": 6, "pending": 0, "errors": 1 },
    { "phaseId": "resolution", "name": "Resolution", "type": "resolution",
      "completed": 16, "running": 7, "pending": 1, "errors": 0 },
    { "phaseId": "validation", "name": "Validation", "type": "validation",
      "completed": 15, "running": 2, "pending": 7, "errors": 0 },
    { "phaseId": "assembly", "name": "Assembly", "type": "assembly",
      "scope": "pipeline", "status": null,
      "completed": 0, "running": 0, "pending": 24, "errors": 0 }
  ],
  "errors": [
    {
      "pipelineDocumentId": "pd_uuid_9",
      "documentId": "doc_uuid_9",
      "phase": "extraction",
      "message": "Extraction failed after 3 attempts."
    }
  ]
}

Errors