Skip to main content

Pipeline Progress

Poll phase-by-phase progress for a pipeline run: per-phase document counts (completed, running, pending, errors) plus per-document error detail.

Poll a pipeline run's phase-by-phase progress. The response reports overall document counts plus one row per phase, each carrying how many documents have completed, are running, are pending, or errored in that phase. Phase rows are derived from the run's actual compiled phase config, not a hardcoded list, so a run with no transfer phase never shows a phantom "transfer" row stuck at pending.

Each document moves through the per-document phases sequentially: transfer, extraction, resolution, then validation checkpoints. Pipeline-scoped phases such as assembly run once after every document is terminal, so they report a single status and scope: "pipeline" instead of per-document counters. The errors array surfaces per-document failures, including documents that errored before reaching any phase, so no failure is invisible.

This endpoint is the polling target after POST /v1/pipelines. Poll every few seconds and watch the per-phase counters advance as documents flow through the rail. When completedDocuments plus errorDocuments reaches totalDocuments, the run is terminal and you can produce a data product. This endpoint requires an API key with the read scope.

A phase row with scope: "pipeline" (such as assembly) reports its own status field. Its per-document counters are approximations for older consumers; trust the status value for pipeline-scoped phases.
GET/v1/pipelines/{id}/progress

Response

Response fields

pipelineIdstringPipeline run UUID.
statusstringOverall pipeline status.
totalDocumentsintegerTotal documents in the run.
completedDocumentsintegerDocuments that reached a terminal completed state.
errorDocumentsintegerDocuments that errored.
phasesarrayOne entry per compiled phase.
phases[].phaseIdstringPhase identifier.
phases[].namestringHuman-readable phase name.
phases[].typestringPhase type (transfer, extraction, resolution, validation, assembly).
phases[].completedintegerDocuments that completed this phase.
phases[].runningintegerDocuments currently in this phase.
phases[].pendingintegerDocuments not yet at this phase.
phases[].errorsintegerDocuments that errored in this phase.
phases[].scopestringOptional. "pipeline" for cross-document phases like assembly; otherwise document-scoped.
phases[].statusstring | nullOptional. Pipeline-scoped phases report their own status here instead of per-document counters.
errorsarrayPer-document error detail (includes docs that errored before any phase).
errors[].pipelineDocumentIdstringPipeline document UUID.
errors[].documentIdstringSource document UUID.
errors[].phasestring | nullPhase the document was in when it errored.
errors[].messagestring | nullError message.

Response

{
  "pipelineId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "active",
  "totalDocuments": 24,
  "completedDocuments": 15,
  "errorDocuments": 1,
  "phases": [
    { "phaseId": "transfer", "name": "Transfer", "type": "transfer",
      "completed": 24, "running": 0, "pending": 0, "errors": 0 },
    { "phaseId": "extraction", "name": "Extraction", "type": "extraction",
      "completed": 23, "running": 6, "pending": 0, "errors": 1 },
    { "phaseId": "resolution", "name": "Resolution", "type": "resolution",
      "completed": 16, "running": 7, "pending": 1, "errors": 0 },
    { "phaseId": "validation", "name": "Validation", "type": "validation",
      "completed": 15, "running": 2, "pending": 7, "errors": 0 },
    { "phaseId": "assembly", "name": "Assembly", "type": "assembly",
      "scope": "pipeline", "status": null,
      "completed": 0, "running": 0, "pending": 24, "errors": 0 }
  ],
  "errors": [
    {
      "pipelineDocumentId": "pd_uuid_9",
      "documentId": "doc_uuid_9",
      "phase": "extraction",
      "message": "Extraction failed after 3 attempts."
    }
  ]
}

Errors

Error responses

400validation_errorInvalid pipeline ID format. Must be a UUID.
401unauthorizedMissing or invalid API key.
404not_foundNo pipeline with this ID exists for your organization.
429rate_limitedToo many requests. Retry after the period indicated in the Retry-After header.

Related

Run Pipeline
Get Pipeline
Create Data Product