Skip to main content

Dataset Templates

A data product packages your extraction, resolution, and validation outputs into a shareable, deliverable dataset. Each product wraps one or more run_ids, a validation_run_id, or a pipeline_id, and moves through a status lifecycle of draft, ready, published, and archived (archiving is a soft delete, so the underlying data is retained).

The standalone Dataset Templates page is a non-functional stub. The earlier model where templates defined the output shape and assemblies combined sources is superseded. There is no dataset-template object and no separate template step: a data product is composed directly from its source runs, validation run, or pipeline.

Data products live under Data Products (/delivery/data-products), a tabbed shell that embeds the data-products list alongside a Delivery tab (the Delivery surface itself is hidden from the main nav). Open the list to see each product with its name, the schema it was built against, document counts, and its status.

Shaping the output

To shape the output of a product, configure the upstream pipeline rather than a separate template. Column order, renamed headers, excluded fields, and transforms are governed by the schema and the Resolution stage of the pipeline that feeds the product. Re-running the pipeline produces a fresh set of run values, and the data product reflects them on its next assembly.

Most teams keep one data product per downstream consumer. If your finance team and operations team need different views of the same documents, build two products from the appropriate runs (or pipelines) rather than reconfiguring a single export each time. Archive a product when it is no longer needed: the soft delete keeps its history available for audit while removing it from the active list.

  • A data product wraps run_ids[], a validation_run_id, or a pipeline_id
  • Status lifecycle: draft → ready → published → archived (archive is a soft delete)
  • Output shape is governed by the schema and pipeline, not a separate template object
  • Lives under Data Products (/delivery/data-products), a tabbed shell with the list plus a Delivery tab
  • The Dataset Templates page is a non-functional stub
  • One data product per downstream consumer is the recommended pattern
List data products, filtered by status
curl -s "https://api.talonic.com/v1/data-products?status=published" \
  -H "Authorization: Bearer $TALONIC_API_KEY"

# Response:
# {
#   "data": [
#     {
#       "id": "6a7b8c9d-…",
#       "name": "Q1 2026 Invoice Extract",
#       "description": "Finance handover set",
#       "schema_id": "…",
#       "run_id": "…",
#       "status": "published",
#       "created_at": "2026-04-01T08:00:00Z"
#     }
#   ],
#   "pagination": { ... }
# }
# status filter accepts: draft, ready, published, archived
Create a data product from completed runs
# The public API create path is run-backed; validation-session and
# pipeline-backed products are created from within the app.
curl -X POST https://api.talonic.com/v1/data-products \
  -H "Authorization: Bearer $TALONIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Q1 2026 Invoice Extract",
    "run_ids": ["d4e5f6a7-…"],
    "thresholds": { "min_confidence": 0.8 }
  }'
# -> 201: the created product, with its share token minted

The data product is the bridge between your pipeline and production-ready delivery. A schema defines what fields to extract; the pipeline (extraction, resolution, validation) governs how those fields appear in the final output. The product wraps the resulting runs (or a validation run, or a pipeline) so the assembled dataset is reproducible. To serve different consumers, build separate products from the appropriate runs rather than maintaining a separate template layer.

Shared group-grain data products

Beyond single-pipeline products, a shared data product is a standing, group-grain dataset that many pipelines feed over time. It is defined once — a name, a stable product_key slug that pipelines bind to, and a definition describing the record identity — and every bound pipeline then contributes its documents into it. The grain is the group: documents from any contributing pipeline that share the same identity key (an order number, a load ID, a contract reference) fold into one group, and one group is one row of the product. This is how a load's invoice, its rate confirmation, and its proof of delivery — each arriving through a different Spec — become a single composed record.

Contribution is keyed. Each contributing pipeline binds a key_field, and every document's key value is normalized (trimmed, inner whitespace collapsed, uppercased by default) before it is hashed into a group identity — so "ord-1042" and "ORD 1042" land in the same group. Each contribution records its evidence: the key field, the raw value, the canonical value, and the pipeline it came from. Contributions are append-only and idempotent — re-running a pipeline with unchanged data is a no-op, and changed data supersedes the earlier contribution rather than duplicating it. A document with no key value is, by default, parked as its own provisional single-document group (or held entirely, if the definition says so) rather than guessed into a group.

The definition can declare roles with cardinalities — say one invoice (exactly 1), one optional rate_confirmation (0..1), any number of pod documents (0..n) — plus a rollup naming which role wins each field on the composed row. Group readiness is then rule-driven: roles_complete requires the named roles to be present, no_open_conflicts requires no cardinality ceiling to be exceeded, and amount_match compares two amounts within a tolerance. Amount checks are role-scoped where both sides share one schema field name: left_role and right_role read the amount from a specific role's contribution, so an invoice's carrier_rate can be checked against the rate confirmation's carrier_rate even though both use the same field. Each group carries a state — provisional, conflict, incomplete, or ready — with the failing rule spelled out in its state detail.

On the Products list, a group-grain product's Records column counts its groups across states (a dash until the first group forms — never a fabricated zero), and per-corpus chips show how many distinct documents each contributing Spec has supplied. Opening the product lists its groups, filterable by state and corpus and searchable by key, each group showing its document count and which corpora contributed. For history that predates the binding, a Talonic administrator can run a backfill over completed pipelines — dry-run by default, scoped to one tenant and product — in plain contribute mode or in rematch mode, which re-runs matching against current reference data before contributing.

Governed read contracts

A cross-pipeline consumer — an App making decisions over the product, an external report — should not read whatever happens to be in the table today. A governed read contract freezes the product's published surface: the served fields with their types, nullability, and sensitivity labels, the record identity, the serve path, and a contract version tied to a schema hash. Fields hidden by suppress_output or demoted by a resolution policy (include: false) are excluded from the contract surface, so a consumer sees only what the product deliberately serves. Publishing a new contract supersedes the old one, and consumers can pin to the version they were built against.

The contract declares one record identity, in one of two ways: record_key_fields names 1-8 served fields that jointly identify a record, or record_key_anchored: true says the record set's own record ID is the identity (an anchored identity is never a join key). Declaring both is rejected with a 400 — a contract has one record identity. A schema whose fields already carry a record_key, business_key, or unique tag publishes with no body at all. When the product cannot be published — no served fields, an unconfigured or non-unique record key — the publish returns a 422 with a machine-readable code (for example source_record_key_unconfigured or source_record_key_not_unique) telling you what to fix.

Publish and read a governed read contract
# Publish (or re-publish) the contract, declaring the record identity:
curl -X POST https://api.talonic.com/v1/data-products/PRODUCT_UUID/read-contract/publish \
  -H "Authorization: Bearer $TALONIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "record_key_fields": ["order_number"] }'
# Declaring record_key_fields AND record_key_anchored is a 400:
# "…a contract has one record identity."

# Read the active contract:
curl -s https://api.talonic.com/v1/data-products/PRODUCT_UUID/read-contract \
  -H "Authorization: Bearer $TALONIC_API_KEY"
# -> { "contractVersion": "…", "schemaHash": "…",
#      "fields": [ { "fieldId": "…", "name": "order_number", "type": "string",
#                    "nullable": false, "sensitivity": "internal" } ],
#      "recordKeyFieldIds": ["…"], "recordKeyAnchored": false,
#      "publishedAt": "2026-08-20T09:00:00.000Z", "supersededAt": null }
# 404 (no active contract) tells you to publish one first.
Data products are workspace-scoped. Any team member can create, view, or archive a product. Archiving is a soft delete: the product leaves the active list but its history is retained for audit and can be referenced later.

Frequently asked questions

Is there a Dataset Templates feature?+
No. The Dataset Templates page is a non-functional stub. The model where templates defined the output shape and assemblies combined sources is superseded. A data product is composed directly from its source runs, a validation run, or a pipeline.
What does a data product wrap?+
A data product wraps one or more run_ids, a validation_run_id, or a pipeline_id. Its status moves through draft, ready, published, and archived, where archiving is a soft delete that retains the underlying history.
How do I shape the output of a data product?+
Configure the upstream pipeline. The schema and the Resolution stage govern column order, renamed headers, excluded fields, and transforms. Re-running the pipeline updates the run values the product assembles from.
How do I serve different downstream consumers?+
Build a separate data product per consumer from the appropriate runs or pipelines. There is no separate template object to maintain. The underlying extraction is shared; only the runs each product references differ.
Can I create a data product via the API?+
Yes. POST /v1/data-products with a name and one or more completed run IDs creates a run-backed product and mints its share token. Optional quality thresholds (min_confidence, require_validation_pass, require_approval) gate which values are included. Validation-session and pipeline-backed products are created from within the app.
What is a shared group-grain data product?+
A standing dataset many pipelines feed over time. Each contributing pipeline binds a key field; documents sharing the same normalized key fold into one group, and one group is one row. The definition can declare roles with cardinalities, a rollup of which role wins each field, and readiness rules such as a role-scoped amount match within a tolerance.
Why would I publish a read contract on a data product?+
To give cross-pipeline consumers like Apps a frozen, governed field surface instead of whatever the table holds today. The contract lists the served fields with types, nullability, and sensitivity, declares exactly one record identity (record_key_fields or record_key_anchored), and is versioned against a schema hash, so consumers can pin to the version they were built on.
What ends up in a group when a document has no key value?+
By default it becomes its own provisional single-document group, clearly separated from keyed groups, so nothing is guessed into the wrong group. A definition can instead hold keyless documents out of the product entirely. Either way the contribution evidence records the key field, the raw value, and the pipeline it came from.

Related

Assemblies
User Templates
How Delivery Works
Sharing & Export