Skip to main content

Export Bundle Format

The tenant export produces a single zip bundle containing everything a workspace holds: original files, extracted content, structured values with provenance, and the complete audit log, each entry hashed into a manifest so the receiver can verify the bundle file by file. The layout follows the spirit of the BagIt packaging convention: payload directories plus a top-level checksum manifest and a bag-info metadata file. The format is versioned; this page specifies format version 1.

Bundle layout (v1)

ParameterTypeDescription
/originals/<docid>-<filename>payloadThe original ingested bytes of each document, named by document ID plus a sanitized filename.
/markdown/<docid>.mdpayloadThe extracted markdown of each document that has one.
/values/<docid>.jsonlpayloadOne JSON object per line per stored value cell: the latest value-plane cells with provenance for that document.
/audit/audit-events.csvpayloadThe workspace audit log as CSV (capped at 50,000 rows in v1).
/bag-info.jsonmetadataBundle metadata: format version, workspace, generation time, and per-payload counts.
/manifest-sha256.txtmanifestOne line per bundle entry: the SHA-256 hex digest, two spaces, the entry path. The manifest itself is the last file added and is not self-referential.
bag-info.json (v1 fields)
{
  "export_format_version": 1,
  "tenant": "<workspace-id>",
  "generated_at": "2026-07-18T04:12:09.331Z",
  "counts": {
    "documents": 3465,
    "originals": 3465,
    "missing_originals": 0,
    "markdown": 3441,
    "values": 3390,
    "audit_events": 48210
  }
}

Each line of a /values/<docid>.jsonl file is one value cell with its provenance: record_set_id, layer, record_id, field_key, data_type, the typed value, normalized_value, confidence, status, source, audit_ref, cell_version, and created_at. This gives the receiving system not just the data but where each value stands in the processing lifecycle and which audit reference produced it.

Value cell fields (one JSON object per line)

ParameterTypeDescription
record_set_id / record_ididentityWhich record set the cell belongs to and which row within it. Cells for one document can span several record sets (for example a Job result and a Data Product).
layerlifecycleThe value layer the cell sits in: capture (raw discovery), structured (schema-guided), resolved (policy-normalized), or product (canonical output).
field_key / data_typeshapeThe field the cell fills and its declared type, so a receiving system can rebuild typed columns without guessing.
value / normalized_valuepayloadThe typed value as extracted, plus its normalized form where a Data Policy or dialect produced one.
confidence / status / sourceprovenanceThe confidence score, the cell status, and the origin of the value (how it was produced).
audit_ref / cell_version / created_atauditThe audit reference that produced the cell, its version number, and the server-side creation timestamp. Only the latest cell version per field is exported.

Consuming the values payload

Because the values payload is line-delimited JSON, you can process it with standard streaming tools without loading whole files into memory: one document is one file, one cell is one line. A typical rebuild pipeline walks /values/, filters to the layer it cares about (usually product for canonical output, or structured if the workspace does not run Data Policies), and pivots field_key into columns keyed by record_id. Keep audit_ref alongside each value if the receiving system needs to answer "where did this number come from" later: it links back to the audit log shipped in the same bundle.

Rebuild a flat table from the values payload
# All product-layer values across the bundle as field/value pairs
cat export/values/*.jsonl \
  | jq -r 'select(.layer == "product") | [.record_id, .field_key, (.normalized_value // .value | tostring)] | @tsv'

# Every low-confidence cell that still made it to the product layer
cat export/values/*.jsonl \
  | jq -c 'select(.layer == "product" and .confidence < 0.8)'

# Cross-check: which exported documents have no values payload at all
comm -23 <(ls export/originals | cut -c1-36 | sort -u) \
         <(ls export/values | cut -c1-36 | sort -u)

Operational patterns

The bundle is designed for four recurring situations. Scheduled escrow: request an export on a fixed cadence through the API, verify the zip hash and manifest on receipt, and store the verified bundle in your own retention infrastructure, so your archive obligations never depend on a single vendor. Offboarding and migration: the bundle is the complete, self-describing hand-back of a workspace, with originals byte-for-byte and values typed and layered. Regulator or auditor handoff: because every entry is hashed into the manifest and the job status reports the bundle hash, you can hand the zip plus two hash values to a third party and let them verify independently. Incident reconstruction: the audit CSV plus the per-cell audit_ref links let you replay who touched what without access to the live system.

Two boundaries are worth planning around. The audit CSV inside the bundle is capped at 50,000 rows in v1: a long-lived workspace with more history should pull the complete log through the audit-events export endpoint on a schedule and treat the in-bundle CSV as the recent window. And the export covers one workspace as it stands at generation time: documents soft-deleted under archive mode are represented through the audit log rather than as payload entries, so a bundle plus the live disposition certificates together give the full account of records that no longer have content.

Request, poll, and download a bundle (admin only; step-up if enabled)
# Enqueue the export (returns the job row)
curl -X POST "$API_URL/records/tenant-export" \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Step-Up-Code: 123456"

# Poll until status is completed
curl "$API_URL/records/tenant-export/$JOB_ID" \
  -H "Authorization: Bearer $TOKEN"
# → { "status": "completed", "download_url": ".../records/tenant-export/<id>/download",
#     "manifest": { "bag_info": { ... }, "bundle_sha256": "…", "bundle_bytes": 1287340032, "entry_count": 10344 } }

# Stream the zip
curl -L "$API_URL/records/tenant-export/$JOB_ID/download" \
  -H "Authorization: Bearer $TOKEN" -o export.zip
Verify a received bundle
unzip -q export.zip -d export/
cd export/
# Verify every entry against the manifest (GNU coreutils)
sha256sum -c manifest-sha256.txt
# The job status also reports bundle_sha256 over the whole zip:
sha256sum ../export.zip

Bundles are built asynchronously on the worker fleet and stored server-side; the download streams through the authenticated API (there is no unauthenticated or presigned link). The job status carries the bundle's own SHA-256 and byte size, so the transfer itself is verifiable end to end: verify the zip hash first, then the per-entry manifest inside. Failed jobs stay failed and are re-requested explicitly; a partial bundle is never silently rebuilt over.

Export is admin-only and honors the workspace step-up requirement (a fresh TOTP code) when enabled. A bundle covers up to a configured document ceiling (50,000 by default); a corpus above the ceiling fails the job loudly instead of shipping a silently partial "full" export. Format version 1 is frozen: additive fields may appear in bag-info counts, but the layout and manifest format only change with a new export_format_version.

Frequently asked questions

How do I verify an export bundle is complete and untampered?+
Two layers: the job status reports the SHA-256 and byte size of the whole zip, and inside the bundle manifest-sha256.txt lists the SHA-256 of every entry. Verify the zip hash after transfer, then run sha256sum -c manifest-sha256.txt against the unpacked contents.
What exactly is included in a tenant export?+
Original files byte-for-byte, extracted markdown, the latest structured values with per-cell provenance as JSONL, and the workspace audit log as CSV, plus bag-info.json metadata and the checksum manifest. Deleted documents' tombstones appear through the audit log rather than as payload files.
Is the export format stable enough to build ingestion tooling against?+
Yes. The bundle carries export_format_version (currently 1) in bag-info.json. The v1 layout, entry naming, JSONL fields, and manifest format are frozen; incompatible changes will increment the version rather than silently change the shape.
Who can request an export, and is it protected against session hijacking?+
Workspace admins only, and workspaces with the step-up requirement enabled additionally demand a fresh TOTP code on the request. Every export request, completion, and failure is recorded in the audit trail, and the download itself requires an authenticated API call.
Does the bundle contain the complete audit history?+
The in-bundle CSV is capped at 50,000 rows in format version 1. Workspaces with more history should pull the complete log through the audit-events export endpoint on a schedule and treat the bundled CSV as the recent window; the two sources carry identical events.