Skip to main content

Export Bundle Format

The tenant export produces a single zip bundle containing everything a workspace holds: original files, extracted content, structured values with provenance, and the complete audit log, each entry hashed into a manifest so the receiver can verify the bundle file by file. The layout follows the spirit of the BagIt packaging convention: payload directories plus a top-level checksum manifest and a bag-info metadata file. The format is versioned; this page specifies format version 1.

Bundle layout (v1)

ParameterTypeDescription
/originals/<docid>-<filename>payloadThe original ingested bytes of each document, named by document ID plus a sanitized filename.
/markdown/<docid>.mdpayloadThe extracted markdown of each document that has one.
/values/<docid>.jsonlpayloadOne JSON object per line per stored value cell: the latest value-plane cells with provenance for that document.
/audit/audit-events.csvpayloadThe workspace audit log as CSV (capped at 50,000 rows in v1).
/bag-info.jsonmetadataBundle metadata: format version, workspace, generation time, and per-payload counts.
/manifest-sha256.txtmanifestOne line per bundle entry: the SHA-256 hex digest, two spaces, the entry path. The manifest itself is the last file added and is not self-referential.
bag-info.json (v1 fields)
{
  "export_format_version": 1,
  "tenant": "<workspace-id>",
  "generated_at": "2026-07-18T04:12:09.331Z",
  "counts": {
    "documents": 3465,
    "originals": 3465,
    "missing_originals": 0,
    "markdown": 3441,
    "values": 3390,
    "audit_events": 48210
  }
}

Each line of a /values/<docid>.jsonl file is one value cell with its provenance: record_set_id, layer, record_id, field_key, data_type, the typed value, normalized_value, confidence, status, source, audit_ref, cell_version, and created_at. This gives the receiving system not just the data but where each value stands in the processing lifecycle and which audit reference produced it.

Request, poll, and download a bundle (admin only; step-up if enabled)
# Enqueue the export (returns the job row)
curl -X POST "$API_URL/records/tenant-export" \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Step-Up-Code: 123456"

# Poll until status is completed
curl "$API_URL/records/tenant-export/$JOB_ID" \
  -H "Authorization: Bearer $TOKEN"
# → { "status": "completed", "download_url": ".../records/tenant-export/<id>/download",
#     "manifest": { "bag_info": { ... }, "bundle_sha256": "…", "bundle_bytes": 1287340032, "entry_count": 10344 } }

# Stream the zip
curl -L "$API_URL/records/tenant-export/$JOB_ID/download" \
  -H "Authorization: Bearer $TOKEN" -o export.zip
Verify a received bundle
unzip -q export.zip -d export/
cd export/
# Verify every entry against the manifest (GNU coreutils)
sha256sum -c manifest-sha256.txt
# The job status also reports bundle_sha256 over the whole zip:
sha256sum ../export.zip

Bundles are built asynchronously on the worker fleet and stored server-side; the download streams through the authenticated API (there is no unauthenticated or presigned link). The job status carries the bundle's own SHA-256 and byte size, so the transfer itself is verifiable end to end: verify the zip hash first, then the per-entry manifest inside. Failed jobs stay failed and are re-requested explicitly; a partial bundle is never silently rebuilt over.

Export is admin-only and honors the workspace step-up requirement (a fresh TOTP code) when enabled. A bundle covers up to a configured document ceiling (50,000 by default); a corpus above the ceiling fails the job loudly instead of shipping a silently partial "full" export. Format version 1 is frozen: additive fields may appear in bag-info counts, but the layout and manifest format only change with a new export_format_version.

Frequently asked questions

How do I verify an export bundle is complete and untampered?+
Two layers: the job status reports the SHA-256 and byte size of the whole zip, and inside the bundle manifest-sha256.txt lists the SHA-256 of every entry. Verify the zip hash after transfer, then run sha256sum -c manifest-sha256.txt against the unpacked contents.
What exactly is included in a tenant export?+
Original files byte-for-byte, extracted markdown, the latest structured values with per-cell provenance as JSONL, and the workspace audit log as CSV, plus bag-info.json metadata and the checksum manifest. Deleted documents' tombstones appear through the audit log rather than as payload files.
Is the export format stable enough to build ingestion tooling against?+
Yes. The bundle carries export_format_version (currently 1) in bag-info.json. The v1 layout, entry naming, JSONL fields, and manifest format are frozen; incompatible changes will increment the version rather than silently change the shape.
Who can request an export, and is it protected against session hijacking?+
Workspace admins only, and workspaces with the step-up requirement enabled additionally demand a fresh TOTP code on the request. Every export request, completion, and failure is recorded in the audit trail, and the download itself requires an authenticated API call.

Related

Security Architecture
Archive Overview
Audit Trail
Retention & Deletion