Reference Data

Reference data is a lookup table, uploaded as a CSV or Excel file, that the matching engine and schema reference strategies compare extracted document data against. These datasets are your system of record: the known records such as customer lists, product catalogs, vendor registries, and contract databases. Each reference dataset is versioned independently and can be shared across multiple schemas and matching configurations without duplication.

When you upload a reference dataset, the platform indexes all columns and rows for fast lookup during matching runs. Because each dataset is versioned independently, you can update your reference data without affecting in-progress matching configurations. Besides file upload, you can import reference data directly from a connected SQL database (MSSQL or PostgreSQL) from the reference data page: the import runs asynchronously, streaming rows in batches while column headers appear immediately so you can preview the structure.

Reference data lives as the Reference Data tab of the Documents page (?tab=reference-data), with a standalone route at /sources/reference-data for deep links. The list shows imported datasets and active live SQL connections side by side, each row carrying its row count, columns, status (importing, ready, or failed — large file parses run in the background, and the list polls until they settle), and a Used in column showing where the dataset is consumed.

Tags and usage counts

Datasets carry user-assigned tags for organizing large libraries. The list has a Filter by tag row of chips with OR semantics — selecting several tags shows every dataset carrying any of them — and matching is case-insensitive, so datasets that spell a tag differently still group together. Tags are edited inline on the dataset detail page: chips with a per-tag remove button plus an Add tag input that commits on Enter. Writes replace the full tag set (the server trims entries, drops empties, and dedupes case-insensitively, keeping the first-seen casing), with up to 32 tags of at most 64 characters each per dataset.

The Used in column answers "what breaks if I change this" at a glance. It shows one chip per non-zero facet — the specs whose schema fields or reference binding cite the table, the matching stages (legacy matching configurations) that score against it, the policies whose current version has a lookup rule referencing it, and the gates (validation stages) that compare against it — and a dash when nothing uses the dataset. The counts are advisory list decoration computed best-effort: a failed usage sweep never takes the reference-data list down with it.

Live connections and the table browser

Active SQL source connections appear in the list as Live connection rows — unlike an imported dataset, the connection itself is live rather than a copy. Each live row shows the host and database, its table count, an estimated row count taken from the database catalog (never a full scan), and how many snapshots and saved queries the workspace has built from it. The stats load lazily, one probe per connection, so a failed probe leaves that row showing dashes without blocking the others. Live rows carry no tags, so an active tag filter hides them along with any other untagged row.

The Browse action on a live connection opens the live table browser. A Table dropdown above the grid switches tables in place, keeping the URL in sync for deep links, and the Search this table box filters server-side with a case-insensitive match across every column — it searches the whole table, not just the loaded page. Pages are unordered catalog reads (the backend never sorts or counts the full table), capped at 50 rows per page, and switching tables clears the active search. Datasets imported from SQL remain one-shot snapshots by contrast: the list shows how long ago each synced and flags a snapshot older than 30 days as stale.

Preparing and refreshing datasets

For best results, ensure your reference data is clean and deduplicated before uploading. Include all columns that you plan to match against, such as names, identifiers, dates, and amounts. Most teams refresh their reference data periodically by re-uploading from their source system or by re-running the SQL import to pull directly from a connected database.

# The API accepts JSON rows directly; CSV/XLSX files are uploaded in the app.
curl -X POST https://api.talonic.com/v1/reference-data \
  -H "Authorization: Bearer $TALONIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "vendor_registry",
    "columns": ["vendor_id", "vendor_name", "country", "tax_id"],
    "data": [
      { "vendor_id": "V-001", "vendor_name": "ACME Corporation", "country": "DE", "tax_id": "DE123456789" },
      { "vendor_id": "V-002", "vendor_name": "Globex GmbH", "country": "DE", "tax_id": "DE987654321" }
    ]
  }'

# Response (201):
# {
#   "id": "b8c9d0e1-…",
#   "name": "vendor_registry",
#   "source_type": "json",
#   "row_count": 2,
#   "columns": ["vendor_id", "vendor_name", "country", "tax_id"],
#   "created_at": "2026-04-22T10:00:00Z"
# }

# Dataset metadata:
curl -s https://api.talonic.com/v1/reference-data/b8c9d0e1-… \
  -H "Authorization: Bearer $TALONIC_API_KEY"

# Rows, paginated (limit capped at 500 per page):
curl -s "https://api.talonic.com/v1/reference-data/b8c9d0e1-…/rows?page=1&limit=100" \
  -H "Authorization: Bearer $TALONIC_API_KEY"
# -> { "data": [ { "vendor_id": "V-001", ... } ], "pagination": { "page": 1, "limit": 100, "total": 2450 } }

• CSV and Excel (XLSX) file uploads in the app for quick one-time imports.
• SQL database imports for live reference data from connected MSSQL or PostgreSQL sources.
• JSON row creation via POST /v1/reference-data for programmatic loading.
• Versioning — each dataset tracks versions independently.
• Cross-schema sharing — one dataset can be referenced by multiple schemas and matching configurations.