Skip to main content

BENCHMARK

RAG cannot rank a corpus. We measured it.

We asked five cross-document ranking questions over 28 SEC 10-K filings: which company has the highest revenue, the lowest revenue, the most assets. Same corpus. Same model. Same parsing. The only difference: structure-first vs retrieve-first.

Talonic (structure-first)
5 / 5
RAG (BM25 top-12)
0 / 5

Why this happens

RAG answers from the documents it retrieves. For a lookup question that is fine: the answer lives in one document, and retrieval only has to find it. A ranking question is different. To know which of 28 companies has the highest revenue, a system has to have read all 28. A top-k retrieval window sees a handful and guesses from those.

Talonic captures every filing once at ingest into a canonical data plane: typed fields with provenance, resolved against a field registry. The ranking question then becomes an ordered scan over captured fields across every company. Ingest is paid once. Every query after that runs over structured data.

The proof, query by query

In each of the three headline queries, the correct filing was not even among the documents RAG pulled. It could not have answered correctly with any prompt.

QueryWhich company has the highest revenue?
RAG retrieved 7 documents
  • FiscalNote Holdings
  • TruBridge
  • Green Dot Corp
  • Rigel Pharmaceuticals
  • Steel Partners Holdings
  • 3D Systems
  • Cedar Realty Trust
  • Walmart Inc.: not retrieved
RAG answered: Green Dot Corp
Talonic scanned all 28 companies

Captured once at ingest. The query is an ordered scan over structured fields.

Talonic answered: Walmart Inc.
The correct filing was not among the documents RAG retrieved.
QueryWhich company has the lowest revenue?
RAG retrieved 10 documents
  • FiscalNote Holdings
  • TruBridge
  • Rigel Pharmaceuticals
  • Steel Partners Holdings
  • 3D Systems
  • Cedar Realty Trust
  • Green Dot Corp
  • Walmart
  • Park Hotels & Resorts
  • Incyte
  • Stirling Hotels & Resorts, Inc.: not retrieved
RAG answered: Cedar Realty Trust
Talonic scanned all 28 companies

Captured once at ingest. The query is an ordered scan over structured fields.

Talonic answered: Stirling Hotels & Resorts, Inc.
The correct filing was not among the documents RAG retrieved.
QueryWhich company has the highest total assets?
RAG retrieved 12 documents
  • Eastman Kodak
  • Lumen Technologies
  • 3D Systems
  • Westrock Coffee
  • TruBridge
  • Steel Partners
  • Rigel Pharmaceuticals
  • Target Group
  • Mainz Biomed
  • FiscalNote
  • SharkNinja
  • Park Hotels & Resorts
  • Walmart Inc.: not retrieved
RAG answered: Lumen Technologies
Talonic scanned all 28 companies

Captured once at ingest. The query is an ordered scan over structured fields.

Talonic answered: Walmart Inc.
Adversarial case: the largest company is not the most keyword-distinctive one, so retrieval misses it entirely.

Supporting queries

Name the top 3 companies by revenue.
Talonic: Walmart Inc., Genuine Parts Co, Lumen Technologies · RAG: Walmart, Green Dot, Incyte
Even when some answers are retrieved, RAG cannot rank documents it never saw. It fills the list with whatever it has.
Which company reported the largest net loss?
Talonic: 3D Systems Corporation · RAG: Lumen Technologies
RAG named the company it saw the most text about, not the actual largest loss. Net-loss gold is period-sensitive, so we treat this query as supporting rather than headline.

The cost line

RAG paid $0.11 in model cost for these five questions, measured from API usage tokens at pinned rates, and pays it again on every repeat. Talonic pays at ingest, once per document. Every ranking query after that runs over already-structured data.

The full battery: 39 queries, three arms

Beyond the five headline questions, we score a 39-query battery over the same corpus (53 filings, 28 companies): ranking, set membership, counting, and numeric reads, against SEC XBRL gold with a deterministic scorer. Hybrid RAG upgrades retrieval to BM25 plus dense embeddings with rank fusion over 24 chunks. The long-context arm is a steelman that feeds the whole corpus into full-context model calls.

TalonicHybrid RAGLong-context
Ranking questions correct87.5%16.7%33.3%
Set questions correct84.0%16.0%40.0%
Entity recall (mean)92.5%31.3%58.9%
Count questions exact75.0%25.0%83.3%
Numeric values correct100%0%100%
Denominators correct100%100%100%
Measured model cost per agent query$1.05$0.16$0.32
Model cost per correct ranking answer$1.20$0.96 at 16.7% accuracy$2.88
Query over structured data via API (no agent)~$0no structured datano structured data
One-time cost per document ingest$0.17$0$0

Read the table honestly: the long-context steelman is competitive on single-document numeric reads. It sees everything, at $0.32 per query and six-figure token counts per call, and it still fails the majority of corpus-ranking and set questions. Hybrid retrieval fails nearly all of them. The structural gap is in questions that require having read the whole corpus.

The cost rows are measured, both ways, and they price two different things. $1.05 is the natural-language agent: a multi-step tool loop (287 model calls across these 39 queries) that plans, reads structured fields, and writes an answer. The data-plane query underneath it is an API read over fields captured once at ingest, with roughly zero marginal model cost: programmatic consumers pay $0.17 per document once and then query for free. RAG has no such split. Its per-query price is its only query path, re-paid on every question, forever, over unstructured text.

Divide by correctness and the picture inverts. On ranking questions, Talonic costs $1.05 per query at 87.5% accuracy: $1.20 per correct answer. The long-context arm costs $0.32 at 33.3%: $2.88 per correct answer. The hybrid arm is cheap per attempt and wrong five times out of six. And this 53-filing corpus is the most favorable size the long-context arm will ever see: its cost grows linearly with corpus size until the corpus no longer fits at all, while a query over structured fields does not re-read the corpus.

Methodology

  • Corpus: SEC 10-K financial statements. Headline needle: 28 latest filings, one per company, fiscal year 2024. Full battery: 53 filings across the same 28 companies.
  • Gold: SEC EDGAR XBRL frames, the numbers each company itself filed. Deterministic scoring, values scale-normalized at 1% tolerance to the gold magnitude band. No LLM judges.
  • Parsing: identical OCR-to-markdown (Mistral OCR) fed to both arms in the needle benchmark, so parse quality is not the variable.
  • Models: Claude Sonnet 4.6 for both needle arms and both battery RAG arms. The battery Talonic column is the production agent over the public API (run 7, 2026-08-18), which routes across models internally.
  • RAG configs: needle arm BM25, top-k 12 chunks, one model call. Battery hybrid arm: BM25 + dense embeddings, reciprocal rank fusion, top-24 chunks. Long-context arm: whole-corpus partitioned full-context calls with merge.
  • Costs: measured, never estimated. RAG arms: API usage tokens at pinned rates. Talonic: the platform's metered model usage for this corpus and query run (2026-08-18), split into ingestion (one-time, $9.06 for 53 documents, all pipeline stages included) and agent queries ($41.05 for 39 queries).
  • Preregistration: protocol, corpus manifest, query set, configs, and price table frozen before any query ran. The RAG arms are fully reproducible from the benchmark scripts; the Talonic arm is auditable end to end.

Frequently asked questions

Is this benchmark cherry-picked?+

The headline result is deliberately narrow: it shows that retrieval-based RAG cannot answer questions that require ranking across a whole corpus, because it only reasons over the documents it retrieves. We do not claim Talonic beats RAG at everything. On single-document lookups the approaches are much closer, and the full 39-query battery on this page includes the metrics where a long-context baseline is competitive. The protocol, corpus manifest, query set, and configurations were frozen before any query ran.

Would a better retriever fix this?+

A denser retriever, hybrid fusion, or a reranker improves recall, and our hybrid baseline already combines BM25 with dense embeddings and reciprocal rank fusion over the top 24 chunks. But the ceiling is structural: a top-k window still cannot rank documents it never fetched. To know which of 28 companies has the highest revenue, a system has to have read all 28.

What exactly are the two RAG baselines?+

The hybrid baseline chunks every filing, retrieves with BM25 plus dense embeddings fused by reciprocal rank, and answers with one model call over the top 24 chunks. The long-context baseline is a steelman: it partitions the whole corpus into full-context model calls and merges the answers, at a measured average of $0.32 per query. The long-context arm reads in-document numbers well but still failed two of three corpus-ranking metrics.

What does structure-first mean?+

Talonic captures every filing once at ingest into a canonical data plane: typed fields with provenance, resolved against a field registry. A ranking question then becomes an ordered scan over captured fields across all companies, rather than a retrieval lottery. Ingest is paid once; queries afterwards run over already-structured data.

What was the gold standard?+

SEC EDGAR XBRL frames: the numbers each company itself filed with the regulator. Scoring is deterministic, with values scale-normalized and matched at 1% tolerance to the gold magnitude band. No LLM judges are involved in scoring.

Was the same model used for both sides?+

In the headline needle benchmark, yes: Claude Sonnet 4.6 for both arms, with identical OCR-to-markdown parsing, so retrieval strategy is the only variable. In the full battery, the RAG arms run Claude Sonnet 4.6 and the Talonic column is the production agent over the public API, which routes across models internally. That difference is stated because the battery compares products, not isolated model calls.

Can I reproduce these results?+

The RAG arms are fully reproducible from the benchmark scripts: corpus manifest, query set, retrieval configurations, and a pinned price table, all frozen before any query ran. The Talonic arm runs against the platform API and is auditable end to end. Contact us for the benchmark bundle.

Run the same question over your documents

Send a sample: a folder of contracts, a stack of filings, a corpus your team queries in spreadsheets. We will return a schema read, an accuracy estimate, and a concrete recommendation within five business days.