Asked which of 28 filings is biggest, RAG got 0 of 5 right.

Retrieval-based RAG fails an entire class of question: anything that requires having read the whole corpus. This is not a tuning problem, and a better retriever does not fix it. A top-k window cannot rank documents it never fetched.

• The headline. Five cross-document ranking questions over 28 SEC 10-K filings, identical model and identical parsing on both sides. Structure-first answered 5 of 5. BM25 retrieval answered 0 of 5. In three cases the correct filing was never retrieved at all, so no amount of reasoning over the retrieved set could have recovered it.
• At battery scale. Across 39 scored queries on 53 filings, all measured on 2026-08-25: ranking accuracy is 83.3% on Talonic’s retrieval fast path (20 of 24) and 79.2% on the Sonnet agent loop measured the same day on the same build (19 of 24), against 37.5% for the long-context steelman that reads the entire corpus and 12.5% for hybrid RAG. The two RAG figures are single draws: run the same battery ten times without changing anything and hybrid RAG lands anywhere from 10.3% to 23.1% overall, long context from 38.1% to 57.1%. The ranges are printed under the table, and they are the point of benchmark 03.
• The economics invert once you divide by correctness. The fast path costs $0.001526 per query at 83.3%, which is $0.0018 per correct ranking answer. The agent loop costs $0.24 at 79.2%, or $0.30. The long-context arm lands at $0.85 and hybrid RAG at $1.26, so the cheaper RAG arm per attempt is the more expensive one per answer. On the same build and the same 39 questions, the fast path is 156 times cheaper and 14 times faster than the loop it replaces, while answering one more question correctly. And neither is the only access path: programmatic consumers read the same structured fields over an API at roughly zero marginal model cost, after a one-time $0.17 per document at ingest.
• Where RAG still wins. The long-context arm still edges us on threshold counting, 83.3% of its 12 counting questions against 81.8% of our 11. That is its most stable row: across ten runs it stays between 71.4% and 85.7% there. We publish that row anyway; the deep dive below explains the failure and what it costs us.

Everything here is measured, not modelled: deterministic scoring against the numbers each company filed with the SEC, no LLM judges, protocol frozen before any query ran. And because a benchmark that only concludes “buy ours” is marketing, the end of this page tells you when to use retrieval or long context instead of us.

RAG answers from the documents it retrieves. For a lookup question that is fine: the answer lives in one document, and retrieval only has to find it. A ranking question is different. To know which of 28 companies has the highest revenue, a system has to have read all 28. A top-k retrieval window sees a handful and guesses from those.

Talonic captures every filing once at ingest into a canonical data plane: typed fields, resolved against a field registry. The ranking question then becomes an ordered scan over captured fields across every company. Ingest is paid once. Every query after that runs over structured data.

Two properties of that data plane matter beyond accuracy. Every captured field carries provenance back to the line in the filing it came from, and the agent cites it inline, so you can click a figure in an answer and land on the source. And the same structured fields serve every consumer: the natural-language agent measured here, and any copilot, pipeline, or API client your team points at the corpus, at near-zero marginal model cost.

Cost per query is the wrong unit, because a cheap wrong answer is not cheap. Divided by correctness, Talonic’s fast path and hybrid RAG are three orders of magnitude apart on this pass, and the gap widens with corpus size: the long-context arm re-reads everything on every question, while a query over structured fields never re-reads the corpus.

Read the table honestly: the long-context steelman is competitive on single-document numeric reads. It sees everything, at $0.32 per query and six-figure token counts per call, and it still fails the majority of corpus-ranking and set questions, on every one of the ten runs we measured. Hybrid retrieval fails nearly all of them. The structural gap is in questions that require having read the whole corpus.

The cost rows are measured, three ways, and they price three different things. $0.24 is the natural-language agent loop: a multi-step tool loop ($9.27 across these 39 queries) that plans, reads structured fields, and writes an answer. $0.001526 is the retrieval fast path, which compiles the question into a query over those same fields and runs it without a reasoning loop. The data-plane query underneath both is an API read over fields captured once at ingest, with roughly zero marginal model cost: programmatic consumers pay $0.17 per document once and then query for free. RAG has no such split. Its per-query price is its only query path, re-paid on every question, forever, over unstructured text.

And this 53-filing corpus is the most favorable size the long-context arm will ever see: its cost grows linearly with corpus size until the corpus no longer fits at all, while a query over structured fields does not re-read the corpus.

The rest of this page is the short version. This section is the long one, for readers who want to check our work rather than take it.

What “correct” means, exactly

Scoring is deterministic and regex-based. No model judges any answer. The entity universe comes from the corpus gold file, so the scorer never reads the filings and cannot be talked into an answer.

• Ranking (rank_set). We read the ranking the answer itself labels: for each rank 1..K we take the first corpus entity after an explicit rank marker (“#2”, “2.”, a table’s rank column), where K is the length of the gold list. An answer with no complete labeled ranking falls back to the first K corpus entities in mention order, the original extraction rule, which we also still score in full. On the fast path the two readers agree exactly, at 83.3% either way, because it emits a bare ranked list with no prose commentary for the labeled reader to recover. On the agent loop they diverge, 79.2% under the labeled reader against 58.3% under mention order, because the loop writes prose around its ranking. We print the labeled-reader figure for both. The labeled-ranking reader is calibrated: an earlier run scores identically under both. Ordering correct requires exact sequence equality; set correct compares the same prediction as an unordered set. Naming the right three companies in the wrong order scores zero on ordering, because a ranking answer that is not ordered is not a ranking answer.
• Entity recall. Intersection over gold size. This is the metric that separates “found the right companies” from “put them in the right order”, and the gap between the two is where our remaining errors live.
• Counting. Exact integer equality, with relative error tracked separately so a near-miss is visible rather than hidden inside a pass/fail rate.
• Numeric values. The first stated USD amount, scale-normalized, correct within a 1% relative tolerance of the figure the company filed.
• Denominators. A word-boundary match on the corpus size, which catches the common failure of answering confidently over a subset without saying so.

Why a better retriever does not close this

Retrieval quality is measured as recall@k: given a query, did the right chunks make the window. For a lookup question that is the whole game, and hybrid search, rerankers and bigger windows all move it. But a ranking answer is a function of every document in the corpus, not of the k most query-similar ones. To know which of 28 companies earned the most, a system needs all 28 revenue figures; retrieving the best 24 chunks yields a confident ordering over whichever subset arrived.

This is why the failure is silent, and why it is dangerous in production. The hybrid arm does not return “insufficient context”. It returns a specific company, in a fluent sentence, with no signal that the actual answer was never on the table. In our headline set the correct filing was absent from the retrieved documents in three of five questions. Adversarially, the largest company is often not the most keyword-distinctive one, so the very query that matters most is the one retrieval is most likely to miss.

What the two Talonic paths actually do per query

The agent loop is not a single model call. It plans, reads captured fields from the data plane, and writes an answer, capped at ten tool rounds. Across a 39-query battery it makes on the order of 230 tool invocations, a mean of six per query, and the hardest queries use the full cap. That loop is why its per-query cost is measured in cents rather than fractions of a cent: each round re-sends the accumulated transcript, which on 2026-08-25 came to $9.27 for the 39 questions, $0.24 each, at a mean latency of 127.6 seconds.

The retrieval fast path skips the loop. It compiles the question into a structured query over the captured fields, runs it deterministically, and returns the result with field-level citations. On the same build and the same 39 questions it cost $0.059496 in total, $0.001526 per query, at a mean latency of 9.1 seconds: 156 times cheaper and 14 times faster than the loop, and one question more accurate. That is the structural advantage made explicit. The fields were captured once at ingest, so answering from them is arithmetic rather than inference, and a programmatic consumer can query the same fields directly without paying for either path.

The fast path is enabled per tenant by a flag rather than shipped to everyone, and the loop is what a tenant without it runs today. Both columns are printed for that reason. One number is deliberately absent from the table: the scorer’s value-correctness metric reads the last labelled dollar amount in an answer, which on the fast path’s headline-first layout picks up a citation row instead of the verdict. It reports 0.0 for the fast path, and that figure is a scorer artefact rather than a result. The true relative errors on those two questions are 1.26% and 1.88% against a 1% tolerance, so they are genuine misses, but narrow ones. We treat the metric as unmeasured on any fast-path run and do not quote it.

Where we lose, and why

The long-context arm still edges us on counting, 83.3% of its 12 counting questions against 81.8% of our 11, though our mean relative error on the aggregation questions is now 1.46%. The mechanism behind a counting miss is straightforward: a threshold count is only as good as the least reliable field in the sweep. An arm that re-reads every filing in full context sidesteps that by recomputing from raw text each time, which is exactly why it costs more per query and stops working as the corpus grows.

The four ranking questions the fast path misses come down to two named defects, and both are in the capture rather than in the query path, so we can name them precisely.

Three of the four are one rule. STIRLING HOTELS & RESORTS files no revenue figure at all, and we capture that absence as a revenue of zero. Ranked as a genuine zero it takes the bottom slot on every bottom-of-the-corpus revenue question and pushes the real answer off the end. The benchmark gold sizes that universe at 27 companies rather than 28 for exactly this reason. Drop Stirling and all three answers match gold exactly. The fix is a rule we have not yet shipped: a concept a filing does not report is missing, not zero, and a missing value must not be ranked. The agent loop gets these three right, and they are the only three questions where it beats the fast path.

The fourth is a single bad cell. Park Hotels & Resorts has its total assets captured as $207 where the filing reports about $9.16 billion. That one number drops the company out of a top-five answer and takes an over-$1-billion count from 12 to 11. It is a capture error rather than a retrieval or reasoning error, it predates the fast path, and it fails identically on every Talonic run we have measured. We are printing it because a benchmark that hides its known-bad cells is not measuring anything.

Our ranking errors have a different shape from RAG’s, and it is visible in the gap between 93.1% entity recall and 83.3% ordering. We are almost always finding the right companies and occasionally sequencing them wrong, which is a near-miss against a scorer that awards nothing for proximity. RAG’s hybrid arm has the opposite problem: at 34.5% entity recall it is mostly not finding the companies at all.

Both numbers are honest limits rather than rounding noise, and both are the kind that move with field-capture coverage rather than with a different prompt.

A benchmark that only concludes “buy ours” is marketing. Here is the honest decision rule, including the cases where you should not use us.

• Use retrieval when the answer lives inside one document and the job is finding it: policy lookup, clause search, support over a large knowledge base. Retrieval is cheaper per query and the ceiling we describe here never binds, because the answer was never a function of the whole corpus.
• Use long context when the corpus genuinely fits, the questions are whole-corpus reads, and you are willing to re-read everything on every question. It is a real answer at this scale, and it beat us on counting. Its cost grows linearly with corpus size until the corpus stops fitting, at which point it stops being an option rather than becoming a slow one.
• Use structure-first when questions aggregate across documents (rank, count, compare, reconcile, “which of these”, “how many”, “across all”), when the same corpus is queried repeatedly, when you need per-field provenance for an audit, or when systems rather than people are the consumer. Ingest is paid once; queries afterwards run over typed fields.

If your questions contain superlatives, thresholds, counts or comparisons across documents, a retrieval system will answer them fluently, confidently, and wrongly, and nothing in its output will tell you which of those three it was doing.

The practical tell is the shape of your questions, not the size of your corpus.