BENCHMARK 03 · CONSISTENCY
Retrieval does not agree with itself. Structure does.
Same corpus, same model, same prompt, ten repeats. The hybrid RAG pipeline gave the same extracted answer all ten times on 2 of 39 questions. This page measures how often a system agrees with itself, which is a different property from being right, and it is the one nobody checks before signing.
2 / 39
Hybrid RAG · questions where all ten runs agreed
6 / 21
Long context RAG · questions where all ten runs agreed
39 / 39
Talonic fast path · 390 turns, one sha256, byte identical
n = 10 repeats of 39 questions for hybrid RAG, 10 repeats of 21 for long context RAG, 10 repeats of 39 for the Talonic fast path, 10 live passes of 100 for the structured path. At the vendor default temperature, 38 of hybrid RAG’s 39 questions produced ten different answers in ten runs over the same unchanged corpus, no two alike. The RAG rows were re-measured 2026-08-25 and both headlines came back unchanged; the temperature 0 rows and the Talonic rows are from 2026-08-24.
The result, in one minute
A retrieval pipeline is a sampler with a database bolted to the front. Two of its stages can move between runs, the retrieval and the generation, and the output gives no signal that either did. This benchmark asks the same questions repeatedly and counts how often the answer holds still.
- The headline. Over 10 repeats of 39 questions at the vendor default temperature, hybrid RAG gave the same extracted answer on all ten repeats for 2 of 39 questions. Long context RAG managed 6 of 21. Both figures are from a fresh ten-run matrix on 2026-08-25, and both are exactly what the same matrix produced three weeks earlier. The worst hybrid question, a bottom-five ranking by total assets, returned 9 distinct answers in 10 runs over an unchanged corpus.
- Both Talonic paths are flat, and one of them has a model in it. The structured path is 100 questions over 10 live passes of the public API, one sha256 digest, byte identical, $0. That one is deterministic by construction rather than by discovery: there is no model in the query path, so there is nothing to sample. The retrieval fast path is the interesting row. It is a natural-language surface, it compiles each question with an LLM, and across 10 repeats of the same 39 questions, 390 live turns, it returned one distinct sha256 and 0 of 39 queries varied. Not one character moved. It is enabled per tenant by a flag rather than shipped to everyone, so it is a real measured number on a production tenant but not yet every tenant’s number.
- Temperature 0 is not a fix. At matched repeats it takes hybrid RAG from 26.1 to 53.8 percent of questions where every repeat agreed and long context RAG from 42.3 to 52.4 percent. It buys no byte determinism: identical text on 0 of 39 questions and 1 of 21. The temperature 0 condition was measured on 2026-08-24 and not re-run on 2026-08-25, so that comparison crosses two run dates; the vendor-default side of it moved by about a point between them.
- The path this replaces was a sampler too. Before the fast path, a natural-language question ran the Talonic agent path, and across 3 repeats of 12 ranking questions it agreed on 7 of 12, with accuracy from 66.7% to 91.7% and a standard deviation of 10.4 points. n = 3 is a direction, not a rate, and we print it with its n rather than leaving it out. It sits last in the tables below, at full size, marked as the replaced baseline, and it is still what a tenant without the fast-path flag runs.
None of this is an accuracy claim. A system can be consistently wrong, and hybrid RAG largely is: 14.1% correct on average across its ten runs. Consistency is a separate property, and it is the one that decides whether a number you saw in a demo will still be there next quarter. It is also the property that decides what a single benchmark number is worth. Benchmark 01 quotes hybrid RAG at 12.5% and long context at 37.5% on a single pass taken the same day as these ten runs. Both sit inside the ranges below, and neither would have come back the same on the next pass.
Agreement across repeats
Three measures of the same runs, from strictest to loosest. Identical text is byte equality of the whole answer. Same extracted answer is mean pairwise agreement on the decision the scorer reads out, so a reworded sentence naming the same company counts as agreement. Every repeat agreed counts the questions where all repeats landed on one decision. Each row carries its own repeat count and question count, because they differ per arm and the rates are not comparable without them.
- The last column exists because an all-agreed rate rises mechanically as the number of draws falls. It is the 10 repeat runs averaged over every 3-of-10 subset, so it can be read against the temperature 0 rows, which have n = 3. Without it, comparing 2 of 39 at ten draws against 21 of 39 at three draws overstates the temperature effect.
- The Talonic structured path produced one distinct output digest across 10 passes of 100 questions: sha256 dafd39332dcd902d. The retrieval fast path produced one distinct digest across 10 repeats of 39 questions on 2026-08-25: sha256 a08bf19ab32b6900. The second of those is the one worth arguing about, because unlike the structured path it has a language model in it.
- Long context RAG’s higher all-identical rate is almost entirely counting questions: it agreed with itself on 5 of its 7 counting questions and on 1 of its 12 ranking questions. On ranking questions, which are 24 of the 39, the two RAG arms are indistinguishable, both at 8.3%. That split reproduced exactly on 2026-08-25.
The questions that work in the demo and fail in production
A burned question is one where the correct answer showed up in at least one repeat and failed in at least one other. It is the variance number that actually costs money, because nothing is broken and no amount of prompt engineering makes it reproducible. Burned counts are n-sensitive: more draws mean more chances to fail at least once, so the temperature 0 rows at n = 3 are not comparable with the n = 10 rows.
Read the always-right column next to the burned column. At the vendor default temperature, hybrid RAG was always right on 0 of 39 questions and burned 11. Long context RAG was always right on 6 of 21 and burned 8. Almost half of long context RAG’s questions were in the state where a buyer sees a correct answer in an evaluation and a wrong one in production, with no way to tell the two situations apart from the output.
The single worst question in the set is worth naming. At the vendor default temperature, a bottom-five ranking by total assets returned 9 distinct answers in 10 runs of the same question over the same unchanged corpus, and got none of them right. One level down, at raw text rather than extracted answer, the picture is starker still: on 38 of the 39 questions no two of the ten runs returned the same words.
What the variance does to the accuracy number
If a pipeline’s answers move, its score moves with them, and a single-run accuracy figure is a draw from a distribution rather than a measurement. Here is that distribution for every arm, with its repeat count and question count attached.
Bars are drawn on a 0 to 100 percent axis. The two zero-spread bars are drawn at a floor of 1 percent because a point has no width. The temperature 0 arms are omitted from the strip because they have n = 3 against n = 10 and are not comparable at face value; they are in the table below with their n.
Hybrid RAG’s accuracy across ten runs ranges from 10.3% to 23.1%, a factor of 2.2 between its worst and best run on identical inputs. Long context RAG ranges from 38.1% to 57.1%, a span of 19 points. Quote either one from a single run and you have quoted a coin flip. This is not a hypothetical: the single-pass figures on benchmark 01, 12.5% and 37.5%, and the ones it published three weeks earlier, 16.7% and 33.3%, are four draws from these two distributions and all four sit inside them.
The structured path’s standard deviation is exactly zero, at 87.2% on the 39 question battery, because accuracy there is a property of the capture rather than of the run. That cuts both ways, and it is the honest reading: a capture error does not average out over repeats either. The three accuracy numbers on this page come from three different batteries and are not comparable as printed. Matched to the same battery, the structured path scores 90.5% against long context RAG’s 48.6% on the 21 questions, and 91.7% against the Talonic agent’s 77.8% on the 12.
Corpus drift, and the control that changes the conclusion
We added five mid-size companies and their most recent 10-K to the corpus: 53 documents become 58, 28 companies become 33. Gold was recomputed from XBRL for the 58-document set and the same 39 question ids regenerated against it, so for every question we know whether the correct answer actually moved. It moved on 14 and did not move on 25. The drift rate below counts only the questions whose true answer did not change, against a control of rerunning each pipeline on the completely unchanged corpus.
The finding is not the one we went looking for, and we are publishing the one we got. Adding five documents changed 48.0% of hybrid RAG’s truth-stable answers, but rerunning it on an unchanged corpus already changed 37.2%. That excess is 10.8 points with a bootstrap 95% interval of [-4.4, 32.0] and p = 0.20: not distinguishable from rerun noise at this n. Long context RAG has no excess at all on the 2026-08-25 run. It changed 30.8% of its truth-stable answers after the ingest against a control of 31.5%, a difference of -0.8 points, interval [-22.3, 23.1], p = 1.0. We are not claiming an ingest effect for either arm, and specifically not for long context, where this run measured none.
So the claim is not that ingesting documents breaks retrieval answers. It is that rerunning retrieval on a completely unchanged corpus already changes about a third of its answers, which means an operator watching these outputs cannot tell an ingest from a rerun. There is no signal to alert on, because every signal is already firing. Both arms do detect the ingest: hybrid RAG moved on 11 of the 14 questions whose truth moved, and long context on all 8 of its 8. What neither can do is move only on those.
The Talonic row is a construction, not a measurement, and it should be read as one. No ingest was run: the benchmark tenant stayed at its pristine 53 documents, so this says nothing about how Talonic captures the five new filings. What is shown is the structured executor over the two field tables corresponding to the two corpora. It changed exactly 14 of 39 answers, which is exactly the set whose truth changed, and nothing else. The executor reads the table and nothing else, so an answer moves if and only if the rows behind it moved.
One further asymmetry, in RAG’s favour on the read and against it on the measurement: the 58-document run also received a reworded prompt, since the preamble states 33 distinct companies instead of 28 and every count question restates its denominator, while the control received no treatment at all. The measured delta is therefore an upper bound on the ingest effect.
What this benchmark does not show
These caveats ship with the numbers. They are the same list the benchmark report carries, transcribed rather than summarised, because a variance claim without its n is not a claim.
- 01 · Fast path is flag-scopedThe Talonic retrieval fast path is a natural-language surface with a language model in it, and across 10 repeats of 39 questions it returned one distinct sha256 with 0 of 39 queries varying. It is enabled per tenant by a flag, so this is a measured number on a production tenant rather than the default every tenant gets today. The 390 turns cost $0.266977 in total, and only 167 of them made a model call at all: a repeated question is served from an answer cache, which is part of why it does not vary.
- 02 · Structured path is deterministic by constructionThe Talonic structured path is deterministic by construction, not by luck. It runs no LLM at query time: it reads captured fields over the API and aggregates them in code, so identical input gives identical output. What the ten live passes add is not tautological: ten independent reads of the capture store returned identical field values for all 53 documents, and that is the part a caching or pagination bug would have broken.
- 03 · RAG arms ran at the vendor defaultThe frozen RAG arms send no temperature, so they ran at the Bedrock default of 1.0 for Anthropic models. Every published rag_b and rag_d number in the repo was produced at temperature 1.0, not 0.
- 04 · Temperature 0 is not a fixPinning temperature to 0 does not make a RAG pipeline deterministic. At temperature 0, hybrid RAG returned byte-identical text on 0 of 39 questions and long context RAG on 1 of 21. The temperature 0 arms have n = 3 against n = 10 at the vendor default, so the raw columns are not comparable at face value. Compared at the same three repeats, it cuts hybrid RAG’s disagreement by about a third (26.1 to 53.8 percent of questions where every repeat agreed) and long context RAG’s by about a quarter (42.3 to 52.4 percent). That comparison also crosses two run dates: the temperature 0 condition was measured on 2026-08-24 and was not re-run on 2026-08-25.
- 05 · The replaced agent path samples tooThe Talonic agent path is a sampler too. It plans and calls tools, and it agreed across 3 repeats on 7 of 12 questions. n = 3 over 12 questions is an indication of direction, not a rate with a tight interval, and it is not comparable at face value with the 10 repeat rows.
- 06 · Long context ran a 21 question subsetThe long context arm ran a 21 question subset, not the full 39, so its rates are not directly comparable with the hybrid arm’s. A hybrid view restricted to the same 21 questions is in the table.
- 07 · Drift is not distinguishable from rerunRerunning RAG on an unchanged corpus already changes about a third of its answers. Adding five documents is indistinguishable from that. The Talonic structured path changes no answer whose underlying rows did not change. Hybrid RAG’s drift excess over its own control is 10.8 points with a bootstrap 95 percent interval of [-4.4, 32.0] points, so a corpus change is not statistically distinguishable from rerunning the pipeline. Long context RAG showed no excess at all on the 2026-08-25 run, -0.8 points at p = 1.0, so no ingest effect should be read into that arm.
- 08 · The 58-document run also got a reworded promptThe 58-document run also received a reworded prompt (the preamble states 33 distinct companies instead of 28, and every count question restates its denominator) while the control received no treatment at all, so the measured delta is an upper bound on the ingest effect.
- 09 · What 83.3 percent measuresThe retrieval fast path’s 83.3 percent in the spread table is its v5 ranking accuracy over 24 ranking questions on 2026-08-25, and its standard deviation is zero because the ten repeats were byte-identical, not because it was rerun and happened to agree. Its value-correctness metric is not quoted anywhere on these pages: the scorer reads the last labelled dollar amount in an answer, which on a headline-first answer picks up a citation row rather than the verdict, so that metric is unmeasured on a fast-path run rather than zero.
- 10 · Structured path accuracy and normalisationThe structured path’s accuracy here is 87.2 percent on 39 questions and 75.0 percent on 100. It carries the same normalisation caveat as the cost benchmark: they share benchmark code, and a strict API-only row is lower. The determinism claim is what this section measures, not the accuracy level.
- 11 · A stricter resolver scores 63.0 percentA second, stricter field resolver scores the same executor at 63.0 percent on the 100 question battery instead of 75.0 percent, and it is byte identical across repeats too.
- 12 · Three batteries, three numbersThe three headline accuracy numbers come from three different batteries. Same battery comparisons: on the long context arm’s 21 questions the structured path scores 90.5 percent against long context RAG’s 48.6 percent; on the agent’s 12 questions the structured path scores 91.7 percent against the agent’s 77.8 percent.
- 13 · 9 of 28 companies have a cell outside tolerance9 of 28 companies have at least one captured cell that disagrees with the XBRL gold by more than tolerance. The corpus is staged extracts from SEC 10-K filings, not full filings.
- 14 · This page sits on two run datesThis page sits on two run dates and every row says which. The vendor-default-temperature RAG rows, the accuracy spread and the drift table were re-measured on 2026-08-25. The temperature 0 rows, the Talonic agent path and the Talonic structured path are from 2026-08-24 and were not re-measured, so any row-to-row comparison that crosses those dates is doing so across two runs of a pipeline that does not repeat itself.
Method
- Corpus: SEC 10-K filings staged as cover page plus income statement plus balance sheet, 53 documents, 28 companies. These are staged extracts, not full filings.
- Gold: XBRL companyfacts, matched to the accession each document is drawn from. Deterministic scoring, no LLM judges.
- Model:
global.anthropic.claude-sonnet-4-6on AWS Bedrock Converse, eu-central-1. Embeddings areamazon.titan-embed-text-v2:0, hybrid arm only. - Retrieval configs: hybrid arm (rag_b in the run logs) is BM25 plus Titan v2 dense embeddings with reciprocal rank fusion over the top 24 chunks. Long context arm (rag_d) puts the whole corpus in one context and sent a measured mean of 103,496 input tokens per call, with max output tokens 4,096.
- Temperature: the frozen arms send
inferenceConfig={maxTokens}and nothing else, so temperature is the provider default, which for Anthropic models on Bedrock Converse is 1.0. Every published hybrid and long-context number in the repo was therefore produced at temperature 1.0, not 0. The temperature 0 rows use an explicit override on the same arms. - Battery: 39 questions (
scaling/queries_n28_v4.jsonl). The long context arm runs a fixed 21 question subset because whole-corpus prompts are expensive. The structured path runs a 100 question battery that is a superset of the 39 the RAG arms ran. - Repeats: 10 at the vendor default temperature, 3 at temperature 0, 3 on the Talonic agent path over 12 ranking questions, 10 live repeats on the retrieval fast path, and 10 live passes on the structured path. Written out: 10 x 39 hybrid RAG, 10 x 21 long context RAG, 3 x 12 Talonic agent, 10 x 39 Talonic fast path, 10 x 100 structured.
- Drift test: one extra run against a 58-document corpus, compared against the 53-document majority answer, with a same-corpus rerun on the same truth-stable questions as the control.
- Dates and spend: the vendor-default-temperature RAG rows, the accuracy spread and the drift table were re-measured on 2026-08-25, at $169.32 of metered Bedrock spend with zero errored queries, on a corpus byte-identical to the originally published one. Both consistency headlines came back identical, 2 of 39 and 6 of 21. The temperature 0 rows, the Talonic agent path and the Talonic structured path are from the 2026-08-24 run ($181.28 of Bedrock spend across the RAG arms, plus $3.05 on the tenant credit ledger for the agent path) and were not re-measured. The structured path cost $0 on both dates.
- Verification: every number was re-derived by an independent adversarial review (
bench/verify-variance/VERDICT.md) and its corrections are applied here.
Reproduce it
The RAG arms are fully reproducible from the talonic-rag-benchmark suite: corpus manifest, query set, retrieval configurations and repeat counts were frozen before any query ran.
The second form re-derives results.json from the existing runs for $0. The structured path’s ten passes run against the public key-authed API, so they are auditable end to end. Contact us for the benchmark bundle.
Frequently asked questions
What exactly is being counted?+
Three different things, and the table prints all three. Identical text is byte equality of the whole answer. Same extracted answer is the mean pairwise agreement on the decision the scorer reads out of the answer, so a reworded sentence that names the same company counts as agreement. Every repeat agreed counts the questions where all repeats landed on one decision. The headline uses the third, which is the friendliest of the three to a retrieval pipeline.
Does setting temperature to 0 fix this?+
It helps and it does not fix it. Read at matched repeats, temperature 0 takes hybrid RAG from 26.1 to 53.8 percent of questions where every repeat agreed, and long context RAG from 42.3 to 52.4 percent. That is about a third off hybrid RAG’s disagreement and about a quarter off long context RAG’s, not a halving, and it buys no byte determinism at all: identical text on 0 of 39 questions and 1 of 21. Note also that the temperature 0 arms have n = 3 while the default arms have n = 10, which is why the matched column exists.
Is the Talonic structured path deterministic, or did it just get lucky?+
It is deterministic by construction. There is no model in the query path: the executor reads captured fields over the public API and aggregates them in code, so identical input gives identical output. That much is a tautology and we say so. What the ten live passes add is not: each pass re-fetched every field over the network, 54 GET requests per pass, and ten independent reads of the capture store returned identical field values for all 53 documents. Ten passes over 100 questions produced exactly one output digest, sha256 dafd39332dcd902d129d8bf1530e3a126a834c3e1cc9d541945fdae1cb927a85. A caching or pagination bug would have broken that.
Is the Talonic agent path deterministic too?+
No. The agent plans and calls tools, and it is a sampler like any other. Across 3 repeats of 12 ranking questions it agreed on 7 of 12, with accuracy ranging from 66.7% to 91.7% and a standard deviation of 10.4 points. We report it with its n rather than omitting it. It is also the path the retrieval fast path replaces: on the current build a natural-language question on a flagged tenant runs the fast path, which is byte-identical across repeats and is covered in its own answer below. The agent path stays on this page as the replaced baseline, at full size, and it is still what a tenant without the flag runs.
The fast path has a model in it. Why does it not vary?+
Because the model does not write the answer. It compiles the question into a structured query over the captured fields, and that query is then executed deterministically in code, so the only sampled step is the compilation and its output is a small fixed-shape query rather than prose. Across 10 repeats of 39 questions on 2026-08-25 the result was one distinct sha256 over all 390 turns, with 0 of 39 questions returning more than one distinct answer. Two honest qualifications. Repeats are served from an answer cache, so 390 questions produced only 167 compilations, which means the run measures cache behaviour as well as compiler stability. And the path is enabled per tenant by a flag, so it is a measured production number rather than the default every tenant runs today.
What is the burned metric?+
A burned question is one where the correct answer showed up in at least one repeat and failed in at least one other. It is the number that costs money in production, because it worked in the demo and broke later, and no amount of prompt engineering makes it reproducible: nothing is broken. At the vendor default temperature, hybrid RAG burned 11 of 39 questions and long context RAG burned 8 of 21. Burned counts are n-sensitive, since more draws mean more chances to fail at least once, so the temperature 0 rows at n = 3 are not comparable with the n = 10 rows.
What happened when documents were added to the corpus?+
Less than we expected, and the honest finding is not the one we set out to measure. Adding five filings changed 48.0% of hybrid RAG’s truth-stable answers and 30.8% of long context RAG’s. But rerunning the same pipelines on a completely unchanged corpus already changed 37.2% and 31.5%. Neither difference is distinguishable from rerun noise at this n: 10.8 points with a bootstrap interval of [-4.4, 32.0] and p = 0.20 for hybrid, and for long context no excess at all, -0.8 points with [-22.3, 23.1] and p = 1.0. We are not claiming that ingesting documents moves long context RAG’s answers, because on this run it did not. The finding is that rerunning RAG on an unchanged corpus already changes about a third of its answers, so an operator cannot tell an ingest from a rerun.
Was anything ingested on the Talonic side of the drift test?+
No, and that row is a construction rather than a measurement. The benchmark tenant stayed at its pristine 53 documents, so the test says nothing about how Talonic captures the five new filings. What it shows is the structured executor run over the two field tables that correspond to the two corpora: 14 of 39 answers changed, which is exactly the set whose true answer changed, and nothing else moved. The executor reads the table and nothing else, so an answer moves if and only if the rows behind it moved.
Does this measure accuracy?+
No. It measures how often a system gives the same answer, which is a different property, and a system can be consistently wrong. The accuracy figures on this page are there to give the variance a denominator: hybrid RAG averaged 14.1% correct across ten runs, long context RAG 48.6%, and the structured path 87.2% with a standard deviation of exactly zero. The accuracy comparison itself lives on the main benchmark page, and the structured path’s accuracy here carries the same normalisation caveat the cost benchmark carries.
The other two benchmarks
Ask your own questions ten times
Send a representative sample: a folder of contracts, a stack of filings, a corpus your team queries in spreadsheets. We will return a schema read, an accuracy estimate, and a concrete recommendation within five business days.