Ask the same question ten times.

A retrieval pipeline is a sampler with a database bolted to the front. Two of its stages can move between runs, the retrieval and the generation, and the output gives no signal that either did. This benchmark asks the same questions repeatedly and counts how often the answer holds still.

• The headline. Over 10 repeats of 39 questions at the vendor default temperature, hybrid RAG gave the same extracted answer on all ten repeats for 2 of 39 questions. Long context RAG managed 6 of 21. One hybrid question, a five-way ranking by net income, returned 10 distinct answers in 10 runs over an unchanged corpus.
• The Talonic structured path is flat. 100 questions, 10 live passes over the public API, one sha256 digest, byte identical, $0. That is deterministic construction rather than a discovery: there is no model in the query path, so there is nothing to sample. What the live passes prove is that ten independent reads of the capture store returned identical field values for all 53 documents.
• Temperature 0 is not a fix. At matched repeats it takes hybrid RAG from 27.0 to 53.8 percent of questions where every repeat agreed and long context RAG from 43.5 to 52.4 percent. It buys no byte determinism: identical text on 0 of 39 questions and 1 of 21.
• The Talonic agent path is a sampler too. Across 3 repeats of 12 ranking questions it agreed on 7 of 12, with accuracy from 66.7% to 91.7% and a standard deviation of 10.4 points. n = 3 is a direction, not a rate, and we print it with its n rather than leaving it out.

None of this is an accuracy claim. A system can be consistently wrong, and hybrid RAG largely is: 15.4% correct on average across its ten runs. Consistency is a separate property, and it is the one that decides whether a number you saw in a demo will still be there next quarter.

Read the always-right column next to the burned column. At the vendor default temperature, hybrid RAG was always right on 0 of 39 questions and burned 12. Long context RAG was always right on 6 of 21 and burned 8. Almost half of long context RAG’s questions were in the state where a buyer sees a correct answer in an evaluation and a wrong one in production, with no way to tell the two situations apart from the output.

The single worst question in the set is worth naming. At the vendor default temperature, a five-way ranking by net income returned 10 distinct answers in 10 runs of the same question over the same unchanged corpus.

Hybrid RAG’s accuracy across ten runs ranges from 7.7% to 20.5%, a factor of 2.7 between its worst and best run on identical inputs. Long context RAG ranges from 38.1% to 57.1%. Quote either one from a single run and you have quoted a coin flip.

The structured path’s standard deviation is exactly zero, at 87.2% on the 39 question battery, because accuracy there is a property of the capture rather than of the run. That cuts both ways, and it is the honest reading: a capture error does not average out over repeats either. The three accuracy numbers on this page come from three different batteries and are not comparable as printed. Matched to the same battery, the structured path scores 90.5% against long context RAG’s 45.7% on the 21 questions, and 91.7% against the Talonic agent’s 77.8% on the 12.

The finding is not the one we went looking for, and we are publishing the one we got. Adding five documents changed 48.0% of hybrid RAG’s truth-stable answers, but rerunning it on an unchanged corpus already changed 38.0%. Long context RAG: 38.5% against a control of 27.7%. Neither difference is distinguishable from rerun noise at this n. Hybrid RAG’s excess is 10.0 points with a bootstrap 95% interval of [-2.4, 28.8] and p = 0.21; long context RAG’s is 10.8 points with an interval of [-13.1, 35.4] and p = 0.37.

So the claim is not that ingesting documents breaks retrieval answers. It is that rerunning retrieval on a completely unchanged corpus already changes about a third of its answers, which means an operator watching these outputs cannot tell an ingest from a rerun. There is no signal to alert on, because every signal is already firing. Both arms do detect the ingest: hybrid RAG moved on 12 of the 14 questions whose truth moved. What neither can do is move only on those.

The Talonic row is a construction, not a measurement, and it should be read as one. No ingest was run: the benchmark tenant stayed at its pristine 53 documents, so this says nothing about how Talonic captures the five new filings. What is shown is the structured executor over the two field tables corresponding to the two corpora. It changed exactly 14 of 39 answers, which is exactly the set whose truth changed, and nothing else. The executor reads the table and nothing else, so an answer moves if and only if the rows behind it moved.

One further asymmetry, in RAG’s favour on the read and against it on the measurement: the 58-document run also received a reworded prompt, since the preamble states 33 distinct companies instead of 28 and every count question restates its denominator, while the control received no treatment at all. The measured delta is therefore an upper bound on the ingest effect.

• Corpus: SEC 10-K filings staged as cover page plus income statement plus balance sheet, 53 documents, 28 companies. These are staged extracts, not full filings.
• Gold: XBRL companyfacts, matched to the accession each document is drawn from. Deterministic scoring, no LLM judges.
• Model: global.anthropic.claude-sonnet-4-6 on AWS Bedrock Converse, eu-central-1. Embeddings are amazon.titan-embed-text-v2:0, hybrid arm only.
• Retrieval configs: hybrid arm is BM25 plus Titan v2 dense embeddings with reciprocal rank fusion over the top 24 chunks. Long context arm puts the whole corpus in one context and sent a measured mean of 103,496 input tokens per call, with max output tokens 4,096.
• Temperature: the frozen arms send inferenceConfig={maxTokens} and nothing else, so temperature is the provider default, which for Anthropic models on Bedrock Converse is 1.0. Every published hybrid and long-context number in the repo was therefore produced at temperature 1.0, not 0. The temperature 0 rows use an explicit override on the same arms.
• Battery: 39 questions (scaling/queries_n28_v4.jsonl). The long context arm runs a fixed 21 question subset because whole-corpus prompts are expensive. The structured path runs a 100 question battery that is a superset of the 39 the RAG arms ran.
• Repeats: 10 at the vendor default temperature, 3 at temperature 0, 3 on the Talonic agent path over 12 ranking questions, and 10 live passes on the structured path. Written out: 10 x 39 hybrid RAG, 10 x 21 long context RAG, 3 x 12 Talonic agent, 10 x 100 structured.
• Drift test: one extra run against a 58-document corpus, compared against the 53-document majority answer, with a same-corpus rerun on the same truth-stable questions as the control.
• Date and spend: run 2026-08-24. $181.28 of Bedrock spend across the RAG arms, plus $3.05 on the tenant credit ledger for the Talonic agent path. The structured path cost $0.
• Verification: every number was re-derived by an independent adversarial review (bench/verify-variance/VERDICT.md) and its corrections are applied here.

The RAG arms are fully reproducible from the talonic-rag-benchmark suite: corpus manifest, query set, retrieval configurations and repeat counts were frozen before any query ran.

The second form re-derives results.json from the existing runs for $0. The structured path’s ten passes run against the public key-authed API, so they are auditable end to end. Contact us for the benchmark bundle.