Skip to main content

Replays & Verdict Diff

Re-run app logic over frozen historical episodes with replays, and compare a candidate version against a baseline with the camelCase verdict-diff endpoint.

Trust before autonomy: an app must be testable against history before it goes live. Replay re-runs a version's logic over the *frozen episode packages* of historical runs — the immutable inputs assembled at original run time, never live data — and diffs the decisions. A changed verdict therefore isolates a logic change; data drift cannot produce it. Replay is absolutely side-effect free: no journal writes, no run mutations, no outbox entries, no reviews — it only reads.

POST/v1/apps/:id/replays

Body

versionSelector*string | number"active" or an explicit version number — the logic that replays the episodes.
window*object{ from?, to?, runIds? } — ISO timestamps bounding run creation (inclusive); explicit runIds override the time window.
limitintegerMax runs replayed; hard cap 500. Default: 100

The result reports one row per run — the original outcome and decider beside the replayed outcome and any thresholds that would have blocked it — with a verdict of same, changed, original_missing, or replay_error, plus a summary carrying the agreement_rate (same / (same + changed)) and skipped_no_package, the count of runs in the window with no frozen episode (counted, never silently dropped). External-mode versions cannot replay (422): their decisions come from outside, so there is no resident logic to re-run. GET /v1/replays/:replayId exists for protocol parity but answers 404 in v1 — the diff arrives synchronously, so re-run the replay instead of polling.

Pre-publish verdict diff

POST /v1/apps/:id/replay/verdict-diff is the pre-publish confidence check: it replays a candidate version and a baseline version over the *same* frozen episodes of the app's last N sealed runs and reports which subjects change verdict and how each rule's pass/fail footprint moves. Reads only — the candidate is never persisted or activated, no actions fire, and nothing lands in the ledger. Supply the candidate as inline content (an unsaved manifest), as candidateVersionId, or supply neither and the app's current draft is the candidate; the baseline defaults to the active version.

This endpoint's request AND response are camelCase (candidateVersionId, lastNRuns, runsCompared, ruleDeltas…) — the deliberate exception to the snake_case convention on this surface, because the diff is one shared contract with the app screen that calls it. The LLM adjudicator is withheld on both sides so the diff is deterministic and free: below-threshold fuzzy pairs stay indeterminate on both sides and cancel out instead of inventing flips.

curl

curl -s -X POST https://api.talonic.com/v1/apps/$APP_ID/replay/verdict-diff \
  -H "Authorization: Bearer tlnc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "lastNRuns": 50 }'
# candidate = current draft, baseline = active version

Response (200)

{
  "runsCompared": 42,
  "flips": [
    { "runId": "8c41f3aa-0a92-4a0f-9d5e-2f60b0f7f2b1",
      "subjectKey": "L-2026-0851", "from": "failed", "to": "passed" }
  ],
  "ruleDeltas": [
    { "ruleId": "quantity-tolerance",
      "name": "Hold any load whose quantity exceeds 150 units.",
      "passesBefore": 198, "passesAfter": 205,
      "failsBefore": 9, "failsAfter": 2 }
  ],
  "summary": { "totalSubjects": 214, "flippedSubjects": 1 }
}

Read the diff in two passes. flips answers "which specific subjects would decide differently" — for a single-decision app, subjectKey is null and the run itself is the subject; a subject one side produced and the other did not (a changed group_by) shows as a flip from or to absent. ruleDeltas answers "which rule moved and how far": a rule whose fails collapse to zero after a threshold change is your change working; a rule whose passes drop unexpectedly is collateral damage the flip list will localize. Flips are capped at 1,000 returned rows; the true totals stay in summary.

The intended publish workflow: edit the draft, run the verdict diff against recent history, check the [acceptance set](app-acceptance), and only then [publish](app-versions). An empty flip list plus a passing acceptance check means the change does exactly what you intended and nothing else you can observe from history.

Frequently asked questions

Can a replay or verdict diff charge credits or trigger actions?+
No. Both are read-only by construction: they never touch the outbox, so no actions, webhooks, or reviews can result, and the LLM adjudicator is withheld from the verdict diff so it cannot meter adjudication either. Run them as often as you like.
Why were some runs skipped?+
Replay needs the frozen episode package assembled at original run time. Runs predating episode capture, or runs that failed before assembly, have none — they are counted in skipped_no_package (replay) or dropped from runsCompared (verdict diff) rather than being replayed against reconstructed data.
Why does the verdict diff reject my External-mode app?+
Both sides of the comparison must be versions whose decisions Talonic itself computes. An External version's decisions came from an outside agent — there is no resident logic to re-execute, so the endpoint answers 422 for either side being external.
How is the verdict diff different from a plain replay?+
Replay compares one version's replayed decisions against what originally happened. The verdict diff compares two versions against each other over the same episodes, at per-subject granularity with per-rule footprints — the sharper tool when deciding whether to publish a draft.

Related

Versions & Lifecycle
Acceptance Sets
Runs & the Verdict Matrix
Logic & Rule Compilation