Acceptance Sets

Give apps a machine-checkable definition of done: labeled expected verdicts with a tolerance, scored against any completed run as a pure ledger comparison.

An acceptance set is an app's machine-checkable definition of done: a list of labeled expectations plus a tolerance. Each case states what the app *should* decide — either per subject (subject_key + expected_outcome, optionally expected_auto) for batch apps, or per run (expected_decision) for single-decision apps — with an optional note explaining why. The check then scores a real run against the set as a pure ledger comparison: no model is involved, so the same run against the same set always scores identically.

GET /v1/apps/:id/acceptance returns the saved set or { "acceptance": null }. PUT /v1/apps/:id/acceptance replaces it wholesale — validation is total, never a partial save: a malformed case anywhere rejects the whole request with 422 naming the offending case (cases[0]: needs either subject_key + expected_outcome, or expected_decision). tolerance is the accepted mismatch share in [0,1]; a check passes when score >= 1 - tolerance, so tolerance 0 demands a perfect match and 0.1 allows one mismatch in ten evaluated cases.

GET /v1/apps/:id/acceptance/check scores the latest completed run by default, or a specific one via ?run_id=<uuid>. The response reports evaluated, matched, and skipped counts, the resulting score, the set's tolerance, the boolean pass, and a mismatches array pairing each expectation with what the ledger actually shows — the exact list a builder iterates on. Cases whose subject did not appear in the run are counted as skipped, not failed.

This is the loop a builder agent runs: draft logic, trigger a [dry run](app-runs), check acceptance, adjust, repeat — and the loop a maintainer re-runs after every edit, because the set survives version changes. Checking requires only the read tier; saving the set is operate. Checking before any set is saved answers 409 with "This app has no acceptance set yet — save one before checking."

curl -s -X PUT https://api.talonic.com/v1/apps/$APP_ID/acceptance \
  -H "Authorization: Bearer tlnc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "tolerance": 0,
    "cases": [
      { "subject_key": "L-2026-0847", "expected_outcome": "passed", "expected_auto": true },
      { "subject_key": "L-2026-0851", "expected_outcome": "failed",
        "note": "Rate mismatch above 2% must never auto-bill." }
    ]
  }'

{
  "run_id": "a6d56a57-bc82-439f-a547-342a8ea81dd4",
  "evaluated": 2,
  "matched": 1,
  "skipped": 0,
  "score": 0.5,
  "tolerance": 0,
  "pass": false,
  "mismatches": [
    {
      "subject_key": "L-2026-0851",
      "expected": "failed",
      "actual": "passed",
      "note": "Rate mismatch above 2% must never auto-bill."
    }
  ],
  "warnings": []
}