# Route only to models with quality evidence (experimental, opt-in) `quality-guarded` serves a cheaper model only where you have evidence it is good enough for the task. It keeps three things apart: **quality** (labels from a validator or a reviewed gold set, never latency, price, HTTP success or silence), **reliability** (the existing health, breaker and ladder machinery), and **cost** (declared prices, used only to rank models that are already qualified). **Status: source-complete and offline-verified only.** Unit, router and end-to-end tests on the fast suite pass and the safety checks are mutation-proven; the full estate, live evaluation, statistical qualification of any real model, enabled-mode endurance and release promotion are **NOT RUN**. Treat it as experimental. Design and acceptance mapping: [design guide](../design/adaptive-routing.md). ## What it does on each request 1. The group's strategy reads the **active quality policy** from memory (refreshed in the background; selection never touches the database or calls a learner, judge or classifier). 2. It picks the request's **task profile**: the group's default, or - only when that profile says it trusts caller hints - the profile a valid `X-Task-Type` names. A hint can never select a profile that does not accept hints. 3. **Qualified now** = the profile's qualified models whose evidence has not expired and whose registry row still matches the row the evidence was bound to. An expired policy, or an organization outside the profile's scope, qualifies nothing. 4. Among the candidates that survived every existing filter (permissions, capability, health, budget) and are qualified now, it picks the **lowest declared price**; an undeclared price ranks last, never as free. Optional exploration (at most 20%, off by default) samples among qualified models only, from trusted label counts; unknown counts switch it off. The counts are those of the request's own qualification: each label counts under the serving key recorded when its decision was served. That key binds organization, profile and its qualification epochs, rubric, model version, registry row and pipeline. Labels from an earlier qualification never steer a new one, and counts published under another policy version than the request's are unavailable, not zero. A profile may instead declare `"selection":"bandit"`: the qualified candidates (and only those) go to the reward-learning bandit, which prefers among them by observed outcomes - success, latency, cost, ratings - and adapts when a provider degrades or recovers. The quality boundary still binds every fallback hop. Without `pointsman.bandit.enabled=true` the profile ranks by price and counts `pointsman.quality.decisions{outcome="bandit_unavailable"}`. Rewards never qualify a model: qualification stays the manual, evidence-bound step above. `"selection":"cost"` instead hands them to the cost sampler: Thompson sampling on each qualified model's OBSERVED cost of a completed call (the registry's cache-aware price of the usage the provider reported), started from the declared prices. On day one it ranks as the prices do; when a model's bill drifts from its price (it writes or reasons more, a prompt-cache discount, repeated failures that are paid again on the fallback) it moves within a few hundred requests and comes back when that stops. State is per organisation and profile, in-process (one instance); a restart starts again from the prices. Latency is not in its score - use the qualification's `latency.max_ms` gate for that. It consumes no quality labels: "completed" means the provider answered, not that the answer was right, so a qualified model that later degrades is not detected - re-qualify on a schedule. Experimental and opt-in. A cost profile may add `"feedback_floor": 0.8`: labels posted with a response's `X-Quality-Receipt` to `POST /v1/quality/assessments` (first revisions only; voluntary or trusted) are counted per model with forgetting, and a model with at least 8 recent labels whose Wilson 95% upper bound on labelled accuracy is below the floor is no longer chosen (all of them below: the declared baseline serves). Labels only ever remove a qualified model, only for the labelling organisation's own profile; a removed model is probed again as its labels age. The removal is part of the request's quality boundary (17 October 2026), so it binds every path, not only the cost choice: exploration, session affinity, family preference and every fallback hop. The declared baseline chain is never removed. One request is judged against one policy version: the boundary the router computed before its shortcuts is the one the choice and the ladder use, and the decision record carries `feedback_floor` and `feedback_excluded`. 5. No qualified candidate survived: the **declared baseline**, then its declared fallbacks. None survived either: `503 quality_policy_unavailable`. Never a random or unqualified model. 6. Session affinity, family preference and **every fallback hop** are held to the same boundary; the group never escalates to a neighbouring group; the **response cache is bypassed**. 7. The response says what served it, and carries a signed receipt for later evidence. | Response header | Meaning | |---|---| | `X-Quality-Route: qualified; profile=; policy=` | a qualified model served | | `X-Quality-Route: baseline; profile=; policy=; reason=` | the declared baseline served; `r` is `none_qualified`, `no_qualified_eligible`, `evidence_stale`, `policy_expired`, `profile_out_of_scope` or `fallback_to_baseline` | | `X-Quality-Receipt: q1..` | cite this when submitting evidence; binds organization, decision, served model, profile, policy and serving conditions | A request with no policy loaded is refused with `503` and `code: quality_policy_unavailable` (`reason` in the message: `policy_absent`, `no_profile` or `no_eligible_baseline`). ## Turn it on ```bash POINTSMAN_QUALITY_ENABLED=true ``` and route a group to it in `routing.yml`: ```yaml pointsman: routing: groups: support-extract: strategy: quality-guarded candidates: - { provider: openai, model: gpt-4o-mini } - { provider: openai, model: gpt-4o } ``` Until a policy is promoted the group refuses every request. Every other setting, with its unit, bound and default, is in [the configuration reference](../reference/configuration.md) under *Optional quality-guarded routing*. ## 1. Register a validator (organization administrator) Only a registered key may submit trusted evidence, for its own organization and listed rubrics. ```bash curl -u admin:pw -H 'Content-Type: application/json' \ -X POST localhost:8080/admin/quality/orgs/$ORG/validators \ -d '{"key_id":"'$VALIDATOR_KEY_ID'","name":"fields-checker","rubrics":["fields"]}' ``` ## 2. Freeze an evaluation and adjudicate one sealed batch (operator) The spec fixes everything the verdict depends on - profile, rubric, baseline, candidates, required groups, `q_min`, `delta`, `severe_max`, confidence, method, design, the EXPECTED COHORT (the manifest), evidence lifetime, scope and pipeline - BEFORE you look at results. There are no defaults; the numbers below are an illustration, not a recommendation. ```json {"schema":"pointsman.quality-eval-spec/v3","profile_id":"support-extract", "rubric":{"id":"fields","version":"3"}, "baseline":{"model":"gpt-4o","model_version":"2024-08-06"}, "candidates":[{"model":"gpt-4o-mini","model_version":"2024-07-18"}], "groups":["en","de"],"q_min":0.95,"delta":0.02,"severe_max":0.01,"confidence":0.95, "method":"clopper-pearson-bonferroni-v1","design":"independent", "evidence_ttl_days":90,"scope":{"orgs":[]}, "pipeline":{"task_source":"group_default","task_types":[],"baseline_fallbacks":[],"cascade":"none"}, "manifest":{"tasks":[{"task_id":"inv-0001","group":"en","content_sha256":""}, {"task_id":"inv-0002","group":"de","content_sha256":"..."}, ...]}, "illustrative":false} ``` `POST /admin/quality/epochs` registers it (the same spec twice returns the same epoch). `POST /admin/quality/epochs/{id}/adjudicate` takes ONE batch (`{"schema":"pointsman.quality-eval-batch/v2","rows":[...]}`) of labelled rows - one row per task per model, `outcome` `success`, `failure` or `missing` - and records the verdict forever. ### Frozen cohort membership The cohort is frozen, not described. The batch must be EXACTLY the manifest: every expected (task, model) pair once - a failure, a timeout and an absent label are all rows (`missing` for the last two) - and nothing else. A row for a task the manifest does not list, a task in another group than the manifest says, or an expected pair left out refuses the batch: an export that drops its failures cannot qualify a model by omission. ### Complete model/task coverage Every compared model answers every task. Qualification compares the baseline and each candidate on the SAME complete task population in every group. A task whose `models` subset leaves out the baseline or a candidate is refused as an unsupported design (`spec_unsupported`). That covers a candidate run on the easy half, disjoint populations and partial overlap. Such a cohort is never intersected away. Independently randomised populations are not certified by this checker. An epoch registered before this rule with such a cohort keeps its verdict readable and cannot promote (`legacy_unpaired_cohort`). ### Corpus identity The corpus identity is the membership, not the order. `corpus_sha256` is derived by the server as the order-independent identity of the manifest's entries: task id, group, model set and `content_sha256` when declared. The same tasks in another order are the same corpus, and so the same repeated look. The model set is canonical as well: leaving `models` out, listing every compared model, or listing them in another order or with a repeat all mean "every compared model" and give one identity. The ordered list is reported separately as `execution_identity`. Every change to a task, group, model or content hash is a different corpus. Declare `content_sha256` per task when the task content lives outside the gateway: without it, the identity binds task ids only, and the verdict says so, because an id does not prove two evaluations saw the same prompt. v3 is the registrable schema. Stored v2 epochs keep their ordered `corpus_sha256` unchanged, adjudicate and promote under the same checks; new v2 registrations are refused (`spec_schema_superseded`). Repeated looks are counted by the canonical identity across every stored epoch, so restating a seen corpus in another order or under another schema version is a repeated look. v1 specs (a self-declared corpus hash) are no longer accepted; v1 epochs stay readable and never promote. The epoch view reports `corpus_identity`, `execution_identity` and `content_bound`. ### Evaluated model identity The evaluated identity is captured, not assumed. At registration the server records the registry digest of every model the spec evaluates (each must be in the registry). The batch is refused if any of those rows changed while the evaluation ran, and promotion compares the CURRENT row with the EVALUATED one - so an alias repointed after the evaluation cannot inherit its verdict, even on the first promotion. A price change does not count as a change: it moves cost ordering, not answer quality. ### Pipeline identity `pipeline` names everything the gateway controls about how a task reached the model: the task source and classifier version, the `task_types` mapping that routes labels to the profile, the baseline's declared fallbacks, and the cascade verifier between answer and client (`"none"` without one). Promotion requires the profile to serve exactly that pipeline (and that baseline), and serving re-checks it on every request - together with the EVALUATED baseline's registry identity, so a baseline alias repointed after promotion makes the candidate's evidence stale rather than relative to a baseline nobody compared it with (price-only changes excepted). Declared baseline fallbacks are part of the pipeline by name only: they are operational, never evaluated identities, so switching a group's cascade on after promotion makes its evidence stale. Your prompts and generation parameters are not gateway configuration and are NOT bound: evidence assumes your traffic looks like the evaluated corpus. How the verdict is reached: every bound the decision uses (per group, the candidate's success and severe-error rates and the baseline's success rate) is an exact Clopper-Pearson interval at a Bonferroni-allocated level, so they hold simultaneously at the spec's confidence. A candidate is QUALIFIED only if, in **every** required group, its lower success bound is at least `q_min`, it is within `delta` of the baseline's upper bound, and its severe-error upper bound is at most `severe_max`. It is REJECTED when the evidence shows it fails; INSUFFICIENT when the evidence cannot tell (an empty group is never a pass); INCOMPLETE when any selected row has no label. Duplicated tasks, declared clusters and weighted or sequential designs are refused as UNSUPPORTED rather than counted as independent. Price is never an input: a cheap model qualifies exactly when its labels support it. ### Optional latency gate Add `"latency":{"quantile":0.95,"ratio_max":1.5}` to the spec and every labelled row carries `latency_ms` (end-to-end; a batch missing one is refused). The candidate then also needs its p95 (here) proved within 1.5x the baseline's, on the same frozen tasks: the upper bound of its p95 must be at most `ratio_max` times the LOWER bound of the baseline's. It is REJECTED when its lower bound already exceeds `ratio_max` times the baseline's upper bound, and INSUFFICIENT in between. `"max_ms":5000` instead (or as well) is an absolute SLO for the surface: the candidate's upper p95 bound must be at most 5 s, and a lower bound above 5 s rejects it - no baseline interval is needed for that part. The bounds are exact, distribution-free order-statistic intervals counted in the same Bonferroni family (one per candidate plus one for the baseline), so the spec's confidence still covers every bound at once. Latency only ever removes a qualification; a fast model with failing labels stays REJECTED. It is a gate, not a learning signal: the bandit (with `selection: bandit`) then adapts only among models that already met it. Audit windows cannot declare it (routing chose who answered what, so the two latency populations are not comparable). Without `latency` the spec, its hash and the verdict are exactly what they were. One adjudication per epoch, ATOMICALLY: two submissions that race are finalized once (an optimistic lock on the epoch); the other - like any later submission - gets 409 `epoch_already_adjudicated`, which says `same_batch=true|false`, and leaves no audit record. A promotion running meanwhile sees the epoch as not adjudicated until the winner commits. Changing a threshold is a new spec; a new spec on a corpus that was already adjudicated is marked `respecified_after_look` and can never support promotion - the repeated-look pattern a fixed-sample interval does not cover. Looks are counted again at promotion: if another epoch on the same corpus was adjudicated before this one, the promotion is refused ("repeated looks, counted at promotion"), even when both were registered before either was adjudicated. Only the first adjudication of a corpus can promote. Specs marked `illustrative` can never promote either. ## 3. Preview, then promote a policy (operator) ```json {"schema":"pointsman.quality-policy/v1","expires_at":"2026-12-31T00:00:00Z", "profiles":[{"id":"support-extract","group":"support-extract","default_for_group":true, "rubric":{"id":"fields","version":"3"},"baseline":"gpt-4o","baseline_fallbacks":[], "orgs":[],"hints":{"trust":"none","task_types":[]}, "pipeline":{"task_source":"group_default"},"exploration":{"epsilon":0.0}, "qualified":[{"model":"gpt-4o-mini","model_version":"2024-07-18","epoch_id":""}]}]} ``` `POST /admin/quality/policy/preview` shows every error, warning and the change against the active version. The server - not the draft - binds each qualified entry: it checks the epoch (adjudicated, not illustrative, not re-specified, same profile, rubric and pipeline, the candidate QUALIFIED, evidence unexpired, scope compatible) and writes the registry digest, spec hash and evidence expiry itself. A draft that asserts its own binding is refused. `POST /admin/quality/policy/promote` with `{"document": ..., "expected_version": , "reason": "..."}` creates the next immutable version and reloads every instance through the control-plane channel. `POST /admin/quality/policy/rollback` with `target_version` re-validates that old document today: evidence that has expired since, or a registry row that has changed, refuses the rollback unless you pass `"drop_expired": true`, which removes those entries. Rollback never revives expired evidence. `GET /admin/quality/status` shows what every instance is actually enforcing. ## 4. Submit evidence `POST /v1/quality/assessments` with an API key: ```json {"receipt":"q1....","rubric_id":"fields","rubric_version":"3","outcome":"failure","severe":false, "source":"trusted_validator","external_id":"job-8812","revision":0,"lane":"audit", "inclusion_probability":0.05,"evidence_ref":"s3://audits/8812"} ``` - `source` is checked by the server: `trusted_validator`/`reviewed_gold` only from a registered validator key; any other key of the organization may send `voluntary`; another organization's receipt does not verify. - One identity (`source`, validator, `external_id`) and `revision` is stored once; the same payload again is a duplicate (`200`), a different payload is `409`; a correction is a higher revision. - No weights, counts or scores are accepted, and no payload: `evidence_ref` is an opaque reference. - Trusted, judge-proxy and voluntary evidence are kept in separate namespaces and never mixed; voluntary ratings are self-selected and never used for qualification. Stored evidence supports qualification only through a sealed batch; this call does not change routing. ## 5. Evidence without rating every response: the audit lane Set `POINTSMAN_QUALITY_AUDIT_ENABLED=true` and a fixed `POINTSMAN_QUALITY_AUDIT_SAMPLE_RATE` (say `0.05`). At serve time - before any outcome exists - that share of quality-guarded responses is SELECTED and the response says so: `X-Quality-Audit: selected; lane=audit; p=0.05`. Your application's validator labels those decisions with `lane: "audit"` and `inclusion_probability` equal to `p`. Every selection is recorded (`GET /admin/quality/audit`): selected, completed, ungraded, failed, expired, and selections DROPPED because the bounded queue was full (tallied per profile-hour). `p` is the probability of selection, never of a label having been obtained. To turn a window of audits into evidence, register the window BEFORE it opens: a v2 spec with `"audit_window":{"from":"...","to":"..."}` instead of a manifest, groups `["all"]`, and `scope.orgs` naming the organizations it samples (a window starting at or before registration is refused: it may already have been looked at). After it closes, `POST /admin/quality/epochs/{id}/seal-audit-window`. Every in-scope selection becomes a row; a selection with no trusted label - including one only a judge looked at - is `missing`, so the cohort is INCOMPLETE rather than quietly smaller. Selections from other organizations are outside the preregistered frame and are excluded (and counted). Each selection records what ACTUALLY served it - the registry digest of the serving row and the profile's pipeline - and sealing refuses a window in which an alias served a different identity than the one evaluated, or a different pipeline, rather than relabelling it. A window with any dropped selection is refused. ### Audit sealing deadline Seal within `pointsman.quality.audit.retention-days` of the window's START. The daily audit purge deletes finished jobs older than `retention-days` (default 30) and does not know which windows are still unsealed. A window sealed later than that adjudicates only the jobs that survived: a quietly smaller cohort, not an INCOMPLETE one, and the adjudication is final. Because an audit window never promotes, this cannot qualify a model, but the recorded counts and bounds would be wrong. Keep windows shorter than `retention-days` and seal promptly. A seal-time refusal is the intended fix; it is not in this release. An audit window is OBSERVATIONAL: routing chose which model answered which request, so candidate and baseline cells hold different task populations. Its verdicts keep every count and bound and can REJECT or be INCOMPLETE, but QUALIFIED becomes INSUFFICIENT and the epoch never promotes. Qualification comes from a frozen manifest where every model answers the same tasks. ### The optional judge (proxy only) `POINTSMAN_QUALITY_JUDGE_ENABLED=true` with an explicit provider and model, payload consent, and an enabled operator account. The judge reads the user turns and the answer (system and developer messages and tool definitions are never sent), fenced as untrusted data, with no tools, and must answer exactly `{"verdict": "pass"|"fail"|"abstain", "severe": bool}`; anything else is UNKNOWN. It never judges its own answer, it is called through the provider adapter directly (no router, cache, reward or tenant charge), and its verdicts land in the PROXY namespace, which never supports qualification in this release. Calibrate it against reviewed gold labels (false acceptance bounded) before trusting it for anything; even then it stays a proxy. ### The operator-funded auxiliary account Judge, audit, classifier, alternative-answer and cascade attempts reserve against ONE operator account held in Postgres (`POINTSMAN_QUALITY_AUX_*`): every limit is fleet-wide and survives restarts. ### Auxiliary attempt reservations Each attempt reserves its own estimated bound, under a deliberately narrow contract. An auxiliary attempt is admitted only when all three hold, and is refused BEFORE anything is sent otherwise: - the provider's adapter type is declared in `POINTSMAN_QUALITY_AUX_BOUNDED_ADAPTERS` (default `openai,openai_compatible`: OpenAI-wire adapters billed per prompt and completion token); - the request is plain text chat: roles system/user/assistant with text content, no tools, tool choice, tool calls or tool results, no images, audio, output modalities, predicted output, web search or response format, `service_tier` absent, `auto` or `default`, not streaming; - the row is priced per 1k input and output tokens, with no cache-write price above its input price. The bound is the UTF-8 bytes of every billed field of that form (role, content, name) plus a framing allowance, plus the output cap the gateway writes onto the outbound request (the tighter caller cap, else the row's maximum output) times `n`, at the rates captured for that attempt. It is an upper bound ONLY under the stated assumption of a byte-level tokenizer (never more than one token per byte) whose chat framing fits the allowance; it is not a universal hard limit, and a provider that breaks the assumption shows up as an overrun that fences the account. The attempt's pricing identity (model and rates) is recorded with its reservation and its observed usage is priced with those same rates, so a registry edit mid-attempt changes neither. `MAX_ATTEMPT_MICROS` caps one attempt's bound; `MAX_SPEND_MICROS` caps spent + reserved + unknown across the fleet, so two replicas racing cannot jointly admit more liability than the account holds. ### Dispatch controls Every send is one governed dispatch. Judge, alternative-answer and cascade calls go through the same single-attempt seam: provider capacity, one observation of the credential pool (a configured pool with no usable key refuses - it never falls through to the adapter's default key - and a key selected for the attempt that disappears before the adapter uses it refuses rather than being replaced by another), breaker admission taken only after the key is chosen (an OPEN breaker, or a HALF_OPEN breaker whose probe permit is taken, refuses before sending; a refusal before sending gives the permit back), exactly one breaker outcome per real send, the outbound tally kept and a rate-limited key cooled. One send, never a hidden retry or failover. ### Transactional admission The account mutation, the attempt row and its pricing identity are written together; a failure inside it rolls the whole admission back (refused, nothing sent). A failure while COMMITTING is reported as uncertain, never as a rollback: nothing is sent, and if the reservation did commit it expires into unknown exposure rather than vanishing. ### Estimation boundary A conditional conservative ESTIMATE for the declared adapters under their tokenizer and framing assumptions - not an absolute invoice-spend guarantee and not a universal provider bound. `openai_compatible` names a wire protocol, not a tokenizer or a pricing guarantee: declaring an adapter bounded is the operator's claim about that provider, and a live evaluation permit must name the concrete providers, models and assumptions it tests. A call whose cost was not observed after it may have executed (a timeout, an interrupted response, a crash before settlement) keeps its WHOLE bound as UNKNOWN exposure - never refunded, never zero - and new attempts stop once that exposure passes `MAX_UNKNOWN_MICROS`. A call proven never sent (connection refused) releases its reservation. An observed cost above its bound is recorded as spent, never clipped, and FENCES the account - whether the cost is observed at settlement or later at reconciliation of an UNKNOWN attempt: nothing more is admitted until `POST /admin/quality/auxiliary/unfence {"reason": "..."}` (the overrun stays recorded). `GET /admin/quality/auxiliary` shows calls, spent, reserved, unknown exposure, the per-attempt ceiling and the fence; `POST /admin/quality/auxiliary/attempts/{id}/reconcile` settles one with a reviewed amount. No tenant budget, usage row or invoice is touched. ## 6. Task profiles from Jev, observation before enforcement ### Jev task attribution A profile with `"pipeline": {"task_source": "jev", "classifier": "jev-1.13.0"}` and `hints.task_types: ["extract"]` is chosen when Jev labels the request `extract` - with Jev's own opt-ins (enabled, the tenant listed in `allowed-tenants`) and bounds, one call per request, and never for a request that declares tools. The label chooses a PROFILE, never eligibility: a confident misclassification lands in that profile's own qualified set and nowhere else. Abstention, Jev off, a tenant not opted in, or a classifier version other than the one the profile names all fall back to the group default, and the reason travels in the decision record (`profile_reason`: `jev:`, `jev_unavailable`, `jev_low_confidence`, `classifier_version_mismatch`, ...). The profile's evidence must have been gathered through the same pipeline: promotion refuses an epoch whose spec names a different task source or classifier. Jev confidence is a classification signal, not answer correctness. ### Observe decisions `POINTSMAN_QUALITY_OBSERVATION_DECISION_ONLY=true`, plus a policy profile for a group that still uses another strategy: every decision there also records what quality-guarded WOULD have served (`GET /admin/quality/observation`, a bounded buffer, and `pointsman.quality.observation` agree/disagree counts). Nothing served, billed or learned changes, and no remote call is made (groups with Jev-sourced profiles are skipped). ### Generate alternative answers A separate, paid permission (`..._ALTERNATIVE_ANSWERS=true`, payload consent, an enabled operator account, a rate of at most 0.1 and a per-minute cap): for sampled disagreements, the would-be qualified model is asked too, in the background, through the provider adapter. Only its cost and status are kept - the content is dropped, so it can never leak into evidence, learning, the cache or a held-out set. ### Evaluation support A quality decision logs propensity 1.0 when it is deterministic (exploration 0, price selection) and NULL otherwise, with `propensity_kind: unknown_marginal`, the branch, epsilon and the qualified set in its decision record. Under `selection: bandit` the bandit logs the decision itself, with its sampled snapshot, as it does for any bandit decision. The offline estimator (`policy_service/ope.py`) refuses those rows and also refuses the legacy bandit exploration-floor rows' logged `1/n`, which is conditional on the floor branch; `marginal_propensity` reconstructs the policy's marginal `eps/n + (1 - eps) * P_thompson` from the logged snapshot and labels it `reconstructed_mc`. ## 7. The bounded cascade (experimental, off) For a listed quality-guarded group, the cascade makes release depend on a FORMAT check of the answer itself. The routed answer is HELD and never written until a deterministic format verifier accepts it. On reject or abstain, ONE escalation to the profile's first baseline model runs, and its answer is released only if the same verifier accepts it. Anything else is refused. Nothing provisional ever reaches the client. ### Verifier scope A format check, not a correctness check. The verifiers decide whether an answer has the shape the task needs. A well-formed WRONG answer - the wrong total in valid JSON, an allowed but wrong label - passes. That is why an acceptance is disclosed as `check=format`, writes no quality event and moves no success count: it is never quality evidence. Semantic verification (checking content against trusted, request-specific context under a versioned rubric) is NOT IMPLEMENTED; a `semantic:`, `judge:` or `llm...` verifier refuses startup by name. Because a cascade group's evaluated pipeline includes its verifier, the spec that qualifies its cheaper model must name the same verifier in `pipeline.cascade` (section 2). ```bash POINTSMAN_QUALITY_CASCADE_ENABLED=true POINTSMAN_QUALITY_CASCADE_GROUPS=support-extract # quality-guarded groups only POINTSMAN_QUALITY_CASCADE_VERIFIER=json-fields:total,currency POINTSMAN_QUALITY_CASCADE_MAX_STAGE_MICROS=20000 # $0.02 upper bound on the escalation stage POINTSMAN_QUALITY_CASCADE_ESCALATION_TIMEOUT_MS=30000 # plus an enabled operator account (section 5): the escalation is paid from it ``` The verifiers are local and deterministic, with no model and no network: - `json-object`: the whole answer is one strict JSON object. A code fence, trailing text or a duplicate key is a rejection. - `json-fields:`: the same, and every named top-level field is present and not null. - `exact-choice:`: the trimmed answer is exactly one of the labels. Case matters. A verifier abstains when there is no complete text to check: a tool call, no choices, more than 65,536 characters, or a provider that cut the answer off (`finish_reason: length`) or filtered it (`content_filter`) - a truncated JSON object that happens to parse is not accepted. A model judge is deliberately not a verifier here. It is uncalibrated, and a cascade must not judge recursively. ### Escalation eligibility The escalation is a destination like any other. Before it is called, the baseline must pass the same per-hop test the router applies to every fallback hop, for this caller and this request: the effective allowed set, lifecycle, capabilities and context fit against the request's pinned admission context, then live health (breaker and a usable credential). Its registry row must be the one admission pinned. Failing any of these is a refusal before any call (`escalation_not_eligible` or `escalation_unavailable`); operator funding never authorizes a destination the caller could not use. The outbound escalation carries an explicit `max_tokens` - the one its cost bound was computed from (section 5). | Outcome (`X-Quality-Cascade` / refusal reason) | What happened | Billed | |---|---|---| | `primary_accepted; stages=1; check=format` | the routed answer passed the format check | tenant: the one call | | `escalated_accepted; stages=2; check=format` | the routed answer failed; the baseline's passed and was served (`X-Quality-Route: baseline`) | tenant: the primary call; operator: the escalation | | 502 `quality_cascade_unverified`, `final_rejected` / `final_abstained` | neither answer passed | tenant: the primary; operator: the escalation | | 502, `no_escalation` | the routed answer failed and was already the baseline (or nothing is qualified) | tenant: the primary | | 502, `escalation_unsupported` | the escalation is outside the supported cost contract (adapter, request form or pricing; section 5) | tenant: the primary | | 502, `escalation_not_eligible` | the baseline is outside what this caller may use for this request (allowed set, lifecycle, capability, context) | tenant: the primary | | 502, `escalation_unfunded` / `escalation_over_ceiling` / `escalation_unbounded` / `escalation_unavailable` | the escalation could not be paid for, bounded in advance (unpriced, no `max_tokens` and no row max output, image or audio) or reached (provider missing, unhealthy, row changed since admission) | tenant: the primary | | 502, `escalation_failed` | the escalation timed out or failed; a timeout stays on the account as UNKNOWN exposure | tenant: the primary; operator: possibly the escalation | | 502, `verifier_failed` | the verifier itself threw; nothing escalates | tenant: the primary | | 400 `quality_cascade_unsupported` | a stream or a request declaring tools: neither can wait for verification | nothing; refused before any provider call | The tenant pays for the primary stage even when nothing is released. The model was asked and answered, the same as any completed call. The escalation is extra work the gateway chose to do, so it is never charged to a tenant. An escalated release teaches no learner: the primary arm's answer was rejected, so reporting it as served would be a hidden success assumption. The escalated answer passes the same output guardrails as the primary. Turn it on only after you have measured it against single-model routing on your own workload: the cost of the escalations, how often each verifier rejects and abstains, and the refusal rate your clients will see. None of that has been measured here. ## What it does not promise - No evidence means baseline routing and "quality unknown", not a cheaper model. - Qualification covers the recorded task distribution, rubric, model versions and pipeline. It is not a per-answer guarantee and does not transfer to a new model version, rubric or classifier. - A judge's score is a proxy until a separate calibration says otherwise; prompt-injection mitigations reduce, not remove, the risk of a manipulated score. - A queued audit selection that was not yet persisted when the process died is lost; a graceful shutdown persists the queue instead. Overload drops are counted, never silent. - A DEGRADE limit rule never carries a request out of a quality-guarded group: the substitution is refused (503 `quality_policy_unavailable`, reason `degrade_leaves_quality_contract`) before anything is resolved, reserved or sent. Groups on other strategies degrade exactly as before. There is no waiver to serve a degraded quality request outside its contract. - The cascade's verifiers check FORMAT (JSON, fields, a label), not truth. A well-formed wrong answer passes. Semantic cascade verification is not implemented. The cascade is not a quality measurement and has no live validation: it is experimental and off by default. - Evidence binds the model's registry identity and the gateway-side pipeline, not your prompt templates or generation parameters: if your traffic changes, the evaluated task distribution no longer describes it.