Routing answers two separate questions: which deployments are allowed and usable, then which of those should serve this request. A learning algorithm operates inside that boundary; it does not grant permission, increase a budget or bypass a quality policy.
Resolution before selection#
Resolve a concrete model, group, semantic group or configured fallback using the request's pinned registry snapshot and resolution policy. Narrow candidates by effective model access, capabilities, context fit, request controls, availability and applicable constraints. Provider credential pools select among configured members without turning each API key into a distinct model.
Selection then applies the group's strategy. Weighted-random is the default. Static priority, cost, latency or other configured strategies are distinct from adaptive bandit behavior. A lower static list price does not itself constitute learned cost efficiency.
Three learning designs#
| Mode | Objective and state | What it does not prove |
|---|---|---|
| Balanced bandit / optional LinTS | Gateway samples local parameters; the Python policy service learns from asynchronous observations and publishes snapshots | A completed answer is not a quality label; blended reward does not guarantee minimum cost |
Bandit group with objective: cost |
In-process cost sampler estimates cost per completed call from priced usage and operational success; scoped per deployment | No Python learner or quality labels on this path; state resets on process restart |
| Quality-guarded profile | A published policy first defines qualified candidates or its declared baseline; selection runs inside that set | Quality evidence has scope, age, rubric and version limits; it is not universal correctness |
These modes are experimental and opt-in. A quality profile may select by price, balanced bandit or cost adaptation as configured. Its boundary constrains every fallback. If no eligible qualified candidate or permitted baseline remains, it refuses instead of quietly choosing an arbitrary model.
How balanced learning works#
The gateway samples a local posterior rather than synchronously calling /policy/decide. Decision records bind the candidate set, selected arm, parameters version, provenance and available selection-probability information. A posterior draw must not be reported as a known marginal propensity when it is not one.
Observed completion/failure and eligible late feedback become self-describing reward events. Cost, latency and length are operational signals. Application ratings or configured judges supply a separate quality signal. Without one, optimizing the reward is not evidence that answers improved. Cache hits do not become fresh provider observations simply because they were served successfully.
Uniform priors represent cold start or a missing learned key; a last usable snapshot can continue serving during a learner outage. Provenance and metrics distinguish learned, prior and fallback behavior. An unexpected strategy failure can use the defined fallback strategy; it must not manufacture a learned decision identifier.
Cost and direct observations#
The cost sampler combines observed usage relative to registry prices with operational completion probability. It can adapt to verbosity, provider prompt-cache pricing and reliability; those quantities remain estimates from reported usage, not invoice reconciliation. Its statistical exploration differs from the balanced path's exploration policy.
Direct concrete-model traffic does not train groups by default. The explicit learn-from-direct: observe mode contributes operational observations to eligible groups. It does not attach quality ratings to direct requests or bypass selection-bias concerns: callers may send harder prompts to one model.
Attempts, fallback and streaming#
An attempt pins the deployment, credential, registry/generation and request contract used to serve it. Failures are classified before deciding whether to retry the same deployment or use the bounded fallback ladder. Availability, permissions, context, quality constraints and model-dependent budget checks still apply on later hops.
A provider 429 can cool the relevant credential member; it is not the same event as a provider transport failure. Once live stream bytes are delivered, transparent replay or switching models would corrupt the response contract. Treat an interrupted stream as a terminal outcome, preserve incurred usage and follow the documented SSE behavior.
What to inspect#
Check requested versus served model, decision provenance, prior/snapshot age, fallback reason, reward delivery failures and actual learner receipts. For quality policies, also check evidence validity, policy expiry and baseline use. Measure an objective against a suitable baseline; adaptive does not mean better by definition. The retained live comparison did not establish an adaptive advantage, and the release makes no quality-preserving savings claim.
Implementation anchors: routing/ModelResolver, ModelRouter, RouteFilterChain, AttemptBinding, FallbackLadder, routing/bandit/BanditStrategy, DirectRequestLearning, quality/routing/QualityCostSelector and the quality-policy selector. Continue with policy service, routing strategies, outcome-learning setup and quality-guarded setup.