# Routing strategies

Choose a routing strategy for each model group. Strategies select among eligible candidates; authentication, model permissions, budgets, health and fallback checks still apply. **Weighted-random is the default.** Adaptive and quality-based paths remain experimental and opt-in; see [release scope](release-status.md).

## Where a strategy applies

A **group** is a model name your callers ask for that stands in front of several real
candidates; the group's `strategy` picks one per request. A request naming a real model is
not a group and none of this applies - groups are purely additive. Defining one:
[send some of your traffic to a different model](how-to/split-traffic-between-models.md).

Two things happen before any strategy runs, whichever you chose:

- **Health filtering.** A candidate whose provider breaker is open, whose deployment is on a
  429 cooldown, or which fails a capability or context-window check is dropped from the set
  the strategy sees. No strategy can pick a candidate the filter removed.
- **Vocabulary checking.** An unknown strategy name refuses at boot and names the legal set -
  A custom strategy you register as a Spring bean is legal even
  though it is not in the shipped list; the vocabulary closes over registered beans, not over
  this page.

## The menu

Use the selection rule and tradeoff together when choosing a strategy.

| Strategy | Selects by | Useful for | Main limitation |
|---|---|---|---|
| `priority` | First eligible candidate | Predictable primary/fallback order | Healthy primary takes all traffic; no spreading |
| `random` / `weighted-random` | Random choice, optionally weighted | Canaries and A/B splits | Weights do not adapt to price or model changes |
| `round-robin` | Even rotation with a counter per group | Equal request distribution | Unequal candidates receive equal traffic |
| `latency` | Peak EWMA × (1 + in-flight requests) | Avoiding slow or overloaded candidates | Speed is not answer quality |
| `cost` | Lowest declared blended input/output price per 1k tokens | Static cost preference | Sees neither latency nor quality |
| `least-busy` | Fewest live in-flight requests; priority breaks ties | Shifting work from occupied deployments | Busyness does not measure suitability |
| `usage` | Most outbound TPM headroom | Spreading traffic across provider quotas | Capacity is not quality or price |
| `bandit` | Observed outcomes under a cost or balanced objective | Experimental adaptive selection | Depends on evidence and cold-start behavior |
| `lints` | Linear Thompson Sampling conditioned on request features | Experimental contextual selection | Requires the learner and meaningful feedback |
| `jev` | External task classification plus operator rankings | Experimental task-specific routing | Exports content and adds operator-paid classification |
| `quality-guarded` | Task-qualified models with valid evidence | Experimental evidence-constrained selection | Qualification covers the recorded rubric and distribution |

### Static selection details

The `latency` strategy probes unmeasured candidates first and randomizes within a buffer of the best score. This avoids permanently starving unmeasured candidates or concentrating every request on a single fast provider.

The static `cost` strategy treats declared zero as free, but excludes undeclared prices and increments `pointsman.routing.unpriced_excluded`. If every candidate is unpriced, it reports the condition and uses priority order.

### Adaptive objectives (`bandit`)

The `bandit` strategy uses Thompson sampling over observed outcomes. Enable `pointsman.bandit.enabled`; without the flag, the strategy name remains legal but the group falls back to the default strategy.

- **Balanced objective:** uses the learner's blended reward. Cold start begins with equal priors; without a quality signal it optimizes efficiency rather than correctness.
- **Cost objective:** learns observed cost per completed call in the gateway. It does not require or send data to the learner.

Read [the learning model](design/adaptive-routing.md) for the objectives and [the setup guide](how-to/turn-on-outcome-learning.md) to enable feedback.

#### Choose an objective

 A bandit group optimises one of two objectives, chosen per group:

  ```yaml
  pointsman:
    routing:
      groups:
        support:
          strategy: bandit
          objective: cost        # or balanced (the default; omit the line for it)
          candidates: [ ... ]
  ```

  - `balanced` (default): the behaviour described above. It is Thompson sampling on the learner's blended reward
    (success, latency, answer length, cost relative to the other candidates, and ratings when you send them).
  - `cost`: Thompson sampling on each member's **observed cost per completed call**. That is the registry's
    cache-aware price applied to the usage the provider reported, started from the declared prices. On day one it
    ranks as the prices do. When a model's real cost per request drifts from its price (it writes or reasons more, a
    prompt-cache discount), it can adapt as observations accumulate. It runs in-process
    and sends nothing to the learner. The group still needs `pointsman.bandit.enabled`, because that flag creates the
    bandit strategy.

  Different from the static **`cost`** strategy: that one always takes the lowest *declared* price and never
  looks at a bill.

#### State and persistence

 `balanced` learns in the policy service (the learner), which publishes
  posteriors the gateway polls; it survives a gateway restart. `cost` learns in the gateway's memory, per tenant and
  group: a restart starts again from declared prices, and the state is bounded (at most 10,000 tenant-group states, the
  least recently used dropped beyond that, which only returns that group to declared prices; at most 64 members per
  state; `pointsman.cost-sampler.*`). The cost-state read and the Observe > Belief page show it.

#### Deployment identity

 Both objectives learn per **provider and model** (`provider:model`): the same model behind
  two providers is two members with their own cost and reliability, and a failure at one never lowers the other.

#### Free and unpriced models

For `objective: cost`, A model whose declared rates are both zero is free: it scores zero
  and wins on cost; its successes and failures are still learned, and among free members the more reliable wins. A
  model with no declared price is never treated as free: it is chosen last. Neither has a cost multiplier to learn. A
  free model that fails is still subject to the health filter, the breaker, cooldown and the fallback ladder.

#### Members without observations

 A member the sampler has not observed yet - a second deployment, or a member
  added later - is judged at its declared price times what the group's observed members actually cost against theirs,
  so it is tried and not starved by siblings whose bills came in under their estimates. With nothing observed in the
  group, the declared prices decide, as on day one.

#### Streaming usage

 A row reports usage on streams only when it declares the `streamUsage: stream_options` quirk. Without it a
  completed stream still counts as completed, but it teaches no cost; use the quirk where the provider supports it.

#### Quality boundary

`objective: cost` does not judge answers. It optimises cost per completed call, so it will happily buy
  cheap wrong answers if the group contains models that are not good enough for the work. Put only models you trust
  for that work in such a group, or use quality-guarded routing with `selection: cost`, which restricts the same
  sampler to models qualified on your own labelled data.

  An unknown objective, or `objective` on a group whose strategy is not `bandit`, refuses startup.

#### Supported parameters

 The objective is the one choice a group makes. Every other bandit and policy-service parameter
  ships at its default and is not a supported tuning surface in v1. A dashboard playground for tuning, with a what-if
  replay before a change takes effect, is planned after v1.0 ([roadmap](release-status.md)).

### Contextual selection (`lints`)

 Linear Thompson Sampling, the contextual variant: the bandit above, but the
posterior conditions on request features. Same flag, same learner.

### Task classification (`jev`)

Jev classifies the task, then applies the operator's ranking within the already eligible shortlist. See [setup and evaluation](how-to/jev-routing.md).

| Aspect | Behavior |
|---|---|
| Classifier | TypeSafe Jev, pinned `jev-1.13.0`; labels code, summarize, extract, reasoning, knowledge or chat |
| Default | Disabled; each tenant must opt in |
| Data and cost | Accepted conversation text, including system/developer messages, goes to the classifier. Calls are operator-paid overhead, not tenant charges. |
| Valid `X-Task-Type` hint | Uses the hint instead of calling the classifier |
| Tools, excessive input, no user turn or uncertain classification | Selects the first eligible candidate, as `priority` would |
| Provenance | Records the decision reason as `jev_task-v1_<reason>` |

A task label is a coarse signal, not evidence that the ranked model is better. This strategy does not learn rankings. Evaluate them on your own labeled traffic before relying on them.

### Evidence-constrained selection (`quality-guarded`)

Quality-guarded routing is experimental and disabled by default (`pointsman.quality.enabled`). It selects among models qualified for the request's task profile. Qualification requires a registered, adjudicated evaluation epoch with valid intervals, absolute and baseline-relative requirements, a severe-error ceiling, unexpired evidence and an unchanged registry row.

| Situation or control | Behavior |
|---|---|
| Qualified models available | Selects the cheapest declared price by default |
| No qualified model | Uses the declared baseline |
| No policy | Refuses `503 quality_policy_unavailable` |
| Group selects this strategy while feature is disabled | Refuses startup |
| Session affinity or family preference | Can retain only a permitted model |
| Fallback | Checks every hop; never escalates to a neighboring group |
| Response cache | Bypassed |
| Response evidence | `X-Quality-Route` identifies the route; signed `X-Quality-Receipt` must be cited by evaluation evidence |

Qualification covers the recorded task distribution and rubric, not every answer. Without evidence, this strategy offers no cost advantage over the baseline.

The optional [bounded cascade](how-to/quality-guarded-routing.md#7-the-bounded-cascade-experimental-off) can hold a listed group's answer until a deterministic format verifier accepts it. That verifier checks shape, not correctness. See the [quality-guarded setup guide](how-to/quality-guarded-routing.md) for policy and evaluation steps.

## What composes on top, regardless of strategy

**Session affinity** (`session-affinity: true` on a group) keeps a multi-turn conversation on
the provider it is already talking to, so the *provider's* prompt cache stays warm - every
major vendor bills a cached prefix at a fraction of input, and the discount only exists when
consecutive turns land on the same provider. It is a wrapper, not a strategy: it decides
whether your strategy runs at all, so it composes with every entry above. A pin yields only
when its provider leaves the filtered candidate set or after five minutes of inactivity (the
provider cache's own expiry) - and, stated plainly as the trade it is: a pinned conversation
does not move to an arm the bandit now prefers until the pin lapses.

**Deprecated deployments** are deprioritised at bandit selection (sampled score times a
configured penalty, default 0.5), never excluded - un-deprecating reverses it instantly
because it is a selection weight, not a reward term the learner would have to unlearn.

**The fallback ladder** catches the failure after any strategy's pick. The same-deployment
retry comes first, but it is not a rung of the ladder: it is the Resilience4j retry inside one
attempt (`pointsman.fallback.same-retries`). The ladder proper then walks six rungs, each
re-derived per hop against live breaker and cooldown state - same model elsewhere, your
explicit fallback chain, same family, same tier, one tier up, one tier down. A context-window
overflow leaves that order and follows the window-ascending chain instead: the next candidate
with a strictly larger declared window. Every rung passes the same eligibility gate, so a
fallback pointer changes candidate order and never permissions. Substitutions are disclosed on
the response (`X-Attempted-Models`, `X-Model-Resolution`).

## When the caller sends no model

A request that names no model still has to go somewhere, and the blank-model path is its own
small ladder, tried in order: the default model (the key's own, or when the key has none, its
tenant's), then **semantic routing**,
then the mode tiers, then the cheapest allowed model. Each rung only sees candidates the key is
allowed to use - nothing on this path can escape the allow-set. The `unknown-model` policy does
not govern this: even the fail-closed `resolve-only` refuses an *unknown* name, not an absent one.
When the rung lands on a group, that group's strategy - a bandit with its objective, for example -
chooses within it; when it lands on a single model, no strategy is involved.

**Fail safe or fail fast: `pointsman.resolution.omitted-model`** (`POINTSMAN_RESOLUTION_OMITTED_MODEL`).

| | `route` - fail safe (default) | `refuse` - fail fast |
|---|---|---|
| A request with no model | is answered, by the path above | gets `400 model_required`; nothing served, nothing billed |
| Which model | one **the gateway chose**, named in `X-Model-Resolution` and `X-Resolved-Model` | always exactly the one the client names |
| Cost | billed as the model that served, which may be cheaper or dearer than the client would have chosen | none for the refused request |
| Quality | whatever the path picks; "cheapest allowed" may be weaker than the work needs | not applicable |
| A client bug that drops the model field | silently served | surfaces immediately as an error |
| Suits | exploration, internal tools, "just answer" clients | production clients with a strict contract |

For fail safe to be *smart*, turn on `smart-routing-enabled` and name the tiers (see below). Without them the path is
the default model, then the semantic group match, then **the cheapest allowed model**. That is safe in that it never
leaves the key's allowed set, but it is not quality-aware. An unknown value refuses the boot.

**Per key and per tenant.** The setting above is the gateway-wide default. A key may override it with `omitted_model`
(`route` or `refuse`) on `PUT /admin/keys/{keyId}/policy`; null clears the override back to the global setting. The
default model is chosen the same way, nearest first: the key's `default_model`, else the tenant's (`PUT
/admin/organizations/{orgId}/default-model`), else the rest of the ladder. A tenant default is a name like any other:
it resolves through the key's allowed set, so a tenant default the key may not use is not served.

In the dashboard, a key's detail panel has a **Model resolution** section: each value, where it comes from (the key,
the tenant or the gateway), and an editor for an admin or a tenant admin of the key's organisation. The tenant default
is under **Control > Model defaults** for a tenant admin, and behind **Model defaults** on the organisation roster for
an admin, with how many keys use it.

## When the caller names a model

A request that names one concrete model is served by that model (with the fallback ladder behind it). No strategy
chooses, and by default no bandit hears of it.

**Learning from it: `pointsman.bandit.learn-from-direct`** (`POINTSMAN_BANDIT_LEARN_FROM_DIRECT`), `off` by default.
Under `observe`, every `strategy: bandit` group with that exact `provider:model` member is told what the call cost,
how long it took and whether it returned:

| Group objective | What a direct call teaches it |
|---|---|
| `cost` | the call's priced usage, or its failure, enters that member's cost estimate. The other members are not discounted, because no choice among them was made |
| `balanced` | one completion or failure reward goes to the policy service under the request's tenant and task type, for that member, marked `provenance: direct` |

The tradeoffs:

| | `off` (default) | `observe` |
|---|---|---|
| Bandit groups warm up | only from their own traffic | also from direct traffic to their members |
| What is learned | | cost, latency, length, success. **Never quality**: a direct response carries no `X-Decision-Id`, so no rating can attach to it, and quality-guarded profiles never see it |
| Bias | none added | callers choose models for their hard prompts, so a member's direct calls describe those prompts as much as the model. A model used directly for long work looks dear and slow to a group that sends it short work |
| Policy-service traffic | unchanged | one reward per direct call to a balanced group's member |

A model behind another provider is not the member, and teaches the group nothing. Group requests are unaffected
either way.

### Semantic routing

Think of a dispatcher who has read every team's one-paragraph job description and routes each
incoming request to the team whose description it most resembles. You write the descriptions;
the matching is automatic.

Give a group a `description:` - a sentence saying what traffic belongs there, in the words
that traffic actually uses - and the gateway embeds that sentence once at startup, exactly the
way the semantic cache embeds prompts. When a blank-model request arrives, the **latest user
turn** is embedded and compared against every group description by cosine similarity. The best
match wins if it clears `pointsman.routing.semantic-threshold` (0.75 by default); the response
then discloses the choice:

```
X-Resolved-Model: sql-helpers (semantic)
X-Semantic-Group: sql-helpers
```

Below the threshold, or when no group has a description, or while the embedder is still
warming after boot, the request simply falls through to the mode tiers - semantic routing
degrades to absence, never to an error. A matched group is also intersected with what the key
may use: a match whose members are all disallowed or incapable falls through the same way,
because a description is a suggestion and the allow-set is a rule.

| Aspect | Semantic group routing |
|---|---|
| Selection input | Latest user turn compared with operator-written group descriptions |
| Configuration | A `description:` on each participating group; changes require restart |
| Embedding dependency | Requires a useful semantic embedder; lexical hashing typically does not clear the similarity threshold |
| Feedback | Not used for group matching; the selected group's own strategy can still learn |
| Selection rule | Deterministic similarity matching for the same vectors and configuration |
| Cost | One embedding on this path. It is separate from the cache's redacted, context-scoped text, so a subsequent semantic-cache lookup can require another embedding |

Choose descriptions and callers with these limits in mind:

- A poor description can select an unsuitable group; inspect `X-Semantic-Group` on responses.
- Matching uses the current turn alone. A follow-up such as “make it shorter” can miss the original group or select another one.
- Use a named group when conversational traffic must retain that group. Evaluate semantic selection on representative single-turn traffic.

### The mode tiers

After semantic routing, **smart routing** (off by default) profiles the request, infers a
mode (cost / balanced / quality - overridable per request with `X-Routing-Mode`), and enters
the tier you configured for that mode, cascading to cheaper tiers when one is absent. Unknown
model *names* enter the cost tier the same way, disclosed via `X-Model-Resolution: fallback`.

## The extension point

A strategy is a Spring bean implementing `RoutingStrategy`
(`gateway/src/main/java/dev/pointsman/gateway/routing/strategy/`). Register a `@Component`
and its bean name is a legal `strategy:` value - the boot-time vocabulary check accepts any
registered bean.