Routing strategies

Choose a routing strategy for each model group. Strategies select among eligible candidates; authentication, model permissions, budgets, health and fallback checks still apply. Weighted-random is the default. Adaptive and quality-based paths remain experimental and opt-in; see release scope.

Where a strategy applies#

A group is a model name your callers ask for that stands in front of several real candidates; the group's strategy picks one per request. A request naming a real model is not a group and none of this applies - groups are purely additive. Defining one: send some of your traffic to a different model.

Two things happen before any strategy runs, whichever you chose:

• Health filtering. A candidate whose provider breaker is open, whose deployment is on a 429 cooldown, or which fails a capability or context-window check is dropped from the set the strategy sees. No strategy can pick a candidate the filter removed.
• Vocabulary checking. An unknown strategy name refuses at boot and names the legal set - A custom strategy you register as a Spring bean is legal even though it is not in the shipped list; the vocabulary closes over registered beans, not over this page.

The menu#

Use the selection rule and tradeoff together when choosing a strategy.

Static selection details#

The latency strategy probes unmeasured candidates first and randomizes within a buffer of the best score. This avoids permanently starving unmeasured candidates or concentrating every request on a single fast provider.

The static cost strategy treats declared zero as free, but excludes undeclared prices and increments pointsman.routing.unpriced_excluded. If every candidate is unpriced, it reports the condition and uses priority order.

Adaptive objectives (bandit)#

The bandit strategy uses Thompson sampling over observed outcomes. Enable pointsman.bandit.enabled; without the flag, the strategy name remains legal but the group falls back to the default strategy.

• Balanced objective: uses the learner's blended reward. Cold start begins with equal priors; without a quality signal it optimizes efficiency rather than correctness.
• Cost objective: learns observed cost per completed call in the gateway. It does not require or send data to the learner.

Read the learning model for the objectives and the setup guide to enable feedback.

Choose an objective#

A bandit group optimises one of two objectives, chosen per group:

yaml pointsman: routing: groups: support: strategy: bandit objective: cost # or balanced (the default; omit the line for it) candidates: [ ... ]

• balanced (default): the behaviour described above. It is Thompson sampling on the learner's blended reward (success, latency, answer length, cost relative to the other candidates, and ratings when you send them).
• cost: Thompson sampling on each member's observed cost per completed call. That is the registry's cache-aware price applied to the usage the provider reported, started from the declared prices. On day one it ranks as the prices do. When a model's real cost per request drifts from its price (it writes or reasons more, a prompt-cache discount), it can adapt as observations accumulate. It runs in-process and sends nothing to the learner. The group still needs pointsman.bandit.enabled, because that flag creates the bandit strategy.

Different from the static cost strategy: that one always takes the lowest declared price and never looks at a bill.

State and persistence#

balanced learns in the policy service (the learner), which publishes posteriors the gateway polls; it survives a gateway restart. cost learns in the gateway's memory, per tenant and group: a restart starts again from declared prices, and the state is bounded (at most 10,000 tenant-group states, the least recently used dropped beyond that, which only returns that group to declared prices; at most 64 members per state; pointsman.cost-sampler.*). The cost-state read and the Observe > Belief page show it.

Deployment identity#

Both objectives learn per provider and model (provider:model): the same model behind two providers is two members with their own cost and reliability, and a failure at one never lowers the other.

Free and unpriced models#

For objective: cost, A model whose declared rates are both zero is free: it scores zero and wins on cost; its successes and failures are still learned, and among free members the more reliable wins. A model with no declared price is never treated as free: it is chosen last. Neither has a cost multiplier to learn. A free model that fails is still subject to the health filter, the breaker, cooldown and the fallback ladder.

Members without observations#

A member the sampler has not observed yet - a second deployment, or a member added later - is judged at its declared price times what the group's observed members actually cost against theirs, so it is tried and not starved by siblings whose bills came in under their estimates. With nothing observed in the group, the declared prices decide, as on day one.

Streaming usage#

A row reports usage on streams only when it declares the streamUsage: stream_options quirk. Without it a completed stream still counts as completed, but it teaches no cost; use the quirk where the provider supports it.

Quality boundary#

objective: cost does not judge answers. It optimises cost per completed call, so it will happily buy cheap wrong answers if the group contains models that are not good enough for the work. Put only models you trust for that work in such a group, or use quality-guarded routing with selection: cost, which restricts the same sampler to models qualified on your own labelled data.

An unknown objective, or objective on a group whose strategy is not bandit, refuses startup.

Supported parameters#

The objective is the one choice a group makes. Every other bandit and policy-service parameter ships at its default and is not a supported tuning surface in v1. A dashboard playground for tuning, with a what-if replay before a change takes effect, is planned after v1.0 (roadmap).

Contextual selection (lints)#

Linear Thompson Sampling, the contextual variant: the bandit above, but the posterior conditions on request features. Same flag, same learner.

Task classification (jev)#

Jev classifies the task, then applies the operator's ranking within the already eligible shortlist. See setup and evaluation.

A task label is a coarse signal, not evidence that the ranked model is better. This strategy does not learn rankings. Evaluate them on your own labeled traffic before relying on them.

Evidence-constrained selection (quality-guarded)#

Quality-guarded routing is experimental and disabled by default (pointsman.quality.enabled). It selects among models qualified for the request's task profile. Qualification requires a registered, adjudicated evaluation epoch with valid intervals, absolute and baseline-relative requirements, a severe-error ceiling, unexpired evidence and an unchanged registry row.

Qualification covers the recorded task distribution and rubric, not every answer. Without evidence, this strategy offers no cost advantage over the baseline.

The optional bounded cascade can hold a listed group's answer until a deterministic format verifier accepts it. That verifier checks shape, not correctness. See the quality-guarded setup guide for policy and evaluation steps.

What composes on top, regardless of strategy#

Session affinity (session-affinity: true on a group) keeps a multi-turn conversation on the provider it is already talking to, so the provider's prompt cache stays warm - every major vendor bills a cached prefix at a fraction of input, and the discount only exists when consecutive turns land on the same provider. It is a wrapper, not a strategy: it decides whether your strategy runs at all, so it composes with every entry above. A pin yields only when its provider leaves the filtered candidate set or after five minutes of inactivity (the provider cache's own expiry) - and, stated plainly as the trade it is: a pinned conversation does not move to an arm the bandit now prefers until the pin lapses.

Deprecated deployments are deprioritised at bandit selection (sampled score times a configured penalty, default 0.5), never excluded - un-deprecating reverses it instantly because it is a selection weight, not a reward term the learner would have to unlearn.

The fallback ladder catches the failure after any strategy's pick. The same-deployment retry comes first, but it is not a rung of the ladder: it is the Resilience4j retry inside one attempt (pointsman.fallback.same-retries). The ladder proper then walks six rungs, each re-derived per hop against live breaker and cooldown state - same model elsewhere, your explicit fallback chain, same family, same tier, one tier up, one tier down. A context-window overflow leaves that order and follows the window-ascending chain instead: the next candidate with a strictly larger declared window. Every rung passes the same eligibility gate, so a fallback pointer changes candidate order and never permissions. Substitutions are disclosed on the response (X-Attempted-Models, X-Model-Resolution).

When the caller sends no model#

A request that names no model still has to go somewhere, and the blank-model path is its own small ladder, tried in order: the default model (the key's own, or when the key has none, its tenant's), then semantic routing, then the mode tiers, then the cheapest allowed model. Each rung only sees candidates the key is allowed to use - nothing on this path can escape the allow-set. The unknown-model policy does not govern this: even the fail-closed resolve-only refuses an unknown name, not an absent one. When the rung lands on a group, that group's strategy - a bandit with its objective, for example - chooses within it; when it lands on a single model, no strategy is involved.

Fail safe or fail fast: pointsman.resolution.omitted-model (POINTSMAN_RESOLUTION_OMITTED_MODEL).

For fail safe to be smart, turn on smart-routing-enabled and name the tiers (see below). Without them the path is the default model, then the semantic group match, then the cheapest allowed model. That is safe in that it never leaves the key's allowed set, but it is not quality-aware. An unknown value refuses the boot.

Per key and per tenant. The setting above is the gateway-wide default. A key may override it with omitted_model (route or refuse) on PUT /admin/keys/{keyId}/policy; null clears the override back to the global setting. The default model is chosen the same way, nearest first: the key's default_model, else the tenant's (PUT /admin/organizations/{orgId}/default-model), else the rest of the ladder. A tenant default is a name like any other: it resolves through the key's allowed set, so a tenant default the key may not use is not served.

In the dashboard, a key's detail panel has a Model resolution section: each value, where it comes from (the key, the tenant or the gateway), and an editor for an admin or a tenant admin of the key's organisation. The tenant default is under Control > Model defaults for a tenant admin, and behind Model defaults on the organisation roster for an admin, with how many keys use it.

When the caller names a model#

A request that names one concrete model is served by that model (with the fallback ladder behind it). No strategy chooses, and by default no bandit hears of it.

Learning from it: pointsman.bandit.learn-from-direct (POINTSMAN_BANDIT_LEARN_FROM_DIRECT), off by default. Under observe, every strategy: bandit group with that exact provider:model member is told what the call cost, how long it took and whether it returned:

The tradeoffs:

A model behind another provider is not the member, and teaches the group nothing. Group requests are unaffected either way.

Semantic routing#

Think of a dispatcher who has read every team's one-paragraph job description and routes each incoming request to the team whose description it most resembles. You write the descriptions; the matching is automatic.

Give a group a description: - a sentence saying what traffic belongs there, in the words that traffic actually uses - and the gateway embeds that sentence once at startup, exactly the way the semantic cache embeds prompts. When a blank-model request arrives, the latest user turn is embedded and compared against every group description by cosine similarity. The best match wins if it clears pointsman.routing.semantic-threshold (0.75 by default); the response then discloses the choice:

X-Resolved-Model: sql-helpers (semantic)
X-Semantic-Group: sql-helpers

Below the threshold, or when no group has a description, or while the embedder is still warming after boot, the request simply falls through to the mode tiers - semantic routing degrades to absence, never to an error. A matched group is also intersected with what the key may use: a match whose members are all disallowed or incapable falls through the same way, because a description is a suggestion and the allow-set is a rule.

Choose descriptions and callers with these limits in mind:

• A poor description can select an unsuitable group; inspect X-Semantic-Group on responses.
• Matching uses the current turn alone. A follow-up such as “make it shorter” can miss the original group or select another one.
• Use a named group when conversational traffic must retain that group. Evaluate semantic selection on representative single-turn traffic.

The mode tiers#

After semantic routing, smart routing (off by default) profiles the request, infers a mode (cost / balanced / quality - overridable per request with X-Routing-Mode), and enters the tier you configured for that mode, cascading to cheaper tiers when one is absent. Unknown model names enter the cost tier the same way, disclosed via X-Model-Resolution: fallback.

The extension point#

A strategy is a Spring bean implementing RoutingStrategy (gateway/src/main/java/dev/pointsman/gateway/routing/strategy/). Register a @Component and its bean name is a legal strategy: value - the boot-time vocabulary check accepts any registered bean.