# Turn on outcome learning **What you get at the end:** a group name that starts as a fair contest between models and drifts, on its own, toward whichever one actually serves your traffic best - measured by latency, cost, and whatever your application reports back. The response examples below were recorded in an earlier live run. The Compose invocation now follows the repository-root quickstart; this documentation refresh does not represent a new provider run. ## What you are turning on With `weighted-random` you choose the split and the gateway obeys it forever. The bandit strategy inverts that: every response is scored, the scores update a per-model belief, and the next request is routed by sampling from those beliefs. Models that keep doing well get chosen more; models that keep timing out get chosen less; and a small exploration floor (5 percent by default) keeps every model in the game so a recovered model can win its way back. Three signals feed the score, and two of them cost you nothing: **latency** and **cost** are measured on every response automatically. The third - did the answer actually help - only your application knows, and there is an endpoint for telling the gateway. Learning works without it, on speed and price alone; it works better with it. ## Prerequisites Two processes and one flag: ```bash POINTSMAN_BANDIT_ENABLED=true ``` and the policy service (the learner) running - the compose stack starts it as `policy-service`. Run the learner **single-worker** (`WEB_CONCURRENCY=1`): its posteriors live in process memory, and two workers would each hold half the truth. Run the Compose commands below from the repository root after the [quickstart](../tutorial/README.md). ## Define a bandit group Same mechanism as any [traffic split](split-traffic-between-models.md), different strategy: ```bash SPRING_APPLICATION_JSON='{"pointsman":{"routing":{"groups":{ "chat-tier":{"strategy":"bandit","candidates":[ {"provider":"openai","model":"gpt-4o-mini"}, {"provider":"deepseek","model":"deepseek-chat"}, {"provider":"anthropic","model":"claude-sonnet-4-5"}]}}}}}' \ docker compose --env-file docker/.env -f docker/docker-compose.yml up -d gateway ``` No weights - the whole point is that the gateway finds them. Candidates start as equals (a uniform prior), which means **the first stretch of traffic is deliberately a coin flip**. Expect exploration, and do not point a latency-sensitive production path at a fresh bandit group; let it learn on traffic that can tolerate variety, or seed it (below). ## Send traffic, keep the decision id ```bash curl -si localhost:8080/v1/chat/completions \ -H "Authorization: Bearer $YOUR_KEY" \ -H 'Content-Type: application/json' \ -d '{"model":"chat-tier","messages":[{"role":"user","content":"say ok"}]}' ``` ``` X-Served-Provider: deepseek X-Decision-Id: 677b7c2d-af1b-4693-a255-d230d5b77fe9 X-Decision-Receipt: v1.Z3ctZDcKdGVuYW50LTQyOmNvZGUK... ``` `X-Decision-Id` appears **only** when the learning router made the choice - it is the receipt your application holds so it can report how the answer turned out. If you do not see it, the request did not go through a bandit group, and the place to look is the group's `strategy`. `X-Decision-Receipt` is the same decision, signed. Either value works as `decision_id` below. ### Attribute feedback across instances : the bare id is correlated from memory held by the instance that decided, so behind a balancer a rating reaches it one time in N, and the rest used to be dropped. The receipt carries what the learner needs, so any instance can attribute it. It is signed rather than encrypted, it is only usable by the key that made the decision, and it expires with the same fifteen-minute window as the id. Latency and cost for this response are already on their way to the learner at this point. You have participated in learning by doing nothing. ## Report how it went When your application knows the answer was good or bad - the user accepted the draft, the JSON parsed, the customer clicked retry - post the verdict with the receipt: ```bash curl -s -X POST localhost:8080/v1/feedback \ -H "Authorization: Bearer $YOUR_KEY" \ -H 'Content-Type: application/json' \ -d '{"decision_id":"677b7c2d-af1b-4693-a255-d230d5b77fe9","rating":1}' ``` ```json {"accepted": true} ``` `rating` is exactly `1` or `-1`. There is no scale, deliberately: "on a scale of one to ten" answers are noise between integrations, thumbs are comparable. **Feedback is rate-limited to 5 per key per minute by default, and that is a security control, not a throughput bug.** A rating revises the reward of the decision it names, and the quality preference decides how far: at a high preference one rating can swing that decision from a success to a failure. It never makes the decision count as more than one observation, but an unthrottled endpoint would still let a single key re-score every decision it made. If a legitimate integration rates every answer - an agent loop, a batch evaluator - raise `POINTSMAN_BANDIT_FEEDBACK_RPM_LIMIT` for real, but know that you are raising the poisoning ceiling by the same factor. To withdraw a rating, send `{"decision_id": ..., "retract": true, "revision": }` with no `rating`: the decision returns to the reward it had before it was rated. A rating of `-1` is a rating, not a withdrawal. ### Feedback revisions The learner applies a revision only above the one it holds, and the decision log does the same. A delayed retry of an older rating, or an older withdrawal that arrives late, changes nothing. A second operation with the same revision is a retry if it says the same thing, and is ignored as a conflict if it says something else. The decision log's `feedback_score`, `feedback_revision` and `feedback_state` (`rated` or `withdrawn`; `GET /admin/ops/decisions`) record what you submitted and the gateway accepted. They do not say the learner applied it: a learner that was down, or that kept a `rate` decision's only rating (`retraction_without_base`), keeps its own receipt. Rows rated before 18 October 2026 have no revision or state, and the first revisioned operation replaces them. A `503` from this endpoint means the bandit is not enabled - the endpoint tells you about the misconfiguration rather than silently swallowing ratings. ## Watch it learn The gateway's belief surface, live: ```bash curl -s -u "$POINTSMAN_ADMIN_USERNAME:$POINTSMAN_ADMIN_PASSWORD" \ localhost:8080/admin/bandit/state | jq ``` Example state values (these are illustrative observations, not comparative model-quality scores): ```json {"arm": "deepseek:deepseek-chat", "mean": 0.867, "live_observations": 470.7}, {"arm": "openai:gpt-4o-mini", "mean": 0.867, "live_observations": 445.5}, {"arm": "anthropic:claude-sonnet-4-5", "mean": 0.670, "live_observations": 193.2}, {"arm": "kimi:kimi-k2.6", "mean": 0.119, "live_observations": 211.7} ``` (Since the learning semantics `ts2`, 17 October 2026, each workload also reports `learning_semantics`, and each arm `effective_mass`, `observed_decisions` and `rated_decisions`: a rating revises its decision, so rated decisions are a subset of the observed ones, never extra evidence. After the upgrade every workload starts from fair coin flips under its new `|ls=ts2` key; the old beliefs are kept, frozen, under their `ts1` keys.) Read `mean` as the learned score and `live_observations` as how much evidence it rests on (what `alpha` and `beta` are, and why the sum matters as much as the ratio: [the bandit, explained](../design/adaptive-routing.md)). This snapshot shows the thing working: two models tied at the top and taking most of the traffic, one clearly losing and being visited just often enough to notice if it improves. Beliefs are kept **per tenant and per task type** (the `sampling_key`), so one tenant's code-heavy traffic does not teach the router the wrong lesson about another tenant's chat. ## What "fast" means is yours to define The latency score is measured against `POINTSMAN_BANDIT_REWARD_LATENCY_TARGET_MS`, default 2000. At that default, a 120ms model and a 500ms model both score essentially perfect - the learner will see no latency difference between them and decide on cost and feedback alone. If latency differences in that range matter to you, lower the target to the number above which you consider a response slow. This is the single most common reason a bandit "is not learning": it is learning, and the signals genuinely do not differ under your definition. ## If the learner dies Nothing happens to your traffic. The gateway samples from beliefs it holds **in its own process**, refreshed from the learner in the background (every 10 seconds by default) - there is no learner call on the request path. With the learner down, routing continues from the last fetched beliefs (or from fair coin flips if it never fetched any), requests keep flowing, and learning resumes when the learner returns. `GET /admin/bandit/state` reports `fetched`, `version` and `age_seconds`, which is where staleness shows. ## Starting warm instead of cold If you already know roughly which models are good - from an offline evaluation, or from another environment - the learner accepts seeded priors so the fair-coin-flip phase starts informed instead of ignorant. Seeded evidence is tracked separately from live evidence (`seeded_observations` vs `live_observations` above) so you can always see how much of a belief is inherited versus earned. The surface is the learner's own `POST /policy/seed` - deliberately an operator endpoint rather than a startup step, so seeding is a decision someone makes with data they chose, and `strength` states how many observations the prior should count for.