What you get at the end: nine requests in ten go to the model you trust, one goes to the model you are evaluating, and your application code does not change - it keeps asking for one name.
This is the same mechanism as a canary deploy, an A/B test, and a gradual migration. They differ only in the numbers you write.
Run the Compose commands below from the repository root after the quickstart.
The idea#
A group is a name your callers ask for that stands in front of several real models. When a request arrives for the group, something has to pick one of them - and which something you choose is the strategy.
For a rollout you want weighted-random: it picks at random, but loaded. A candidate with weight 9
wins about nine times in ten against one with weight 1. The randomness is the point. A rule that
sent the first ten percent of requests to the new model would give you the ten percent that arrive
first, which is not a sample of anything.
Define the group#
Groups are boot configuration, supplied as a JSON blob:
SPRING_APPLICATION_JSON='{"pointsman":{"routing":{"groups":{
"chat-tier":{"strategy":"weighted-random","candidates":[
{"provider":"openai","model":"gpt-4o-mini","weight":9},
{"provider":"openai","model":"gpt-4o","weight":1}]}}}}}' \
docker compose --env-file docker/.env -f docker/docker-compose.yml up -d gateway
Persist the configuration#
This is boot-scoped and leaves on the next plain restart. That is a real constraint and it cuts
both ways: an experiment cannot outlive its container by accident, and a split you meant to keep
disappears the next time somebody runs docker compose up without the variable. Put it in your
.env if you mean to keep it.
Wait for readiness before sending anything, or the first requests land between the old container dying and the new one answering:
until curl -sf -o /dev/null localhost:8080/actuator/health/readiness; do sleep 2; done
Send traffic to the name, not the model#
curl -s localhost:8080/v1/chat/completions \
-H "Authorization: Bearer $YOUR_KEY" \
-H 'Content-Type: application/json' \
-d '{"model":"chat-tier","messages":[{"role":"user","content":"hello"}]}'
Your callers only ever say chat-tier. Changing the split is a config change on the gateway, not a
deploy of every application that calls it. That is most of the value.
Check which model actually answered#
X-Provider-Model on the response is the identifier the provider itself reported:
X-Provider-Model: gpt-4o-mini-2024-07-18
Count them across a few dozen requests and you will see roughly your weights.
Interpret the observed split#
A small sample does not establish the configured proportions. With a 90/10 split, the probability that twenty independent choices all select the 90% model is about 12%. Verify the resolved strategy and collect a larger sample before interpreting the split. For example:
strategy=weighted-random chose openai:gpt-4o-mini (decisionId=5924a680-...)
If you want to know that the group is live rather than that the dice were kind, read that line instead of counting. It appears once per request in the gateway log, names the strategy, and would be absent entirely if the group had not resolved.
What you will not see#
There is no X-Decision-Id header on a weighted-random group. That header exists so a caller
can report how an answer turned out, and a random draw has nothing to learn from feedback. It
appears only when the outcome-learning router made the choice. Its absence here is correct, not a
missing feature - see response headers.
Moving the dial#
Change the weights and restart. There is nothing else to do: no draining, no warm-up, no coordination between instances, because each request is an independent draw. Going from 9:1 to 1:1 to 0:1 is three restarts and no application changes.
When you want the gateway to move the dial for you based on how the answers actually turn out rather than on numbers you chose, that is the bandit strategy, and it is a different guide.