Limits, budgets, and the other controls

Limit rules control requests, tokens and spend before provider dispatch. This guide also explains context admission and provider health. For the underlying flow, see rate limiting and settlement design.

One rule shape, not four features#

Rate limits, token limits, and spend budgets are the same object here: a limit rule. There is no separate budget system to learn - a monthly dollar cap and a per-minute request cap are the same JSON with different words in three fields:

{
  "applies_when": {"model": "gpt-4o"},
  "count_by":     ["credential"],
  "amount":       "cost",
  "budget":       25,
  "window":       {"kind": "calendar", "count": 1, "unit": "month",
                   "anchor_zone": "UTC", "anchor_reset_time": "00:00", "anchor_day": 1},
  "action":       "block"
}

Read it as a sentence: when a request matches (applies_when), count its amount into a counter keyed by count_by, over window; when the counter would pass budget, take action. Every part has a closed vocabulary, and an unknown word is refused at the door rather than stored and ignored:

• amount - what is counted: requests, tokens, input_tokens, output_tokens, or cost (dollars, priced through the same chokepoint the ledger uses).
• applies_when / count_by - eleven attributes: scope, principal, credential, session, agent, task, model, provider, endpoint, environment, request_fingerprint. applies_when decides which requests the rule governs; count_by decides how many counters it fans into. "50 requests a minute per credential per model" is count_by: ["credential", "model"] - one rule, a counter per pair.
• window - rolling (the last N units, sliding) or calendar (resets at an anchor: zone, time, day). A monthly budget that resets at midnight UTC on the 1st is a calendar window; "no more than X in any 10 minutes" is a rolling one.

Rules attach to the tenancy tree - POST /admin/organizations/{org}/rules is the common door - and compose downward: a parent's rule counts a child's traffic too, so a child cannot escape what a parent caps - but the resolver does not enforce "only tighten": rules with identical identities compose tightest-wins, and everything else stacks as independent counters, each able to refuse on its own.

The five actions, and what each is for#

Each action has a distinct effect when its rule reaches the limit.

What enforcement actually is#

Enforcement follows this sequence:

1. Estimate applicable resource demand before calling a provider.
2. Reserve all counters within a Redis slot group atomically, or reserve none of them.
3. If a platform ceiling and tenant rules both apply, reserve the platform group first, then the tenant group.
4. If tenant admission refuses, release the platform reservation best-effort. A lost release can conservatively count high until its window rolls.
5. After work completes, reconcile usage and release unused reservation and concurrency permits.

Atomicity is per slot group, not a distributed transaction across platform and tenant groups. A refusal before dispatch incurs no provider work.

Applied rules expose headroom through X-RateLimit-Remaining-Requests and X-RateLimit-Remaining-Tokens. An absent header means no rule of that kind matched; check the rule's scope if you expected one to apply.

When Redis is down#

Counter availability and counter trust are separate conditions. Cache Redis has its own clients and failure policy; cache fail-open does not make budgets fail open.

Counter-store outage#

The configured budget failure mode determines how capped traffic behaves when the store does not answer.

The optional ledger fallback can answer caps keyed by credential, user, application scope or organization scope. It cannot answer authored deep-scope, platform, model or session caps; these increment pointsman.budget.oracle.unanswerable.

pointsman.budget.ledger-fallback-max-concurrent bounds fallback reads. If no permit is available, the request receives 503. Request-path spend sums have a one-second execution bound. Advisory reads also take permits on the healthy path; fenced requests do not query a balance that cannot change their refusal. Administrative spend reads are a separate operator query.

Counter state cannot be trusted#

An enforcing cost rule returns 503 budget_recovering when its trusted epoch cannot be established. This applies in both failure modes.

The atomic reserve checks detect these conditions:

• The tenant-slot epoch is absent or belongs to another generation.
• The Redis process differs from the one that wrote the epoch, including restart, failover or reload.
• A counter expected inside the epoch is missing.

The fence prevents lost state from appearing as new unspent capacity. Even a clean Redis restart requires recovery evidence. A warn cost rule remains advisory; request and token limits are unaffected by this trust fence.

Recovery: restore or independently attest the balance using the operating procedure. Do not clear counters or restart with empty state to bypass the fence. The initial upgrade adopts existing counters once; this is not a recurring recovery mechanism.

Store checks and recovery signals#

The counter store needs noeviction and an ACL denying destructive commands such as RESTORE and FLUSHALL to the gateway user. Check the configured store-checks mode: reporting and enforcing a failed check have different effects.

The context window#

Input and output share a model's context window. Context admission and budget reservation use the same estimator implementation, including tool definitions; the estimate reports its coverage and uncertainty.

Admission rules#

A capless request is checked for input fit, not a guaranteed answer size. The context-admitted output and the flat budget reservation are distinct values.

Admission policy#

Set pointsman.context.admission-policy according to the behavior you require.

Truncation capability comes from the model row, not its adapter. Without that declaration, both policies refuse the over-ask. Anthropic's model_context_window_exceeded maps to finish_reason: "length" on streaming and non-streaming responses. The caller cannot distinguish a requested-cap stop from a context-window stop through that field alone.

Counting accuracy#

For a routing group, tokenizer mode counts using the candidate rows' encodings and takes the largest result. Every candidate must declare an encoding; a missing encoding produces 400. Select the mode and test representative prompts using the configuration reference.

The context-window pre-check#

A model whose registry row declares context_window refuses an over-long prompt before the network call - a zero-cost 400 naming the limit, instead of a provider round-trip that was always going to fail. A model with no declared window cannot be pre-checked; the failure then comes back from the provider and enters the fallback ladder like any upstream error. Declaring windows on your model rows is the cheap fix.

Provider health, and what routing does with it#

Health is tracked per provider and consumed by routing automatically:

• Probes - a one-token call you trigger from the registry (POST /admin/registry/providers/{id}/probe), advisory, recorded on the row.
• Circuit breakers - repeated retryable failures open a breaker and take the provider out of candidacy until it half-opens. Caller mistakes (4xx-class errors) never open one: one malformed request must not take a provider away from everybody.
• Cooldowns - a 429 puts the specific key on a cooldown clock rather than tripping the breaker: rate pressure heals by waiting, and probing again on the clock is the fastest honest recovery. Billing exhaustion also cools the key; inspect the provider account and cooldown reason.

None of this needs configuration to be safe: the fallback ladder re-derives its candidates against live breaker and cooldown state on every hop. The strategy layer above it - including what each strategy does badly - is routing-strategies.md.

"My requests stopped reaching a provider" - three conditions, three answers#

Use the condition and its evidence to choose the response.

Cooldown timing#

The gateway reads provider timing in this order:

1. Retry-After as seconds or an HTTP date.
2. retry-after-ms.
3. The earliest supported reset header: OpenAI x-ratelimit-reset-* durations or Anthropic anthropic-ratelimit-*-reset timestamps.
4. POINTSMAN_ROUTING_COOLDOWN_SECONDS when no timing is available (default five seconds).

POINTSMAN_ROUTING_COOLDOWN_MAX_SECONDS caps the wait (default five minutes). Logs disclose the provider's requested interval and any clamp. A probe that receives another 429 relearns the interval. Rate limiting is a capacity signal and does not teach answer quality.

Breaker scope#

The provider breaker opens after at least five recorded calls when half or more have failed. It waits ten seconds, then allows three half-open probes. It applies to the provider, while cooldown applies to a credential.

A pre-attempt substitution can report fallback(<model> -> <served>, pre-attempt, unavailable). unavailable deliberately does not expose another caller's detailed failure history. Every fallback still passes health, permissions, context and budget eligibility.

Reading what is in force#

Use GET /admin/control/effective to inspect control values, provenance and whether a restart is required. The dashboard renders this surface; a configured value still needs request-level verification when validating enforcement.