# Operational limits and tuning

These are instance-wide startup settings. Change the environment or application YAML and restart
that service; this change does not add runtime configuration mutation or tenant overrides.
The defaults live in the bean binding declarations. The environment reference is generated from
`docker/.env.example` and `docker/learner.env.example`; regression tests exercise binding and behavior.

The L1 environment names follow Spring Boot record binding: hyphens in a property name are removed,
so `max-weight-bytes` is `POINTSMAN_CACHE_L1_MAXWEIGHTBYTES`. Use the exact spellings below.

## L1 weight accounting

```text
effective budget = min(max-weight-bytes, max-size * min-entry-weight-bytes)
entry weight = max(min-entry-weight-bytes, entry-overhead-bytes + 2 * counted UTF-16 units)
```

The counted payload includes the key, content, reasoning, tool arguments and retained logprobs.
Two bytes per UTF-16 unit is a representation constant; the metadata term is a tunable estimate.
Caffeine takes a weight ceiling or an entry ceiling, so the per-entry minimum preserves an
entry-equivalent bound with weight eviction. Defaults yield 40,960,000 bytes (39.1 MiB), because
10,000 * 4,096 is smaller than 64 MiB. This is approximate payload accounting, not a JVM RSS promise.

For an effective 128 MiB budget, both the byte ceiling and entry-equivalent ceiling must permit it:

```dotenv
POINTSMAN_CACHE_L1_MAXWEIGHTBYTES=134217728
POINTSMAN_CACHE_L1_MAXSIZE=32768
POINTSMAN_CACHE_L1_MINENTRYWEIGHTBYTES=4096
POINTSMAN_CACHE_L1_ENTRYOVERHEADBYTES=1024
```

The admin effective-configuration view shows configured values and the derived effective budget.
An entry heavier than the budget is not retained. Eviction is asynchronous; this is a steady-state
cache budget. The L2 memory backend, active requests, SDKs and other caches have separate allocations.

## Audit inventory

All values in the following table are configurable without a source change. Units are part of the names.
Invalid nonpositive values refuse startup; the L1 metadata overhead may be zero, and the
minimum entry weight cannot exceed its byte ceiling. Large values still require operator sizing.

| Environment variable | Default | Meaning / consequence |
|---|---:|---|
| `POINTSMAN_CACHE_L1_MAXSIZE` | 10000 | Maximum entry-equivalent count; the per-entry weight floor enforces it. |
| `POINTSMAN_CACHE_L1_TTLMINUTES` | 15 | L1 TTL in minutes; refresh-on-hit determines sliding versus fixed expiry. |
| `POINTSMAN_CACHE_L1_MAXWEIGHTBYTES` | 67108864 | Estimated payload ceiling in bytes (64 MiB), not a measured JVM heap limit. |
| `POINTSMAN_CACHE_L1_MINENTRYWEIGHTBYTES` | 4096 | Minimum weight per entry. Effective budget=min(max-weight-bytes, max-size * this). |
| `POINTSMAN_CACHE_L1_ENTRYOVERHEADBYTES` | 1024 | Estimated metadata bytes added to key/text UTF-16 bytes; zero is allowed. |
| `POINTSMAN_STREAMING_ASSEMBLY_MAX_CHARS` | 524288 | Maximum cumulative UTF-16 characters accepted while assembling one provider stream; overflow aborts the stream. |
| `POINTSMAN_STREAMING_ASSEMBLY_MAX_TOOL_CALLS` | 128 | Maximum distinct tool-call indices assembled in one stream; overflow aborts the stream. |
| `POINTSMAN_TOKENS_MAX_CONCURRENT_TOKENIZATIONS` | 2 | Concurrent tokenizer-mode requests per instance. Up to `POINTSMAN_TOKENS_MAX_WAITING_TOKENIZATIONS` (16) further requests wait at most `POINTSMAN_TOKENS_ADMISSION_WAIT_MS` (25 ms) for a slot; beyond that, or after the wait, the request receives 503. |
| `POINTSMAN_USAGE_JOURNAL_REPLAY_BATCH_SIZE` | 100 | Maximum terminal journal records delivered per replay pass. |
| `POINTSMAN_USAGE_JOURNAL_REPLAY_INTERVAL_MS` | 1000 | Delay after each journal replay pass and before its first pass, milliseconds. |
| `POINTSMAN_CACHE_CONTEXT_MEMO_MAX_SIZE` | 1000 | Maximum conversation summaries held per instance; eviction only causes recomputation. |
| `POINTSMAN_CACHE_CONTEXT_MEMO_TTL_MS` | 120000 | Fixed conversation-summary memo lifetime, milliseconds. |
| `POINTSMAN_CACHE_REDIS_SCAN_COUNT` | 500 | Redis SCAN work hint for cache clearing; a hint, not a guaranteed batch size. |
| `POINTSMAN_PROMETHEUS_CONNECT_TIMEOUT_MS` | 2000 | Prometheus TCP connection timeout, milliseconds. |
| `POINTSMAN_PROMETHEUS_READ_TIMEOUT_MS` | 3000 | Prometheus response read timeout, milliseconds. |
| `POINTSMAN_BANDIT_PARAMS_TIMEOUT_MS` | 2000 | Background learner parameter-fetch timeout, milliseconds; independent of poll cadence. |
| `POINTSMAN_BANDIT_REWARD_TIMEOUT_MULTIPLIER` | 4 | Reward timeout is existing POINTSMAN_BANDIT_TIMEOUT_MS times this multiplier. |
| `POINTSMAN_BANDIT_BREAKER_WINDOW_SIZE` | 5 | Number of reward calls in the learner circuit-breaker sliding window. |
| `POINTSMAN_BANDIT_BREAKER_HALF_OPEN_CALLS` | 3 | Number of permitted learner reward probes in half-open state. |
| `POINTSMAN_BANDIT_FEEDBACK_WINDOW_MAX_ENTRIES` | 100000 | Local feedback throttle key capacity; eviction can reset a local-only throttle. Use Redis for fleet limits. |
| `POINTSMAN_AUTH_LOGIN_THROTTLE_MAX_ENTRIES` | 100000 | Local failed-login username capacity; eviction forgets that username window. Size for exposed account cardinality. |
| `POINTSMAN_ROUTING_AFFINITY_MAX_ENTRIES` | 32768 | Maximum local conversation pins; at capacity new pins are declined after expired pins are removed. |
| `POINTSMAN_ROUTING_AFFINITY_TTL_MS` | 300000 | Sliding conversation affinity lifetime, milliseconds; this does not extend provider-side prompt cache lifetime. |
| `POINTSMAN_PROVIDERS_MODEL_CACHE_MAX_ENTRIES` | 64 | Maximum dynamic provider/credential client entries; new entries at capacity are refused. |
| `POINTSMAN_PROVIDERS_CLOSE_QUEUE_CAPACITY` | 100 | Queued retired clients awaiting closure; saturation runs the close on the caller, never drops it. |
| `POINTSMAN_COST_SAMPLER_MAX_SCOPES` | 10000 | In-memory cost-learning states (one per tenant and `objective: cost` group, or per organisation and `selection: cost` profile). Beyond it the least recently used state is dropped, which returns it to declared prices; counted as `pointsman_cost_sampler_evictions_total`. Measured 17 October 2026 at the cap with 8 arms each: about 10 MB of heap (about 1 KB per state), a choice in 0.8 us (p99 4.4 us); under continuous churn of 100,000 new states the heap stayed at 12 MB and a new state took p99 1.5 ms (eviction runs in batches of 1%). |
| `POINTSMAN_COST_SAMPLER_MAX_ARMS_PER_SCOPE` | 64 | Members learned per state. A member beyond it is ranked from its declared price and not learned; counted as `kind=arm_cap`. |
| `POINTSMAN_COST_SAMPLER_MAX_LABEL_SCOPES` | 100000 | Quality-label states (`feedback_floor`). Never evicted - forgetting a label could re-admit an excluded model - so a NEW organisation and profile beyond it records no labels (counted as `kind=label_scope_refused`, warned once). Measured 17 October 2026: about 0.7 KB of heap per label state with four labelled models, so the full 100,000 is about 73 MB; on a small heap, lower it to the number of organisations times cost profiles you expect. |

Learner settings: `BANDIT_DB_TIMEOUT_SECS=5.0` bounds SQLite lock waiting, and
`BANDIT_PENDING_SWEEP_INTERVAL_SECS=60.0` controls cleanup of the dormant decide endpoint's pending
state. They are typed and range-validated in `LearnerSettings`; the live posterior algorithm is unchanged.

## Learning capacity (measured 17 October 2026)

**The learner** (policy service, one worker by invariant) applies rewards one at a time under one lock, each committed to
SQLite. On the 16-core validation host an update took about 6 ms (p99 about 10 ms) whether a key had 2 or 64 arms and
whether the store held 100 or 5,000 keys, so it sustains about 150 rewards per second. Above that rate the gateway's
reward queue (`POINTSMAN_BANDIT_REWARD_QUEUE_SIZE`, 1,000) fills and drops the newest rewards, counted as
`gateway_bandit_rewards_dropped_total{reason="overflow"}`; routing is not affected. At 5,000 keys the published parameter
document was 1.9 MB (built in 30 ms) and a restart loaded the store in 0.17 s.

**Learning has a cumulative horizon.** The learner keeps every reward event for idempotent retries and revisions, up to
`BANDIT_REWARD_EVENT_LIMIT` (1,000,000). At the limit it refuses new rewards (503) rather than evict that history, and
balanced bandit groups stop learning - they keep routing on the last posteriors. At a sustained R rewards per second the
limit is reached after 1,000,000 / R seconds: about 2.3 days at 5 per second, 11.6 days at 1 per second. The limit can be
raised to 100,000,000; the store grew by roughly 600 bytes per event in the measurement. There is no archival or
checkpoint tool yet, so plan the limit against the expected reward rate. `objective: cost` groups learn in the gateway
and do not use this store.

**The gateway's cost sampler** is bounded by the three `POINTSMAN_COST_SAMPLER_*` settings above, and starts again from
declared prices after a restart.

## Constants deliberately retained

The audit covers resource capacities, client timeout budgets, cache lifetimes, batching and worker
admission. Existing configured values (request body ceiling, output cap, retry timing, retention,
queue depth and posterior parameters) retain their existing configuration paths.

Protocol limits (four stop sequences, JavaScript-safe feedback revisions), encoding widths,
hash constants, feature-vector dimensions, schema identifiers and numerical guards are not deployment
knobs. The tokenizer's named chat-framing allowance remains an estimate of a provider format, not an
operator-controlled admission limit. Redis SCAN count is configurable, but Redis may return more or
fewer entries than requested. Executor worker count and one-chunk stream prefetch preserve execution
and backpressure semantics; the closer's idle timeout has no effect while its core thread is retained.
Admin page/range ceilings and bounded diff rendering remain API complexity limits. Changing those
contracts warrants API review rather than silently allowing deployment-specific response shapes.

No claim is made that every numeric literal should become a setting. A future operational limit
must have an explicit unit, documented default, validation, saturation behavior and a test proving
its configured value reaches the consumer. Preserve immutable protocol and algorithm constants by name.

## TCP admission before the request gate

`server.tomcat.accept-count` defaults to **1024 queued TCP connections**, configurable
through Spring Boot's `SERVER_TOMCAT_ACCEPTCOUNT` environment variable. Set a positive
integer and restart the gateway. The operating system may cap the effective listen
backlog; inspect it on Linux with `ss -lnt` and check `net.core.somaxconn`.

This queue is separate from `pointsman.intake.max-in-flight`, which bounds admitted
`/v1/` requests. Increasing the TCP backlog does not grant more application permits.
It lets short connection bursts reach the request gate instead of waiting for TCP
retransmission after the listen queue overflows. It is still finite and cannot make
unbounded connection floods or exhausted CPU capacity safe.

Tomcat 10.1 closes HTTP/1.1 connections after a 503 response, so repeated overload
refusals create connection churn. On the CPX62 diagnostic workload (200 simultaneous
requests, 32 application permits), the inherited backlog of 100 produced approximately
one-second TCP retries. An otherwise identical 1024-backlog arm removed captured
handshake retries and brought refusal p99 below the unchanged 100-ms gate. This is
a measured diagnostic profile, not an independent-generator capacity guarantee.

Measure both client latency and server processing time. A fast filter cannot reject
a request that has not yet crossed the TCP connection queue. Keep ingress/network
latency, HTTP protocol and connection reuse in the report. See the
overload investigation.