Operational limits and tuning

These are instance-wide startup settings. Change the environment or application YAML and restart that service; this change does not add runtime configuration mutation or tenant overrides. The defaults live in the bean binding declarations. The environment reference is generated from docker/.env.example and docker/learner.env.example; regression tests exercise binding and behavior.

The L1 environment names follow Spring Boot record binding: hyphens in a property name are removed, so max-weight-bytes is POINTSMAN_CACHE_L1_MAXWEIGHTBYTES. Use the exact spellings below.

L1 weight accounting#

effective budget = min(max-weight-bytes, max-size * min-entry-weight-bytes)
entry weight = max(min-entry-weight-bytes, entry-overhead-bytes + 2 * counted UTF-16 units)

The counted payload includes the key, content, reasoning, tool arguments and retained logprobs. Two bytes per UTF-16 unit is a representation constant; the metadata term is a tunable estimate. Caffeine takes a weight ceiling or an entry ceiling, so the per-entry minimum preserves an entry-equivalent bound with weight eviction. Defaults yield 40,960,000 bytes (39.1 MiB), because 10,000 * 4,096 is smaller than 64 MiB. This is approximate payload accounting, not a JVM RSS promise.

For an effective 128 MiB budget, both the byte ceiling and entry-equivalent ceiling must permit it:

POINTSMAN_CACHE_L1_MAXWEIGHTBYTES=134217728
POINTSMAN_CACHE_L1_MAXSIZE=32768
POINTSMAN_CACHE_L1_MINENTRYWEIGHTBYTES=4096
POINTSMAN_CACHE_L1_ENTRYOVERHEADBYTES=1024

The admin effective-configuration view shows configured values and the derived effective budget. An entry heavier than the budget is not retained. Eviction is asynchronous; this is a steady-state cache budget. The L2 memory backend, active requests, SDKs and other caches have separate allocations.

Audit inventory#

All values in the following table are configurable without a source change. Units are part of the names. Invalid nonpositive values refuse startup; the L1 metadata overhead may be zero, and the minimum entry weight cannot exceed its byte ceiling. Large values still require operator sizing.

Learner settings: BANDIT_DB_TIMEOUT_SECS=5.0 bounds SQLite lock waiting, and BANDIT_PENDING_SWEEP_INTERVAL_SECS=60.0 controls cleanup of the dormant decide endpoint's pending state. They are typed and range-validated in LearnerSettings; the live posterior algorithm is unchanged.

Learning capacity (measured 17 October 2026)#

The learner (policy service, one worker by invariant) applies rewards one at a time under one lock, each committed to SQLite. On the 16-core validation host an update took about 6 ms (p99 about 10 ms) whether a key had 2 or 64 arms and whether the store held 100 or 5,000 keys, so it sustains about 150 rewards per second. Above that rate the gateway's reward queue (POINTSMAN_BANDIT_REWARD_QUEUE_SIZE, 1,000) fills and drops the newest rewards, counted as gateway_bandit_rewards_dropped_total{reason="overflow"}; routing is not affected. At 5,000 keys the published parameter document was 1.9 MB (built in 30 ms) and a restart loaded the store in 0.17 s.

Learning has a cumulative horizon. The learner keeps every reward event for idempotent retries and revisions, up to BANDIT_REWARD_EVENT_LIMIT (1,000,000). At the limit it refuses new rewards (503) rather than evict that history, and balanced bandit groups stop learning - they keep routing on the last posteriors. At a sustained R rewards per second the limit is reached after 1,000,000 / R seconds: about 2.3 days at 5 per second, 11.6 days at 1 per second. The limit can be raised to 100,000,000; the store grew by roughly 600 bytes per event in the measurement. There is no archival or checkpoint tool yet, so plan the limit against the expected reward rate. objective: cost groups learn in the gateway and do not use this store.

The gateway's cost sampler is bounded by the three POINTSMAN_COST_SAMPLER_* settings above, and starts again from declared prices after a restart.

Constants deliberately retained#

The audit covers resource capacities, client timeout budgets, cache lifetimes, batching and worker admission. Existing configured values (request body ceiling, output cap, retry timing, retention, queue depth and posterior parameters) retain their existing configuration paths.

Protocol limits (four stop sequences, JavaScript-safe feedback revisions), encoding widths, hash constants, feature-vector dimensions, schema identifiers and numerical guards are not deployment knobs. The tokenizer's named chat-framing allowance remains an estimate of a provider format, not an operator-controlled admission limit. Redis SCAN count is configurable, but Redis may return more or fewer entries than requested. Executor worker count and one-chunk stream prefetch preserve execution and backpressure semantics; the closer's idle timeout has no effect while its core thread is retained. Admin page/range ceilings and bounded diff rendering remain API complexity limits. Changing those contracts warrants API review rather than silently allowing deployment-specific response shapes.

No claim is made that every numeric literal should become a setting. A future operational limit must have an explicit unit, documented default, validation, saturation behavior and a test proving its configured value reaches the consumer. Preserve immutable protocol and algorithm constants by name.

TCP admission before the request gate#

server.tomcat.accept-count defaults to 1024 queued TCP connections, configurable through Spring Boot's SERVER_TOMCAT_ACCEPTCOUNT environment variable. Set a positive integer and restart the gateway. The operating system may cap the effective listen backlog; inspect it on Linux with ss -lnt and check net.core.somaxconn.

This queue is separate from pointsman.intake.max-in-flight, which bounds admitted /v1/ requests. Increasing the TCP backlog does not grant more application permits. It lets short connection bursts reach the request gate instead of waiting for TCP retransmission after the listen queue overflows. It is still finite and cannot make unbounded connection floods or exhausted CPU capacity safe.

Tomcat 10.1 closes HTTP/1.1 connections after a 503 response, so repeated overload refusals create connection churn. On the CPX62 diagnostic workload (200 simultaneous requests, 32 application permits), the inherited backlog of 100 produced approximately one-second TCP retries. An otherwise identical 1024-backlog arm removed captured handshake retries and brought refusal p99 below the unchanged 100-ms gate. This is a measured diagnostic profile, not an independent-generator capacity guarantee.

Measure both client latency and server processing time. A fast filter cannot reject a request that has not yet crossed the TCP connection queue. Keep ingress/network latency, HTTP protocol and connection reuse in the report. See the overload investigation.