Operating notes

Running more than one gateway? Read multi-instance.md first - it states, for every piece of state, whether it is shared and correct, per-instance by design, or single-instance by invariant.

Choose an operating procedure#

Use the relevant section for your task. The deployment guide describes the full v1 profile.

Preserve state across upgrades#

Preserve the database, counter and journal volumes. Keep a verified backup and the key-hash pepper, rehearse migrations on a restored copy, and verify database names and issued-key compatibility before switching traffic.

Early development renamed illg database/user names and configuration prefixes to pointsman. Recreating disposable development volumes during that rename is not an upgrade procedure for a populated deployment. See supported deployment and recovery scope.

Embedding artifacts and startup dependencies#

The shipped gateway image includes the MiniLM model and tokenizer. A fresh boot can still download DJL native libraries into the persistent model_cache volume.

For custom file: or classpath: model/tokenizer resources, configure pointsman.cache.minilm.model-uri and tokenizer-uri. A remote URI can use temporary container storage; the DJL volume does not automatically persist that download. See startup network dependencies before restricting egress.

Readiness and fallback#

Set a memory limit, or the heap flags mean nothing#

Set both the container limit and the intended heap policy. Without a container limit, JVM percentage-based ergonomics can size the heap against host memory rather than the deployment's budget.

docker-compose.minimal.yml contains dependency services only. It does not define a gateway whose memory it could bound.

What 2 GB buys, measured under a real cgroup#

Ubuntu 24.04, cgroup v2, the shipped entrypoint unmodified, MiniLM embedder, 15,000 cache-miss requests at 50 RPS:

The JVM picked 25 percent of the limit in this profile. Its sampled latency was similar at 2g and 3g; that does not establish spare capacity or equal behavior under larger prompts, more concurrency, or a different native-embedding workload.

Read memory.peak carefully#

At 2g the measured peak reached the limit. Cgroup memory includes anonymous memory, file cache and kernel charges. File cache can often be reclaimed, but reclaim can cost latency; reaching the limit without an OOM is evidence of completion, not a headroom qualification. A separate 3g sample showed:

anon  1378 MB     heap, native, direct buffers, metaspace, thread stacks
file  1308 MB     page cache: the jar, the ONNX model, DJL native libraries

The 3g sample cannot establish 670 MB of available headroom in a 2g deployment. Measure the assembled candidate at its actual limit, retaining cgroup pressure/reclaim, RSS, heap and native-memory series. No minimum safe memory size is established by subtracting this sample from a different limit; use the resource qualification and its workload boundaries in the readiness record.

Direct buffers are pinned, and it is a guard rather than a saving#

When MaxDirectMemorySize is unset the JVM sets the direct-buffer ceiling equal to the max heap. Since the max heap is itself implicit here, that ceiling silently tracks the container limit: move the limit from 2 GB to 4 GB and the ceiling goes from 512m to 1g with nothing written down. The Dockerfile now pins it at 512m so a limit change moves one number instead of two.

It does not free memory on Linux. Measured direct-buffer USE plateaus and does not track the heap:

Adding a 256m cap at a 1024m heap moved total committed memory by 15 MB, which is noise. 512m rather than 256m because measured peak use reached 213 MB at only 50 RPS - 83 percent of a 256m cap. The 512m limit is about 2.4 times that observed direct-buffer peak; this ratio is not total-process headroom or a concurrency capacity estimate. Recheck it on the intended workload before reducing the bound.

Leave MaxRAMPercentage alone#

The measured profile used MaxRAMPercentage=25. Keep heap and total process limits explicit in the qualification record; change this setting only with measurements of GC behavior and the native/direct-memory needs of the intended workload.

A warning about numbers measured on macOS#

Measure on the deployment platform. Native allocation, direct-buffer commitment and thread-stack accounting differ between macOS and Linux; a memory model fitted on a developer laptop does not establish Linux cgroup capacity.

The unset direct-buffer ceiling follows the heap on both platforms, but observed committed memory need not scale with that ceiling. Use the actual workload's RSS, cgroup pressure, heap and native-memory series.

Size the local caches against the heap#

ExactMatchCache now bounds L1 by estimated payload weight and an entry-equivalent count. The defaults are a 64 MiB weight ceiling, 10,000 entries and a 4 KiB minimum entry weight. The effective weight budget is therefore 40,960,000 bytes (about 39.1 MiB). All three are configurable; see operational limits for the formula and examples. Weights include keys, response text and a configurable metadata estimate. They are not exact heap measurements. Measure GC and retained heap on representative responses when setting this budget.

The separate in-memory L2 response store, CaffeineResponseStore, still uses the L1 max-size as its entry count. Its whole-response objects and the in-memory vector index consume additional heap outside L1's weight budget. The Redis L2 backend moves those shared stores off the JVM. TTL bounds age; it does not replace capacity planning for either tier or the process as a whole.

Gateway target alerts#

The gateway-availability group in docker/grafana/alert-rules.yml monitors whether Prometheus can scrape the gateway. It does not depend on optional Jev enablement.

Both wait GATEWAY_TARGET_DOWN_FOR before firing; the shipped default is 5 minutes, so a restart or a short network blip does not page. Both are critical because every other gateway alert (journal, budget, invariant, Jev) reads series from this scrape: while either holds, those alerts are blind, not quiet.

Change the alert duration#

Prometheus rules cannot read configuration, so the value is rendered into the rule file at deployment time, the same way the Jev values are. Edit GATEWAY_TARGET_DOWN_FOR in docker/grafana/alert-policy.env (a Prometheus duration such as 2m or 10m), run scripts/render-alert-rules.sh (it rewrites only the block between the gateway-availability-rules markers), optionally scripts/alert-rules-test.sh (promtool checks and unit tests; those tests assume the shipped 5m), then make Prometheus reload its rules: in the shipped compose stack, docker compose -f docker/docker-compose.yml restart prometheus (or send it SIGHUP). JevAlertRulesRenderedTest refuses a commit whose rendered block disagrees with the policy file, and scripts/render-alert-rules.sh --check exits 1 in the same case.

Alert coverage#

What is and is not paged. Three situations are kept apart:

• A failed expected target (up == 0): GatewayTargetDown, one alert per instance.
• A disappearing expected target: when a target leaves discovery or Prometheus loses it, its up series goes stale rather than reading 0, so up == 0 can never say so. GatewayTargetAbsent covers it, for the whole job: it fires when no gateway target is left. With several gateway targets, one of them vanishing is indistinguishable from an intentional scale-down or a replaced pod, so it is not paged; a deployment with a fixed fleet size should add its own rule on count(up{job="gateway"}).
• Optional components that are not deployed: the group reads job="gateway" and nothing else. The learner (policy-service), tracing (the OTel collector and Tempo) and Jev never raise an alert from this group, whether they are down, absent or never configured.

Prometheus job naming#

This rule file is for a Prometheus whose gateway job is expected, as in docker/grafana/prometheus.yml. On a Prometheus that is not configured to scrape a gateway, or that names the job differently, GatewayTargetAbsent fires permanently and GatewayTargetDown can never fire; rename the job in the rules (the templates under docker/grafana/) or do not load this file there.

Combine availability and Jev alerts#

On an instance that had Jev enabled in the last hour, a scrape failure fires both GatewayTargetDown (critical, the page: the gateway is unreachable) and JevTargetDown (warning, the context: the Jev panels for that instance are unobservable). This is deliberate - the Jev rules are kept byte-identical and the availability rule must not depend on a feature flag. If your Alertmanager should send only one notification, inhibit the warning by the critical one:

inhibit_rules:
  - source_matchers: [alertname="GatewayTargetDown"]
    target_matchers: [alertname="JevTargetDown"]
    equal: [instance, job]

Monitoring-system failure#

Rules evaluated by a Prometheus cannot report the loss of that Prometheus, of its host, or of the path from it to whoever receives the notification: if the monitoring host goes down, nothing in this file fires. Detecting that needs something outside it, such as an always-firing watchdog alert routed to an external dead-man's-switch service, or a second Prometheus that scrapes the first. The shipped compose stack also has no Alertmanager: alerts are visible on Prometheus's own alerts page until a deployment routes them somewhere.

Gateway memory alerts#

The gateway-memory group in docker/grafana/alert-rules.yml watches the gateway container's memory as the kernel accounts it, not the JVM heap. Its source is the gateway's own cgroup v2 files, read by CgroupMemoryMetrics at most once per pointsman.observability.cgroup-memory.min-interval-ms (5 s):

Interpret memory measurements#

What the number is, and is not. memory.current - inactive_file = anonymous + shared + active file cache + all kernel memory (reclaimable and unreclaimable slab, kernel stacks, page tables, sockets). It is therefore AT OR ABOVE the v1 resource policy's footprint (anonymous + shared + unreclaimable kernel memory, 80 % of the limit; docs/how-to/deploy.md), so GatewayMemoryPressure warns early, never late, and can fire while the kernel could still reclaim part of what it counts - this gateway has grown the dentry cache (reclaimable slab) before. It is an early operational warning, not a measure of the policy footprint and not proof of an imminent OOM: on a page, read memory.stat (anon, shmem, slab_unreclaimable, slab_reclaimable, active_file) to see what is actually held. The 0.80 reuses the policy's percentage; the frozen qualification resource policy keeps its own formula and is not replaced by this alert. Raw memory usage alerts constantly (the kernel keeps the cgroup near its limit with page cache); the heap is a quarter of the limit and misses direct buffers, native and kernel memory.

Memory-alert coverage#

What this covers, and what it does not. The metric is the gateway process's own cgroup: on a cgroup v1 host, a gateway outside a container, or a container without a memory limit, it is unobservable and says so (GatewayMemoryUnobservable), rather than reporting zero. It does not watch the other containers (PostgreSQL, the two Redis stores); a deployment that needs them alerted adds a container-metrics collector (cAdvisor, a Kubernetes kubelet), and must alert on that collector's own absence the same way. Disabling the reader (POINTSMAN_OBSERVABILITY_CGROUP_MEMORY_ENABLED=false) removes the series, which GatewayMemoryUnobservable reports.

What to alert on when the counter store fails open#

Rate limits may fail open when Redis is unreachable. With the default pointsman.budget.failure-mode=deny, requests with an enforcing spend cap or unknown cap applicability refuse 503 budget_unverifiable without querying the ledger. Uncapped requests can continue. Only the explicit allow mode can use historical ledger estimates, without atomic reservations. Counter-trust loss is a separate fence: budget_recovering refuses capped work in either mode.

Use these signals together:

• pointsman_limits_engine_errors_total{stage="attempt"} increments once per request that reached reservation and received a store error. It is not a count of successful fail-open admissions: capped requests can instead be refused. Correlate it with budget refusals and client outcomes.
• The counterStore health indicator reports DEGRADED with consequence: rate limits fail open, and logs the transition once per state change. It probes only when /actuator/health is read.
• There is NO gauge that says "failing open right now". The budget side's pointsman_budget_failclosed_active is, despite its name, a counter of outage admission decisions, including uncapped requests and denied capped requests. It does not mean the ledger was queried and is neither a current-state gauge nor a count of successful admissions. pointsman.budget.unverifiable_refusals is the refusal half on its own. The clean signal would be a 0/1 gauge on RedisBreaker's open state, which is what actually decides fast-failing; it is not implemented. Until then, alert on the rate of the errors counter above.
• The Lettuce connection pools (commons-pool2, pointsman.redis.pool.*) register no metrics; pool exhaustion shows up only as RedisConnectionFailureException in the same errors counter. Micrometer's CommonsObjectPool2Metrics binder would expose them once the pools are JMX-enabled.

Run the counter Redis with AOF persistence#

The counter store holds rate and budget counters. Use AOF persistence with a declared fsync/durability policy, noeviction, and the ACL restrictions verified by enforced store checks. Persistence reduces data-loss exposure; it does not reconstruct missing reservations or establish consistency with a separately restored database and journals.

A Redis restart changes its run ID. The trusted-epoch check refuses budgeted traffic with budget_recovering until recovery or operator attestation supplies a valid balance, even if the restart appears clean. An in-process rollback is why ACL and eviction-policy checks matter as well. Do not delete the fence or assume a PostgreSQL ledger sum alone can repopulate every counter safely.

During an unreachable-store outage, the default deny policy refuses capped traffic without reading the ledger; allow is an explicit availability tradeoff with possible overspend. Neither setting silently authorizes a lost-trust epoch. See the manual procedure below and gateway controls.

Running against Redis Cluster#

Cache and counter topology#

Set the endpoint mode explicitly. Counter and cache endpoints have different cluster support; Redis Cluster remains outside the qualified v1 profile.

Declare the mode explicitly per endpoint; it is never guessed from the shape of the host list, because a typo that silently produced standalone behaviour against a cluster would work perfectly until a key moved to another shard.

pointsman:
  redis:
    counter:
      mode: cluster
      nodes: "redis-1:6379,redis-2:6379,redis-3:6379"
    cache:
      mode: cluster
      nodes: "redis-1:6379,redis-2:6379,redis-3:6379"

Under the shipped compose stack the same settings arrive as environment variables, which is the door docker/.env.example documents:

REDIS_COUNTER_MODE=cluster
REDIS_COUNTER_NODES=redis-1:6379,redis-2:6379,redis-3:6379
REDIS_CACHE_MODE=cluster
REDIS_CACHE_NODES=redis-1:6379,redis-2:6379,redis-3:6379

Note the compose stack itself runs redis/redis-stack, a single-node distribution that cannot run cluster mode. Cluster means your own Redis 8; the stack cannot demonstrate it.

Atomic reservations across hash slots#

Rate limiting reserves every counter a request touches in ONE atomic script. That is not a performance choice: the script checks every counter has room before it writes any of them, which is what stops a request half-spending a budget it was about to be refused for. Under cluster mode every key in one such call must live on one hash slot.

Counter keys use two kinds of hash tag:

A request spans at most two slots, with one call per slot. Giving a shared ceiling a tenant tag would split it into independent tenant ceilings; giving every scope its own tag would scatter a request across slots.

The second call costs one extra round trip and only where a deployment configures a deployment-wide ceiling - global-rpm, global-tpm and global-budget-micros are unset by default, so most deployments pay nothing at all.

What an operator sees on deploy#

Counter keys are renamed by this, because the hash tag is part of the key. Two consequences, both bounded:

• Rate counters reset. They are all TTL'd within a minute, so the worst case is one window in which rate limits are looser than configured. No action needed.
• Cost counters do not rebuild themselves from usage_events. A renamed cost counter is a counter the budget fence has not seen: a tag at generation 1 starts it at zero, and in a later generation it fences (503 budget_recovering) until attested. Redis Cluster is outside the v1 profile and the reconcile applier refuses it; do not rely on this section for budget correctness. Reopen affected tags through the attestation procedure below, reading GET /admin/budget/trust/{tag}/evidence.

Budgeted traffic remains refused wherever the recovery fence requires attestation; a restart alone does not establish safe balances.

The one thing that still refuses#

pointsman.redis.cache.mode=cluster refuses when pointsman.cache.vector-backend=redis. The L2 semantic tier issues RediSearch queries through a client whose cluster interface does not expose them, so it would dial one node and miss most of its data rather than failing. Either run the cache endpoint on a single Redis, or set vector-backend=memory - the L1 exact-match tier is unaffected and keeps working. The refusal names both ways out.

Recovering budget trust after counter-store loss#

This is the manual-attestation procedure for the documented default recovery scope. It is not automatic spend reconstruction, and the small retained restore drill does not establish a production-volume RTO/RPO. Rehearse on disposable restored stores before using it in an incident.

Recovery scope#

• RPO. The ledger's recovery point is the last supported stopped-write PostgreSQL backup. No point-in-time recovery is provided or tested. Spend acknowledged after that backup exists in the gateway only where a surviving store still holds it - a retained counter Redis, the journal (held records, pending records, quarantine) - and otherwise only in provider usage and billing records.
• Recovery is manual and evidence-based. The gateway refuses budgeted traffic for every tag it cannot prove; an operator reopens each tag by attesting its balances. Where the backup gap cannot be reconstructed, attest conservative exhaustion (the counter's cap): the period's remaining allowance is zero until it rolls over.
• The gateway checks a floor, not completeness. An attestation below what the gateway's own durable evidence proves is refused (409). Spend after the restored backup is invisible to it, so passing the floor never proves the backup gap was included.
• Not promised: automatic reconstruction, point-in-time recovery, or a hard cap matched to provider invoices.

1. Pause ingress and preserve evidence#

Pause new ingress for the affected deployment and preserve incident evidence. Drain what can finish; retain the database backup/WAL, counter-store persistence, each replica's own durable journal, and provider request/usage records. Capture the failure time and the last consistent backup boundary. Do not share one journal between replicas or let one restore consume the only failure-point copy.

2. Restore stores and declare the restore#

Restore the intended database and store configuration with the same key pepper. Keep gateway replicas attached to their own restored journal volumes. A journal record that cannot be parsed, such as a truncated copy, is moved aside intact as <name>.unreadable. It is counted in gateway_usage_journal_unreadable_total and named in a warning, and the rest of the journal still replays. Its usage was not delivered, so reconcile it from the file before treating the restore as complete. Verify the enforcing counter/cache store checks and retained generation/fence state before reopening traffic. Do not delete trust rows or disable the fence to suppress 503s.

Declare every database restore before traffic resumes. A retained counter store alone does not establish a complete balance.

Declare a recovery boundary#

Start every gateway instance with POINTSMAN_BUDGET_RECOVERY_RESTORE_ID=<restore or backup id> and confirm the boundary in GET /admin/budget/recovery. Every tag the restored database holds TRUSTED is fenced at boot (FENCED, fenceReason declared_restore): a retained counter store is not proof of a complete balance - a settlement whose counter adjustment was still deferred, or lost, leaves the counter below the recorded usage.

The restored database has no trust row for a tenant whose first budgeted request came after the backup, and it cannot tell that tenant from one that never spent; without the declaration such a tenant would start again at zero and its post-backup spend would be granted twice. With it, a tenant first seen without a trust row whose scope predates the boundary (or whose creation time cannot be read, such as the shared platform tag) starts RECOVERING: refused 503 budget_recovering before any provider call until you attest it in step 4. Tenants created after the boundary start normally.

If the boundary cannot be recorded the gateway refuses to start; the same id at a later restart is a no-op. The gateway also records a COUNTER_STORE_LOSS boundary on its own when a previously active tag's epoch is missing from the counter store, but that safety net sees nothing in a deployment with no active epoch, so the declaration is the supported step. The boundary records WHICH scopes exist at that moment (not a timestamp), so clock skew between hosts cannot make an old tenant look new.

Use a new id for every restore: the same id at a later restart is the same incident and a no-op, while a reused id for a new restore keeps the old membership and would not list tenants created since.

A declared-restore boundary cannot be closed (409): its tenants not yet seen are protected only at their first request, so each is resolved by attestation when it appears. A boundary recorded for a counter-store loss on a database that was NOT restored can be closed with POST /admin/budget/recovery/{id}/close {"reason": ...}; closing stops it holding tags first seen later and never reopens a tag it already holds.

3. Review each tag’s evidence#

Read the trust inventory with a root-admin identity. FENCED or RECOVERING tags require investigation (RECOVERING tags held by a recovery boundary appear here on their first request after the restore, not before). For each, read the evidence report, GET /admin/budget/trust/{tag}/evidence: per cost counter of the current generation, the generation's starting balance (init, or UNKNOWN), settled charges, held journal records (settlements whose rows the restore lost), outstanding and uncertain reservations, deferred adjustments, the live counter value (information only), the lowerBound/upperBound, a status (EXACT, BOUNDED, UNKNOWN) and the attestationFloor.

Under an open recovery boundary backupGap says spend after the backup is not observable. Then list the held records, GET /admin/budget/held. A database ledger total alone cannot supply the balances: add post-backup acknowledged spend from outside the gateway (provider usage), outstanding reservations and unresolved provider outcomes. A counter-minus-ledger difference does not identify which requests are missing. If the balances cannot be justified, leave the tag fenced or attest the cap.

4. Prepare the attestation#

Prepare a reviewed attestation file containing expected_generation, evidence_ref and balances. Copy the generation from the trust inventory whose incident evidence you reviewed. A 409 generation mismatch requires a new review, not replacing the number and resubmitting the old balances. Missing/invalid generation is a 400. Values are integer micro-dollars already charged/reserved to each cost counter, not remaining budget.

Keys must be the exact counter keys belonging to the selected tag. Do not copy a backup-time snapshot without accounting for the backup gap; the retained drill explicitly measured spend that such a snapshot failed to recover. An empty map is not evidence that spend was zero. Preserve the review, inputs and checksums with the incident record.

A balance below the counter's attestationFloor is refused 409 and writes nothing: raise it to at least the floor (the floor is init + settled + held + outstanding, or without init when that is UNKNOWN). A counter you omit from the map stays refused until it is attested or its period rolls over.

5. Submit, verify activation and archive included evidence#

Submit the attestation once, retain its returned generation/recovery ID, and re-read state if the response is uncertain before deciding whether to retry. The database transaction and Redis epoch installation are separate steps. A successful response is not a complete recovery test; confirm activation and all store checks on each replica.

Archive included held records#

The attestation records which held records its balances included. Once a held record is included for every counter it names it is archivable: archive it with POST /admin/budget/held/{event_id}/archive {"evidence_ref": ...}. The record moves intact into the journal's a/ directory under its sha256 (<event>.<sha256>.held.json), stops being retried, and frees its journal slot; a repeat is a no-op; it is never deleted. load-test/journal-records.py <journal> count --selector archived lists it.

Keep the archive available for later restores#

Archived records are never replayed. The gateway reads the archive without modifying it: if the database is later restored to before the attestation that included a record, that inclusion is gone, and the archived record counts again - once - in its counters' evidence floor, appears in GET /admin/budget/held as archived: true, and fires BudgetEvidenceHeld until an attestation includes it again (attest as above; there is nothing to re-archive).

Retention and repair requirements:

• Keep a/ on the journal volume for as long as any restorable database backup predates the attestation that included its records. Copy it to the incident record, but do not remove it from the volume before then.
• If it was removed, copy it back before attesting after a restore. Without it the gateway cannot see that spend: it is then part of the backup gap - supply it from the incident record or provider usage, or attest the cap. The gateway cannot tell that an archive is missing.
• A corrupt or conflicting archive file (bytes not matching the sha256 in its name, an unreadable record, one event archived twice, or an event both held and archived) is listed in the evidence report's archiveProblems and REFUSES every attestation (409) until reviewed. Review it against the incident record; move only a file you have established to be a bad copy out of a/, and supply the spend it described yourself.

6. Verify before reopening#

Admit a small identified verification cohort, including an exhausted-budget key. Check that the exhausted key is refused without provider dispatch, admitted work reconciles by request/attempt identity, and journal/backlog counters drain without duplicates. Compare counters to the reviewed balances plus this cohort's charges. Reopen gradually only after those checks. Keep unresolved work visible; manual attestation does not create recovery inclusions for old owed operations.

Correct an existing trust attestation#

Correcting a trusted tag without a restore. When an operator knows a TRUSTED tag's counter is wrong (a lost correction, say), fence it: POST /admin/budget/trust/{tag}/fence {"expected_generation", "evidence_ref", "reason"}. PostgreSQL is marked first, then that generation's Redis epoch is deactivated so every replica refuses at once, then the local cache is dropped. Requests already admitted settle normally. Reopen by attestation as above.

Unresolved provider attempts#

Requests in flight when a gateway process died (BudgetReservationUnresolved). A gateway killed with requests in flight (SIGKILL, OOM, host loss) and restarted on its own journal writes, at boot, one request row per in-flight request - status 503, error_type accounting_interrupted, usage UNKNOWN, zero cost - and one UNKNOWN provider dispatch row per send that had begun. Their budget reservations stay ADMITTED, held on the counter at what was reserved. Nothing is refunded automatically: the provider may have received and billed those requests, and a process dying is not evidence that it did not. With the reconcile applier on, the reservation resolver marks each such reservation reserve_unresolved:interrupted_dispatched (a dispatch row exists) or reserve_unresolved:interrupted_no_dispatch_record (none has landed - missing evidence, not proof that nothing was sent) once it is older than reserve-resolve-delay-ms, without changing its state, the counter or the tag's trust. They count in pointsman_budget_reservation_unresolved, and BudgetReservationUnresolved fires after 15 minutes. The tag is not fenced for this and keeps serving with the holds in place; a period rollover does not clear the alert.

1. List them. GET /admin/budget/trust/{tag}/evidence returns unresolvedReservations: the total and the oldest 100, each with requestId, counterKey, generation, reservedMicros, reason, createdAt and the request's dispatches / dispatchOutcomes. The same holds are already inside each counter's outstanding, so they are in the attestationFloor. The gauge is per instance and counts every tag; the report is per tag.
2. Establish what the provider did, per request id, from provider request and usage records (the dispatch rows name provider, model, credential and send time). reservedMicros is the amount held, not a maximum of the bill: a provider can bill more than the estimate that was reserved.
3. Decide. If provider usage accounts for every listed request, fence the tag (POST /admin/budget/trust/{tag}/fence with the generation from the report), re-read the evidence and attest each counter at the floor plus whatever the provider billed above the reserves. A request billed less than its reserve, or not at all, cannot be given back by attestation: the floor includes the full reserve and a balance below it is refused 409. If the liability cannot be established, keep the tag fenced, or attest the cap (conservative exhaustion: no allowance until the period rolls over).
4. Confirm. The attestation installs generation + 1; the reservations of the older generation stop counting and the alert resolves; the rows themselves stay as they were, as the record. Repeating an attestation with the old generation is 409 and writes nothing. An interrupted request cannot settle later beside its 503 row (the delivery is refused as a conflict); a slow request of the old generation that does settle after the attestation has its adjustment PRESERVED (never applied under the attested balance, never voided) and fences the tag again, as for any older-generation settlement. Never edit the rows or the counter store by hand.

Alerts (docker/grafana/alert-rules.yml, group budget-recovery): BudgetTagFenced (a tag refuses until attested), BudgetReconcilePreserved, BudgetReservationUnresolved (15 minutes), BudgetStoreChecksFailing, BudgetEvidenceHeld (held settlement evidence exists: list, attest, archive) and RetentionRetainingUnresolved. Each fires on a gauge that returns to zero only when the condition is resolved.

The current administrative endpoints are:

# ADMIN_CURL_CONFIG: private mode-600 curl config containing the admin credentials.
# ADMIN_URL: the protected administrative origin; TAG_URI: URL-encoded inventory tag.
curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  "$ADMIN_URL/admin/budget/trust" > trust-before.json

curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  -H 'Content-Type: application/json' --data-binary @attestation-reviewed.json \
  "$ADMIN_URL/admin/budget/trust/$TAG_URI/attest" > attestation-result.json

curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  "$ADMIN_URL/admin/budget/trust/$TAG_URI/evidence" > evidence-before-attest.json

curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  "$ADMIN_URL/admin/budget/held" > held-records.json

The endpoint requires a positive matching expected_generation, a nonempty evidence reference, nonnegative balances, a known fenced/recovering tag, and counters with that tag's prefix. Those validations cannot prove the operator's balance reconstruction is complete or detect a database rollback that restores both balances and generation. Its audit records the attestation; do not mistake an accepted API payload for proof that no budget was regranted. Record the measured restore duration and recoverable data window for the actual dataset. See the budget recovery design and restore evidence.

Quality evidence storage (retention off by default)#

Quality-guarded routing is opt-in and off by default. When it is enabled, its evidence tables (quality_events, quality_aux_attempts, quality_audit_drops) grow without bound, because their retention job is disabled by default. Choose retention explicitly for your deployment. Disabled retention is not a disk cap.

• Visibility. pointsman_quality_retention_rows{table} reports each table's row count hourly (-1 = unreadable). Size the database volume for your evaluation traffic and alert on growth against your own threshold.
• Operator action. To bound growth, enable POINTSMAN_QUALITY_RETENTION_ENABLED with lifetimes you have chosen (docs/reference/configuration.md). Retention never deletes the events of a decision an audit job selected, nor an unresolved (RESERVED or UNKNOWN) auxiliary attempt; epochs, policy versions, validators and audit jobs are never touched. Test any lifetime on a restored copy first.
• Budget recovery evidence is separate. The journal's archive of held settlement evidence (a/) is kept on the journal volume while any restorable backup predates the attestation that included it (see "Recovering budget trust" above); size the journal volume for it.

Local embedding CPU#

The performance candidate uses three settings when creating the local ONNX session:

Environment names are POINTSMAN_CACHE_MINILM_INTRA_OP_THREADS, POINTSMAN_CACHE_MINILM_ALLOW_SPINNING and POINTSMAN_CACHE_MINILM_FORCE_SPINNING_STOP. Changes require a gateway restart. They apply to auto and forced minilm, not hosted or hashing embedders. The previously qualified runtime shipped allow-spinning=false and did not expose explicit worker sizing or stop-on-return. Its acceptance does not cover these new defaults.

Explicit sizing avoids ONNX's automatic host-wide worker affinities targeting CPUs outside a container cpuset. Spinning during active inference reduces wake-up cost; stop-on-return prevents that policy from consuming idle CPU. CPU cost and latency still depend on sequence length, concurrent requests and CPU limits. Compare both under your workload; a short benchmark is not a capacity or readiness guarantee.

Spring AI 1.1.8 creates private SessionOptions without a public customization hook. SessionConfiguredEmbeddingModel is an Apache-2.0 adaptation that sets worker options before session creation. Model/tokenizer resources, input IDs and masks, execution mode, mean pooling, normalization and vector namespace are unchanged. The model precision and context window are unchanged.

The real-model regression compares single and mixed-length batches, full-window, Unicode and empty inputs against the pinned upstream implementation at the existing 1e-6 component tolerance. It also exercises overlapping calls and repeated worker parking. Cache similarity, lexical verification, tenant isolation and replay checks are unchanged. See candidate measurements and limits.

No reflection or dependency upgrade is used. Keep the adaptation aligned when upgrading Spring AI; remove it when equivalent public session customization is available. It does not resolve upstream resource-lifecycle limitations.

ONNX defines stop-on-return in its 1.20 session option source.

Upstream source: Spring AI 1.1.8 source archive. Original TransformersEmbeddingModel.java SHA256: e18f8e56cc29ed3451554e8101b38cdabf01b0a0fd6f98df9b29f8b9ef00b74e. ONNX option semantics: ONNX Runtime 1.20.0.

Bound MiniLM eligibility work#

Before MiniLM counts tokens to check its actual embedding window, pointsman.cache.minilm.max-check-characters limits the input to 16,384 UTF-16 code units and pointsman.cache.minilm.max-concurrent-window-checks allows two checks at once per embedder. Both are configurable positive integers; invalid values fail startup for auto/MiniLM selection. These are resource limits for optional cache work, separate from the model's derived token window. Raising them increases possible native allocation and must be qualified under load.

An oversized input or busy check bypasses semantic caching immediately. It does not reject the provider request or change provider context accounting; exact L1 reuse remains available. Inputs within these limits still require the full, untruncated native token count to fit the model window. This can conservatively skip a whitespace-heavy prompt that would fit in tokens, and contention can reduce the semantic hit rate. The existing gateway.cache.l2_skipped{reason="window"} metric and SKIPPED_WINDOW outcome include this inability to establish window eligibility; they do not mean every skipped input has too many tokens.

The limits bound an identified optional-work path; the limits alone are not proof of resource stability. The documented v1 profile subsequently passed its frozen 12-hour and 24-hour resource qualification; see the final record for measured bounds and retained diagnostic failures. Changes to these limits or the workload require qualification of the affected resource behavior.

Tokenizer admission under short bursts#

Provider tokenizer mode keeps pointsman.tokens.max-concurrent-tokenizations=2 active requests per instance. Up to pointsman.tokens.max-waiting-tokenizations=16 additional requests may wait at most pointsman.tokens.admission-wait-ms=25 ms for an active slot. These defaults absorb brief overlap without raising tokenizer CPU concurrency. Both new limits are nonnegative; either zero restores immediate refusal. A full queue, expired wait or interruption returns retryable 503 with error.code=tokenizer_busy. Interrupted threads retain their interrupt status. There is no fallback to heuristic counting. The wait budget excludes execution and scheduling delay and is additional request latency, not a response deadline.

These settings are included in the final v1 profile qualified on 28 October. They do not guarantee admission under sustained overload. Keep the zero-unexpected- failure qualification gate for the declared supported workload; do not waive it merely because overload refusal is intentional.

Optional Jev task routing: data export and operator-paid overhead#

Off by default, and it stays off unless you set POINTSMAN_JEV_ENABLED=true and name the tenants, the credential and a task-to-model ranking - the gateway refuses to start with any of the three missing. Once on, every request from a tenant in POINTSMAN_JEV_ALLOWED_TENANTS that reaches a strategy: jev group without a valid X-Task-Type hint has its accepted conversation text - system and developer messages included - sent to the configured TypeSafe endpoint. Obtain those tenants' permission first; the export is the feature, not a side effect. Requests that declare tools, carry images or tool results, or exceed the input bounds are never exported (reason tools_declared, unsupported_context, input_limit).

Classification calls are operator overhead. They never appear in any tenant's usage ledger or budget, and a call whose usage the provider did not report is counted under pointsman_jev_usage_unverified_total, never as free. The attempt, rate and concurrency bounds are per process: they reset on restart and multiply with replicas, so allocate the account's limits across replicas yourself.

Watch pointsman_jev_invocations_total{outcome=...} (one count per classifier entry: classified, abstained, skipped, refused, failed) beside pointsman_jev_decisions_total (one per routing decision, and larger, because a request's assessment serves every tier), and the jev-classifier alert rules (per instance; rendered from docker/grafana/alert-policy.env). A successful classification or hint can select a later-ranked candidate; every skip, refusal, dependency failure or abstention falls back to the first eligible candidate, so none of those is a request failure - the alerts say the feature is not doing its job, which is a different page from "requests are failing". Setup, bounds, the metric vocabulary and the evaluation to run before trusting a ranking: docs/how-to/jev-routing.md.