# Operating notes

> Running more than one gateway? Read [multi-instance.md](design/README.md) first - it states,
> for every piece of state, whether it is shared and correct, per-instance by design, or
> single-instance by invariant.

## Choose an operating procedure

Use the relevant section for your task. The [deployment guide](how-to/deploy.md) describes the full v1 profile.

| Task | Start here |
|---|---|
| Size memory and caches | [Container memory](#set-a-memory-limit-or-the-heap-flags-mean-nothing), [local caches](#size-the-local-caches-against-the-heap) |
| Configure monitoring | [Availability alerts](#gateway-target-alerts), [memory alerts](#gateway-memory-alerts) |
| Recover counter trust | [Budget recovery](#recovering-budget-trust-after-counter-store-loss) |
| Tune local embedding | [Embedding CPU](#local-embedding-cpu) |
| Operate durable accounting | [Journal epochs and recovery](operations/journal-epochs.md) |
| Maintain learning state | [Learning history](operations/learning-history.md) |

## Preserve state across upgrades

Preserve the database, counter and journal volumes. Keep a verified backup and the key-hash pepper, rehearse migrations on a restored copy, and verify database names and issued-key compatibility before switching traffic.

Early development renamed `illg` database/user names and configuration prefixes to `pointsman`. Recreating disposable development volumes during that rename is **not an upgrade procedure for a populated deployment**. See [supported deployment and recovery scope](how-to/deploy.md#persistence-and-upgrades).

<a id="the-embedding-model-is-fetched-at-first-boot"></a>

## Embedding artifacts and startup dependencies

The shipped gateway image includes the MiniLM model and tokenizer. A fresh boot can still download DJL native libraries into the persistent `model_cache` volume.

| Artifact | Location | Startup behavior |
|---|---|---|
| MiniLM model and tokenizer | Bundled image resources; configurable model/tokenizer URI | Bundled resources need no model download; remote URI overrides can require a fetch |
| DJL native runtime | `~/.djl.ai` on `model_cache` | First boot downloads the native libraries; later boots reuse the volume |
| ONNX Runtime JNI library | `${java.io.tmpdir}/onnxruntime-java*` | Extracted again at boot |

For custom `file:` or `classpath:` model/tokenizer resources, configure `pointsman.cache.minilm.model-uri` and `tokenizer-uri`. A remote URI can use temporary container storage; the DJL volume does not automatically persist that download. See [startup network dependencies](how-to/deploy.md#capacity-and-observability) before restricting egress.

### Readiness and fallback

| Condition | Behavior | Observation |
|---|---|---|
| Real embedding readiness probe passes | Semantic embedder becomes available | `gateway.cache.embedder.semantic = 1` |
| Embedder is still warming | L2 lookups miss | No semantic readiness claim yet |
| MiniLM cannot load, fallback allowed | Degrade to lexical hashing | Boot warning and semantic gauge `0`; hashing is not an equivalent semantic model |
| `pointsman.cache.require-semantic=true` | Refuse startup when semantic embedding cannot be established | Prevents accepting that degradation |

## Set a memory limit, or the heap flags mean nothing

Set both the container limit and the intended heap policy. Without a container limit, JVM percentage-based ergonomics can size the heap against host memory rather than the deployment's budget.

| Profile | Gateway memory limit | Heap policy |
|---|---|---|
| Base evaluation Compose | 2 GiB | `MaxRAMPercentage=25` unless explicitly overridden |
| Qualified v1 overlay | 2.5 GiB | Maximum heap 640 MiB |

`docker-compose.minimal.yml` contains dependency services only. It does not define a gateway whose memory it could bound.

### What 2 GB buys, measured under a real cgroup

Ubuntu 24.04, cgroup v2, the shipped entrypoint unmodified, MiniLM embedder, 15,000 cache-miss
requests at 50 RPS:

| | 2g limit | 3g limit |
|---|---|---|
| max heap the JVM chose | 512 MB | 768 MB |
| NMT committed | 1139 MB | 1456 MB |
| peak RSS (`VmHWM`) | 1796 MB | 2217 MB |
| `memory.peak` | 2048 MB (the limit) | 2733 MB |
| OOM-killed | no | no |
| p99 / p99.9 | 45.18 / 55.90 ms | 45.28 / 55.26 ms |

The JVM picked 25 percent of the limit in this profile. Its sampled latency was similar
at 2g and 3g; that does not establish spare capacity or equal behavior under larger
prompts, more concurrency, or a different native-embedding workload.

### Read `memory.peak` carefully

At 2g the measured peak reached the limit. Cgroup memory includes anonymous memory,
file cache and kernel charges. File cache can often be reclaimed, but reclaim can cost
latency; reaching the limit without an OOM is evidence of completion, not a headroom
qualification. A separate 3g sample showed:

```
anon  1378 MB     heap, native, direct buffers, metaspace, thread stacks
file  1308 MB     page cache: the jar, the ONNX model, DJL native libraries
```

The 3g sample cannot establish 670 MB of available headroom in a 2g deployment.
Measure the assembled candidate at its actual limit, retaining cgroup pressure/reclaim,
RSS, heap and native-memory series. No minimum safe memory size is established by
subtracting this sample from a different limit; use the resource qualification and its
workload boundaries in the [readiness record](release-status.md).

### Direct buffers are pinned, and it is a guard rather than a saving

When `MaxDirectMemorySize` is unset the JVM sets the direct-buffer ceiling **equal to the max
heap**. Since the max heap is itself implicit here, that ceiling silently tracks the container
limit: move the limit from 2 GB to 4 GB and the ceiling goes from 512m to 1g with nothing written
down. The Dockerfile now pins it at `512m` so a limit change moves one number instead of two.

It does not free memory on Linux. Measured direct-buffer USE plateaus and does not track the heap:

| max heap | direct buffers in use |
|---|---|
| 256m | 68 MB |
| 512m | 169 MB |
| 1024m | 203 MB |

Adding a 256m cap at a 1024m heap moved total committed memory by 15 MB, which is noise. **512m
rather than 256m** because measured peak use reached 213 MB at only 50 RPS - 83 percent of a 256m
cap. The 512m limit is about 2.4 times that observed direct-buffer peak; this ratio
is not total-process headroom or a concurrency capacity estimate. Recheck it on the
intended workload before reducing the bound.

### Leave MaxRAMPercentage alone

The measured profile used `MaxRAMPercentage=25`. Keep heap and total process limits
explicit in the qualification record; change this setting only with measurements of
GC behavior and the native/direct-memory needs of the intended workload.

### A warning about numbers measured on macOS

Measure on the deployment platform. Native allocation, direct-buffer commitment and thread-stack accounting differ between macOS and Linux; a memory model fitted on a developer laptop does not establish Linux cgroup capacity.

The unset direct-buffer ceiling follows the heap on both platforms, but observed committed memory need not scale with that ceiling. Use the actual workload's RSS, cgroup pressure, heap and native-memory series.

## Size the local caches against the heap

`ExactMatchCache` now bounds L1 by estimated payload weight and an entry-equivalent count.
The defaults are a 64 MiB weight ceiling, 10,000 entries and a 4 KiB minimum entry weight.
The effective weight budget is therefore 40,960,000 bytes (about 39.1 MiB). All three are
configurable; see [operational limits](reference/operational-limits.md) for the formula and examples.
Weights include keys, response text and a configurable metadata estimate. They are not exact heap
measurements. Measure GC and retained heap on representative responses when setting this budget.

The separate in-memory **L2 response store**, `CaffeineResponseStore`, still uses the L1 max-size
as its entry count. Its whole-response objects and the in-memory vector index consume additional
heap outside L1's weight budget. The Redis L2 backend moves those shared stores off the JVM.
TTL bounds age; it does not replace capacity planning for either tier or the process as a whole.

## Gateway target alerts

The `gateway-availability` group in `docker/grafana/alert-rules.yml` monitors whether Prometheus can scrape the gateway. It does not depend on optional Jev enablement.

| Alert | Expression | Severity | Means |
|---|---|---|---|
| `GatewayTargetDown` | `up{job="gateway"} == 0` | critical | An expected gateway target is known to Prometheus and its scrape has failed, per instance. |
| `GatewayTargetAbsent` | `absent(up{job="gateway"})` | critical | The `gateway` job has no target at all: every expected target was removed from service discovery or the scrape configuration, or Prometheus lost them. |

Both wait `GATEWAY_TARGET_DOWN_FOR` before firing; the shipped default is **5 minutes**, so a
restart or a short network blip does not page. Both are critical because every other gateway
alert (journal, budget, invariant, Jev) reads series from this scrape: while either holds, those
alerts are blind, not quiet.

### Change the alert duration

 Prometheus rules cannot read configuration, so the value is rendered
into the rule file at deployment time, the same way the Jev values are. Edit
`GATEWAY_TARGET_DOWN_FOR` in `docker/grafana/alert-policy.env` (a Prometheus duration such as
`2m` or `10m`), run `scripts/render-alert-rules.sh` (it rewrites only the block between the
`gateway-availability-rules` markers), optionally `scripts/alert-rules-test.sh` (promtool checks
and unit tests; those tests assume the shipped 5m), then make Prometheus reload its rules: in the
shipped compose stack, `docker compose -f docker/docker-compose.yml restart prometheus` (or send it
`SIGHUP`). `JevAlertRulesRenderedTest` refuses a commit whose rendered block disagrees with the
policy file, and `scripts/render-alert-rules.sh --check` exits 1 in the same case.

### Alert coverage

What is and is not paged. Three situations are kept apart:

- *A failed expected target* (`up == 0`): `GatewayTargetDown`, one alert per instance.
- *A disappearing expected target*: when a target leaves discovery or Prometheus loses it, its
  `up` series goes stale rather than reading 0, so `up == 0` can never say so. `GatewayTargetAbsent`
  covers it, for the **whole job**: it fires when no gateway target is left. With several gateway
  targets, one of them vanishing is indistinguishable from an intentional scale-down or a replaced
  pod, so it is not paged; a deployment with a fixed fleet size should add its own rule on
  `count(up{job="gateway"})`.
- *Optional components that are not deployed*: the group reads `job="gateway"` and nothing else.
  The learner (`policy-service`), tracing (the OTel collector and Tempo) and Jev never raise an
  alert from this group, whether they are down, absent or never configured.

### Prometheus job naming

 This rule file is for a Prometheus whose `gateway` job is expected,
as in `docker/grafana/prometheus.yml`. On a Prometheus that is not configured to scrape a gateway,
or that names the job differently, `GatewayTargetAbsent` fires permanently and `GatewayTargetDown`
can never fire; rename the job in the rules (the templates under `docker/grafana/`) or do not load
this file there.

### Combine availability and Jev alerts

 On an instance that had Jev enabled in the last hour, a scrape failure
fires both `GatewayTargetDown` (critical, the page: the gateway is unreachable) and `JevTargetDown`
(warning, the context: the Jev panels for that instance are unobservable). This is deliberate - the
Jev rules are kept byte-identical and the availability rule must not depend on a feature flag. If
your Alertmanager should send only one notification, inhibit the warning by the critical one:

```yaml
inhibit_rules:
  - source_matchers: [alertname="GatewayTargetDown"]
    target_matchers: [alertname="JevTargetDown"]
    equal: [instance, job]
```

### Monitoring-system failure

 Rules evaluated by a Prometheus cannot report the loss of that
Prometheus, of its host, or of the path from it to whoever receives the notification: if the
monitoring host goes down, nothing in this file fires. Detecting that needs something outside it,
such as an always-firing watchdog alert routed to an external dead-man's-switch service, or a
second Prometheus that scrapes the first. The shipped compose stack also has no Alertmanager:
alerts are visible on Prometheus's own alerts page until a deployment routes them somewhere.

## Gateway memory alerts

The `gateway-memory` group in `docker/grafana/alert-rules.yml` watches the gateway container's memory **as the kernel
accounts it**, not the JVM heap. Its source is the gateway's own cgroup v2 files, read by `CgroupMemoryMetrics` at most
once per `pointsman.observability.cgroup-memory.min-interval-ms` (5 s):

| Metric | Meaning |
|---|---|
| `gateway_cgroup_memory_working_set_bytes` | `memory.current` minus `inactive_file`: the working-set ESTIMATE (as cAdvisor and the kubelet report it). It still includes active file cache and reclaimable slab. NaN when unreadable. |
| `gateway_cgroup_memory_limit_bytes` | `memory.max`. NaN when the container has no limit, or the file is unreadable. |
| `gateway_cgroup_memory_limit_set` | 1 when a numeric limit is set. |
| `gateway_cgroup_memory_readable` | 1 when the files were read on the last sample. |

| Alert | Fires when | Severity |
|---|---|---|
| `GatewayMemoryPressure` | working set / limit > `GATEWAY_MEMORY_RATIO` (0.80) for `GATEWAY_MEMORY_FOR` (5m), with a limit set | warning |
| `GatewayMemoryUnobservable` | the files are unreadable, no limit is set, or the scrape succeeds without the series, for `GATEWAY_MEMORY_UNOBSERVABLE_FOR` (10m) | warning |

### Interpret memory measurements

What the number is, and is not. `memory.current - inactive_file` = anonymous + shared + active file cache + all
kernel memory (reclaimable and unreclaimable slab, kernel stacks, page tables, sockets). It is therefore AT OR ABOVE the v1
resource policy's footprint (anonymous + shared + unreclaimable kernel memory, 80 % of the limit; `docs/how-to/deploy.md`),
so `GatewayMemoryPressure` warns early, never late, and can fire while the kernel could still reclaim part of what it
counts - this gateway has grown the dentry cache (reclaimable slab) before. It is an early operational warning, not a
measure of the policy footprint and not proof of an imminent OOM: on a page, read `memory.stat` (`anon`, `shmem`,
`slab_unreclaimable`, `slab_reclaimable`, `active_file`) to see what is actually held. The 0.80 reuses the policy's
percentage; the frozen qualification resource policy keeps its own formula and is not replaced by this alert. Raw memory
usage alerts constantly (the kernel keeps the cgroup near its limit with page cache); the heap is a quarter of the limit
and misses direct buffers, native and kernel memory.

### Memory-alert coverage

What this covers, and what it does not. The metric is the gateway process's own cgroup: on a cgroup v1 host, a
gateway outside a container, or a container without a memory limit, it is unobservable and says so
(`GatewayMemoryUnobservable`), rather than reporting zero. It does not watch the other containers (PostgreSQL, the two
Redis stores); a deployment that needs them alerted adds a container-metrics collector (cAdvisor, a Kubernetes
kubelet), and must alert on that collector's own absence the same way. Disabling the reader
(`POINTSMAN_OBSERVABILITY_CGROUP_MEMORY_ENABLED=false`) removes the series, which `GatewayMemoryUnobservable` reports.

## What to alert on when the counter store fails open

Rate limits may fail open when Redis is unreachable. With the default
`pointsman.budget.failure-mode=deny`, requests with an enforcing spend cap or unknown
cap applicability refuse `503 budget_unverifiable` without querying the ledger.
Uncapped requests can continue. Only the explicit `allow` mode can use historical
ledger estimates, without atomic reservations. Counter-trust loss is a separate fence:
`budget_recovering` refuses capped work in either mode.

Use these signals together:

- `pointsman_limits_engine_errors_total{stage="attempt"}` increments once per request that
  reached reservation and received a store error. It is not a count of successful
  fail-open admissions: capped requests can instead be refused. Correlate it with
  budget refusals and client outcomes.
- The `counterStore` health indicator reports DEGRADED with `consequence: rate limits fail
  open`, and logs the transition once per state change. It probes only when `/actuator/health`
  is read.
- There is NO gauge that says "failing open right now". The budget side's
  `pointsman_budget_failclosed_active` is, despite its name, a counter of outage admission decisions, including uncapped
  requests and denied capped requests. It does not mean the ledger was queried and is
  neither a current-state gauge nor a count of successful admissions. `pointsman.budget.unverifiable_refusals` is the refusal half on its own. The clean signal would be a 0/1
  gauge on `RedisBreaker`'s open state, which is what actually decides fast-failing; it is
  not implemented. Until then, alert on
  the rate of the errors counter above.
- The Lettuce connection pools (commons-pool2, `pointsman.redis.pool.*`) register no metrics;
  pool exhaustion shows up only as `RedisConnectionFailureException` in the same errors counter.
  Micrometer's `CommonsObjectPool2Metrics` binder would expose them once the pools are JMX-enabled.

## Run the counter Redis with AOF persistence

The counter store holds rate and budget counters. Use AOF persistence with a declared
fsync/durability policy, `noeviction`, and the ACL restrictions verified by enforced
store checks. Persistence reduces data-loss exposure; it does not reconstruct missing
reservations or establish consistency with a separately restored database and journals.

A Redis restart changes its run ID. The trusted-epoch check refuses budgeted traffic
with `budget_recovering` until recovery or operator attestation supplies a valid balance,
even if the restart appears clean. An in-process rollback is why ACL and eviction-policy
checks matter as well. Do not delete the fence or assume a PostgreSQL ledger sum alone
can repopulate every counter safely.

During an unreachable-store outage, the default `deny` policy refuses capped traffic
without reading the ledger; `allow` is an explicit availability tradeoff with possible
overspend. Neither setting silently authorizes a lost-trust epoch. See the manual
procedure below and [gateway controls](gateway-controls.md#when-redis-is-down).

## Running against Redis Cluster

### Cache and counter topology

Set the endpoint mode explicitly. Counter and cache endpoints have different cluster support; Redis Cluster remains outside the qualified v1 profile.

| endpoint | serves | cluster? |
|---|---|---|
| `pointsman.redis.counter` | rate limits, budgets, cooldowns, outbound provider tallies | **yes** |
| `pointsman.redis.cache` | identity L2, the invalidation and registry pub/sub channels | **yes** |
| `pointsman.redis.cache` with `vector-backend=redis` | the L2 semantic cache | **no - refuses at boot** |

Declare the mode explicitly per endpoint; it is never guessed from the shape of the host list,
because a typo that silently produced standalone behaviour against a cluster would work perfectly
until a key moved to another shard.

```yaml
pointsman:
  redis:
    counter:
      mode: cluster
      nodes: "redis-1:6379,redis-2:6379,redis-3:6379"
    cache:
      mode: cluster
      nodes: "redis-1:6379,redis-2:6379,redis-3:6379"
```

Under the shipped compose stack the same settings arrive as environment variables, which is the
door `docker/.env.example` documents:

```
REDIS_COUNTER_MODE=cluster
REDIS_COUNTER_NODES=redis-1:6379,redis-2:6379,redis-3:6379
REDIS_CACHE_MODE=cluster
REDIS_CACHE_NODES=redis-1:6379,redis-2:6379,redis-3:6379
```

Note the compose stack itself runs `redis/redis-stack`, a single-node distribution that cannot run
cluster mode. Cluster means your own Redis 8; the stack cannot demonstrate it.

<a id="what-changed-and-why-the-refusal-existed"></a>

### Atomic reservations across hash slots

Rate limiting reserves every counter a request touches in ONE atomic script. That is not a
performance choice: the script checks every counter has room before it writes any of them, which
is what stops a request half-spending a budget it was about to be refused for. Under cluster mode
every key in one such call must live on one hash slot.

Counter keys use two kinds of hash tag:

| Counter ownership | Hash tag | Reservation |
|---|---|---|
| Tenant-local | Tenant's tag | Co-located atomic call |
| Shared across tenants | Fixed `platform` tag | Separate atomic call |

A request spans at most two slots, with one call per slot. Giving a shared ceiling a tenant tag would split it into independent tenant ceilings; giving every scope its own tag would scatter a request across slots.

**The second call costs one extra round trip and only where a deployment configures a
deployment-wide ceiling** - `global-rpm`, `global-tpm` and `global-budget-micros` are unset by
default, so most deployments pay nothing at all.

### What an operator sees on deploy

**Counter keys are renamed by this**, because the hash tag is part of the key. Two consequences,
both bounded:

- **Rate counters reset.** They are all TTL'd within a minute, so the worst case is one window in
  which rate limits are looser than configured. No action needed.
- **Cost counters do not rebuild themselves from `usage_events`.** A renamed cost counter is a counter the budget fence has not seen: a tag at generation 1
  starts it at zero, and in a later generation it fences (`503 budget_recovering`) until attested. Redis Cluster is
  outside the v1 profile and the reconcile applier refuses it; do not rely on this section for budget correctness.
  Reopen affected tags through the attestation procedure below, reading `GET /admin/budget/trust/{tag}/evidence`.

Budgeted traffic remains refused wherever the recovery fence requires attestation; a restart alone does not establish safe balances.

### The one thing that still refuses

`pointsman.redis.cache.mode=cluster` refuses when `pointsman.cache.vector-backend=redis`. The L2
semantic tier issues RediSearch queries through a client whose cluster interface does not expose
them, so it would dial one node and miss most of its data rather than failing. Either run the
cache endpoint on a single Redis, or set `vector-backend=memory` - the L1 exact-match tier is
unaffected and keeps working. The refusal names both ways out.


## Recovering budget trust after counter-store loss

This is the manual-attestation procedure for the documented default recovery scope.
It is not automatic spend reconstruction, and the small retained restore drill does
not establish a production-volume RTO/RPO. Rehearse on disposable restored stores
before using it in an incident.

### Recovery scope

- **RPO.** The ledger's recovery point is the last supported **stopped-write PostgreSQL backup**. No point-in-time
  recovery is provided or tested. Spend acknowledged after that backup exists in the gateway only where a surviving
  store still holds it - a retained counter Redis, the journal (held records, pending records, quarantine) - and
  otherwise only in provider usage and billing records.
- **Recovery is manual and evidence-based.** The gateway refuses budgeted traffic for every tag it cannot prove; an
  operator reopens each tag by attesting its balances. Where the backup gap cannot be reconstructed, attest
  **conservative exhaustion** (the counter's cap): the period's remaining allowance is zero until it rolls over.
- **The gateway checks a floor, not completeness.** An attestation below what the gateway's own durable evidence
  proves is refused (409). Spend after the restored backup is invisible to it, so passing the floor never proves the
  backup gap was included.
- **Not promised:** automatic reconstruction, point-in-time recovery, or a hard cap matched to provider invoices.

### 1. Pause ingress and preserve evidence

Pause new ingress for the affected deployment and preserve incident evidence.
Drain what can finish; retain the database backup/WAL, counter-store persistence,
each replica's own durable journal, and provider request/usage records. Capture
the failure time and the last consistent backup boundary. Do not share one journal
between replicas or let one restore consume the only failure-point copy.

### 2. Restore stores and declare the restore

Restore the intended database and store configuration with the same key pepper.
Keep gateway replicas attached to their own restored journal volumes. A journal record
that cannot be parsed, such as a truncated copy, is moved aside intact as `<name>.unreadable`.
It is counted in `gateway_usage_journal_unreadable_total` and named in a warning, and the
rest of the journal still replays. Its usage was not delivered, so reconcile it from the
file before treating the restore as complete. Verify the
enforcing counter/cache store checks and retained generation/fence state before
reopening traffic. Do not delete trust rows or disable the fence to suppress 503s.
> **Declare every database restore before traffic resumes.** A retained counter store alone does not establish a complete balance.

#### Declare a recovery boundary

Start every gateway
instance with `POINTSMAN_BUDGET_RECOVERY_RESTORE_ID=<restore or backup id>` and confirm
the boundary in `GET /admin/budget/recovery`. **Every tag the restored database holds `TRUSTED` is fenced at boot**
(`FENCED`, `fenceReason` `declared_restore`): a retained counter store is not proof of a complete
balance - a settlement whose counter adjustment was still deferred, or lost, leaves the counter below the recorded
usage.

The restored database has no trust row for
a tenant whose first budgeted request came after the backup, and it cannot tell that
tenant from one that never spent; without the declaration such a tenant would start
again at zero and its post-backup spend would be granted twice. With it, a tenant first
seen without a trust row whose scope predates the boundary (or whose creation time cannot
be read, such as the shared `platform` tag) starts `RECOVERING`: refused 503
`budget_recovering` before any provider call until you attest it in step 4. Tenants
created after the boundary start normally.

If the boundary cannot be recorded the gateway
refuses to start; the same id at a later restart is a no-op. The gateway also records a
`COUNTER_STORE_LOSS` boundary on its own when a previously active tag's epoch is missing
from the counter store, but that safety net sees nothing in a deployment with no active
epoch, so the declaration is the supported step. The boundary records WHICH scopes exist at
that moment (not a timestamp), so clock skew between hosts cannot make an old tenant look
new.

**Use a new id for every restore**: the same id at a later restart is the same incident
and a no-op, while a reused id for a new restore keeps the old membership and would not list
tenants created since.

A declared-restore boundary cannot be closed (409): its tenants not
yet seen are protected only at their first request, so each is resolved by attestation when
it appears. A boundary recorded for a counter-store loss on a database that was NOT restored
can be closed with `POST /admin/budget/recovery/{id}/close {"reason": ...}`; closing stops it
holding tags first seen later and never reopens a tag it already holds.

### 3. Review each tag’s evidence

Read the trust inventory with a root-admin identity. `FENCED` or `RECOVERING` tags
require investigation (`RECOVERING` tags held by a recovery boundary appear here on their
first request after the restore, not before). For each, read the **evidence report**,
`GET /admin/budget/trust/{tag}/evidence`: per cost counter of the current generation, the generation's starting
balance (`init`, or `UNKNOWN`), settled charges, **held** journal records (settlements whose rows the restore lost),
outstanding and uncertain reservations, deferred adjustments, the live counter value (information only), the
`lowerBound`/`upperBound`, a status (`EXACT`, `BOUNDED`, `UNKNOWN`) and the `attestationFloor`.

Under an open
recovery boundary `backupGap` says spend after the backup is not observable. Then list the held records,
`GET /admin/budget/held`. A database ledger total alone cannot supply the balances:
add post-backup acknowledged spend from outside the gateway (provider usage), outstanding reservations and
unresolved provider outcomes. A counter-minus-ledger difference does not identify which requests are missing. If
the balances cannot be justified, leave the tag fenced or attest the cap.

### 4. Prepare the attestation

Prepare a reviewed attestation file containing `expected_generation`, `evidence_ref` and `balances`.
Copy the generation from the trust inventory whose incident evidence you reviewed.
A 409 generation mismatch requires a new review, not replacing the number and
resubmitting the old balances. Missing/invalid generation is a 400.
Values are integer **micro-dollars already charged/reserved to each cost counter**,
not remaining budget.

Keys must be the exact counter keys belonging to the selected
tag. Do not copy a backup-time snapshot without accounting for the backup gap; the
retained drill explicitly measured spend that such a snapshot failed to recover.
An empty map is not evidence that spend was zero. Preserve the review, inputs and
checksums with the incident record.

A balance below the counter's `attestationFloor` is refused 409 and writes
nothing: raise it to at least the floor (the floor is init + settled + held + outstanding, or without init when
that is `UNKNOWN`). A counter you omit from the map stays refused until it is attested or its period rolls over.

### 5. Submit, verify activation and archive included evidence

Submit the attestation once, retain its returned generation/recovery ID, and re-read
state if the response is uncertain before deciding whether to retry. The database
transaction and Redis epoch installation are separate steps. A successful response
is not a complete recovery test; confirm activation and all store checks on each replica.

#### Archive included held records

The attestation records which held records its balances included. Once a held record is included for every
counter it names it is `archivable`: archive it with `POST /admin/budget/held/{event_id}/archive
{"evidence_ref": ...}`. The record moves intact into the journal's `a/` directory under its sha256
(`<event>.<sha256>.held.json`), stops being retried, and frees its journal slot; a repeat is a no-op; it is never
deleted. `load-test/journal-records.py <journal> count --selector archived` lists it.

#### Keep the archive available for later restores

Archived records are never replayed. The gateway reads the archive without modifying it: if the database is later restored to before the attestation that included a record, that
inclusion is gone, and the archived record counts again - once - in its counters' evidence floor, appears in
`GET /admin/budget/held` as `archived: true`, and fires BudgetEvidenceHeld until an attestation includes it again
(attest as above; there is nothing to re-archive).

Retention and repair requirements:

- **Keep `a/` on the journal volume** for as long as any restorable database backup predates the attestation that
  included its records. Copy it to the incident record, but do not remove it from the volume before then.
- If it was removed, **copy it back before attesting** after a restore. Without it the gateway cannot see that
  spend: it is then part of the backup gap - supply it from the incident record or provider usage, or attest the
  cap. The gateway cannot tell that an archive is missing.
- A corrupt or conflicting archive file (bytes not matching the sha256 in its name, an unreadable record, one event
  archived twice, or an event both held and archived) is listed in the evidence report's `archiveProblems` and
  REFUSES every attestation (409) until reviewed. Review it against the incident record; move only a file you have
  established to be a bad copy out of `a/`, and supply the spend it described yourself.

### 6. Verify before reopening

Admit a small identified verification cohort, including an exhausted-budget key.
Check that the exhausted key is refused without provider dispatch, admitted work
reconciles by request/attempt identity, and journal/backlog counters drain without
duplicates. Compare counters to the reviewed balances plus this cohort's charges.
Reopen gradually only after those checks. Keep unresolved work visible; manual
attestation does not create recovery inclusions for old owed operations.

### Correct an existing trust attestation

Correcting a trusted tag without a restore. When an operator knows a TRUSTED tag's counter is wrong (a lost
correction, say), fence it: `POST /admin/budget/trust/{tag}/fence {"expected_generation", "evidence_ref", "reason"}`.
PostgreSQL is marked first, then that generation's Redis epoch is deactivated so every replica refuses at once, then
the local cache is dropped. Requests already admitted settle normally. Reopen by attestation as above.

### Unresolved provider attempts

Requests in flight when a gateway process died (BudgetReservationUnresolved). A gateway killed with requests in
flight (SIGKILL, OOM, host loss) and restarted on its own journal writes, at boot, one request row per in-flight
request - status 503, `error_type` `accounting_interrupted`, usage `UNKNOWN`, zero cost - and one `UNKNOWN` provider
dispatch row per send that had begun. Their budget reservations stay `ADMITTED`, held on the counter at what was
reserved. **Nothing is refunded automatically**: the provider may have received and billed those requests, and a
process dying is not evidence that it did not. With the reconcile applier on, the reservation resolver marks each such
reservation `reserve_unresolved:interrupted_dispatched` (a dispatch row exists) or
`reserve_unresolved:interrupted_no_dispatch_record` (none has landed - missing evidence, not proof that nothing was
sent) once it is older than `reserve-resolve-delay-ms`, without changing its state, the counter or the tag's trust. They
count in `pointsman_budget_reservation_unresolved`, and BudgetReservationUnresolved fires after 15 minutes. The tag is
not fenced for this and keeps serving with the holds in place; a period rollover does not clear the alert.

1. **List them.** `GET /admin/budget/trust/{tag}/evidence` returns `unresolvedReservations`: the total and the oldest
   100, each with `requestId`, `counterKey`, `generation`, `reservedMicros`, `reason`, `createdAt` and the request's
   `dispatches` / `dispatchOutcomes`. The same holds are already inside each counter's `outstanding`, so they are in the
   `attestationFloor`. The gauge is per instance and counts every tag; the report is per tag.
2. **Establish what the provider did**, per request id, from provider request and usage records (the dispatch rows name
   provider, model, credential and send time). **`reservedMicros` is the amount held, not a maximum of the bill**: a
   provider can bill more than the estimate that was reserved.
3. **Decide.** If provider usage accounts for every listed request, fence the tag
   (`POST /admin/budget/trust/{tag}/fence` with the generation from the report), re-read the evidence and attest each
   counter at the floor plus whatever the provider billed above the reserves. A request billed less than its reserve,
   or not at all, cannot be given back by attestation: the floor includes the full reserve and a balance below it is
   refused 409. If the liability cannot be established, keep the tag fenced, or attest the cap (conservative
   exhaustion: no allowance until the period rolls over).
4. **Confirm.** The attestation installs generation + 1; the reservations of the older generation stop counting and the
   alert resolves; the rows themselves stay as they were, as the record. Repeating an attestation with the old
   generation is 409 and writes nothing. An interrupted request cannot settle later beside its 503 row (the delivery is
   refused as a conflict); a slow request of the old generation that does settle after the attestation has its
   adjustment PRESERVED (never applied under the attested balance, never voided) and fences the tag again, as for any
   older-generation settlement. Never edit the rows or the counter store by hand.

**Alerts** (`docker/grafana/alert-rules.yml`, group `budget-recovery`): BudgetTagFenced (a tag refuses until
attested), BudgetReconcilePreserved, BudgetReservationUnresolved (15 minutes), BudgetStoreChecksFailing,
BudgetEvidenceHeld (held settlement evidence exists: list, attest, archive) and RetentionRetainingUnresolved. Each
fires on a gauge that returns to zero only when the condition is resolved.

The current administrative endpoints are:

```sh
# ADMIN_CURL_CONFIG: private mode-600 curl config containing the admin credentials.
# ADMIN_URL: the protected administrative origin; TAG_URI: URL-encoded inventory tag.
curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  "$ADMIN_URL/admin/budget/trust" > trust-before.json

curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  -H 'Content-Type: application/json' --data-binary @attestation-reviewed.json \
  "$ADMIN_URL/admin/budget/trust/$TAG_URI/attest" > attestation-result.json

curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  "$ADMIN_URL/admin/budget/trust/$TAG_URI/evidence" > evidence-before-attest.json

curl --fail-with-body --silent --show-error --config "$ADMIN_CURL_CONFIG" \
  "$ADMIN_URL/admin/budget/held" > held-records.json
```

The endpoint requires a positive matching `expected_generation`, a nonempty evidence
reference, nonnegative balances, a known
fenced/recovering tag, and counters with that tag's prefix. Those validations cannot
prove the operator's balance reconstruction is complete or detect a database rollback
that restores both balances and generation. Its audit records the
attestation; do not mistake an accepted API payload for proof that no budget was regranted.
Record the measured restore duration and recoverable data window for the actual dataset.
See the [budget recovery design](design/rate-limiting.md) and
restore evidence.


## Quality evidence storage (retention off by default)

Quality-guarded routing is opt-in and off by default. When it is enabled, its evidence tables (`quality_events`,
`quality_aux_attempts`, `quality_audit_drops`) **grow without bound**, because their retention job is disabled by
default. Choose retention explicitly for your deployment. Disabled retention is not a disk cap.

- **Visibility.** `pointsman_quality_retention_rows{table}` reports each table's row count hourly (-1 = unreadable).
  Size the database volume for your evaluation traffic and alert on growth against your own threshold.
- **Operator action.** To bound growth, enable `POINTSMAN_QUALITY_RETENTION_ENABLED` with lifetimes you have chosen
  (`docs/reference/configuration.md`). Retention never deletes the events of a decision an audit job selected, nor an
  unresolved (RESERVED or UNKNOWN) auxiliary attempt; epochs, policy versions, validators and audit jobs are never
  touched. Test any lifetime on a restored copy first.
- **Budget recovery evidence is separate.** The journal's archive of held settlement evidence (`a/`) is kept on the
  journal volume while any restorable backup predates the attestation that included it (see "Recovering budget trust"
  above); size the journal volume for it.

## Local embedding CPU

The performance candidate uses three settings when creating the local ONNX session:

| Setting under `pointsman.cache.minilm` | Candidate default | Behavior |
|---|---|---|
| `intra-op-threads` | `0` | Use half the JVM's available processors (rounded down), bounded to 1–4, honoring container CPU limits. A positive number explicitly sizes the pool; negative values fail startup. |
| `allow-spinning` | `true` | Allow workers to spin during active inference. `false` selects fully blocking workers. |
| `force-spinning-stop` | `true` | Park workers when the last concurrent inference call returns. With spinning enabled, `false` restores unrestricted spinning. |

Environment names are `POINTSMAN_CACHE_MINILM_INTRA_OP_THREADS`,
`POINTSMAN_CACHE_MINILM_ALLOW_SPINNING` and
`POINTSMAN_CACHE_MINILM_FORCE_SPINNING_STOP`. Changes require a gateway restart.
They apply to `auto` and forced `minilm`, not hosted or hashing embedders. The
previously qualified runtime shipped `allow-spinning=false` and did not expose
explicit worker sizing or stop-on-return. Its acceptance does not cover these new defaults.

Explicit sizing avoids ONNX's automatic host-wide worker affinities targeting CPUs
outside a container cpuset. Spinning during active inference reduces wake-up cost;
stop-on-return prevents that policy from consuming idle CPU. CPU cost and latency
still depend on sequence length, concurrent requests and CPU limits. Compare both
under your workload; a short benchmark is not a capacity or readiness guarantee.

Spring AI 1.1.8 creates private SessionOptions without a public customization hook.
`SessionConfiguredEmbeddingModel` is an Apache-2.0 adaptation that sets worker
options before session creation. Model/tokenizer resources, input IDs and masks,
execution mode, mean pooling, normalization and vector namespace are unchanged.
The model precision and context window are unchanged.

The real-model regression compares single and mixed-length batches, full-window,
Unicode and empty inputs against the pinned upstream implementation at the existing
`1e-6` component tolerance. It also exercises overlapping calls and repeated worker
parking. Cache similarity, lexical verification, tenant isolation and replay checks
are unchanged. See [candidate measurements and limits](performance.md#performance-candidate).

No reflection or dependency upgrade is used. Keep the adaptation aligned when
upgrading Spring AI; remove it when equivalent public session customization is
available. It does not resolve upstream resource-lifecycle limitations.

ONNX defines stop-on-return in its [1.20 session option source](https://github.com/microsoft/onnxruntime/blob/v1.20.0/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h).

Upstream source: [Spring AI 1.1.8 source archive](https://repo.maven.apache.org/maven2/org/springframework/ai/spring-ai-transformers/1.1.8/spring-ai-transformers-1.1.8-sources.jar).
Original `TransformersEmbeddingModel.java` SHA256:
`e18f8e56cc29ed3451554e8101b38cdabf01b0a0fd6f98df9b29f8b9ef00b74e`.
ONNX option semantics: [ONNX Runtime 1.20.0](https://github.com/microsoft/onnxruntime/blob/v1.20.0/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h).

### Bound MiniLM eligibility work

Before MiniLM counts tokens to check its actual embedding window,
`pointsman.cache.minilm.max-check-characters` limits the input to 16,384 UTF-16
code units and `pointsman.cache.minilm.max-concurrent-window-checks` allows two
checks at once per embedder. Both are configurable positive integers; invalid
values fail startup for auto/MiniLM selection. These are resource limits for
optional cache work, separate from the model's derived token window. Raising
them increases possible native allocation and must be qualified under load.

An oversized input or busy check bypasses semantic caching immediately. It does
not reject the provider request or change provider context accounting; exact L1
reuse remains available. Inputs within these limits still require the full,
untruncated native token count to fit the model window. This can conservatively
skip a whitespace-heavy prompt that would fit in tokens, and contention can
reduce the semantic hit rate. The existing `gateway.cache.l2_skipped{reason="window"}`
metric and `SKIPPED_WINDOW` outcome include this inability to establish window
eligibility; they do not mean every skipped input has too many tokens.

The limits bound an identified optional-work path; the limits alone are not proof
of resource stability. The documented v1 profile subsequently passed its frozen
12-hour and 24-hour resource qualification; see the
[final record](release-status.md) for measured bounds
and retained diagnostic failures. Changes to these limits or the workload require
qualification of the affected resource behavior.

### Tokenizer admission under short bursts

Provider tokenizer mode keeps `pointsman.tokens.max-concurrent-tokenizations=2`
active requests per instance. Up to `pointsman.tokens.max-waiting-tokenizations=16`
additional requests may wait at most `pointsman.tokens.admission-wait-ms=25` ms
for an active slot. These defaults absorb brief overlap without raising tokenizer
CPU concurrency. Both new limits are nonnegative; either zero restores immediate
refusal. A full queue, expired wait or interruption returns retryable 503 with
`error.code=tokenizer_busy`. Interrupted threads retain their interrupt status.
There is no fallback to heuristic counting. The wait budget excludes execution
and scheduling delay and is additional request latency, not a response deadline.

These settings are included in the final v1 profile
[qualified on 28 October](release-status.md).
They do not guarantee admission under sustained overload. Keep the zero-unexpected-
failure qualification gate for the declared supported workload; do not waive it
merely because overload refusal is intentional.

## Optional Jev task routing: data export and operator-paid overhead

Off by default, and it stays off unless you set `POINTSMAN_JEV_ENABLED=true` *and* name the
tenants, the credential and a task-to-model ranking - the gateway refuses to start with any of
the three missing. Once on, every request from a tenant in `POINTSMAN_JEV_ALLOWED_TENANTS` that
reaches a `strategy: jev` group without a valid `X-Task-Type` hint has its accepted
conversation text - system and developer messages included - sent to the configured TypeSafe
endpoint. Obtain those tenants' permission first; the export is the feature, not a side effect.
Requests that declare tools, carry images or tool results, or exceed the input bounds are
never exported (reason `tools_declared`, `unsupported_context`, `input_limit`).

Classification calls are operator overhead. They never appear in any tenant's usage ledger or
budget, and a call whose usage the provider did not report is counted under
`pointsman_jev_usage_unverified_total`, never as free. The attempt, rate and concurrency
bounds are per process: they reset on restart and multiply with replicas, so allocate the
account's limits across replicas yourself.

Watch `pointsman_jev_invocations_total{outcome=...}` (one count per classifier entry:
classified, abstained, skipped, refused, failed) beside `pointsman_jev_decisions_total`
(one per routing decision, and larger, because a request's assessment serves every tier), and
the `jev-classifier` alert rules (per instance; rendered from `docker/grafana/alert-policy.env`).
A successful classification or hint can select a later-ranked candidate; every skip, refusal,
dependency failure or abstention falls back to the first eligible candidate, so none of those is
a request failure - the alerts say the feature is not doing its job, which is a different page
from "requests are failing". Setup, bounds, the metric vocabulary
and the evaluation to run before trusting a ranking: [docs/how-to/jev-routing.md](how-to/jev-routing.md).