# The semantic cache

Pointsman can reuse an exact request or an eligible semantically similar request. This guide explains what permits a hit, what rejects one and which controls an operator can inspect. See the [cache architecture](design/cache.md) for the request flow and [cache tuning](how-to/tune-the-semantic-cache.md) for configuration steps.

## Two tiers, two questions

- **L1** is exact: an in-process cache answering "has *this tenant* sent the same
  content and answer contract before?" Its configured entry-equivalent and payload-weight
  bounds limit retention; the default TTL is 15 minutes. Estimated payload weight is not
  a measurement of JVM heap. See [configuration](reference/configuration.md). It is
  consulted first, and its hits read `X-Cache: HIT-L1`.
- **L2** is semantic: a Redis vector index answering "has this tenant asked *this question in
  other words*?" The prompt is embedded (MiniLM, locally, 384 dimensions), the index searched,
  and a stored entry whose cosine similarity clears the threshold (`0.92` by default,
  disclosed on every response as `X-Cache-Threshold`) is a candidate hit: `HIT-L2`.

L2 hits avoid provider tokens, but approximate reuse can return an unsuitable answer. An approximate index can also omit an existing exact vector; a shared store does not guarantee recall.

A successful L2 replay can warm the serving replica's L1 only when its stored exact-cache
fingerprint matches the current request. The controller first checks the current caller's
allowed model/output cap and applies output guardrails; it then stores the guarded response
in L1. Paraphrases and older entries without that fingerprint are not promoted. The first
response still reports `HIT-L2`; subsequent identical requests can report `HIT-L1`, under
L1's normal TTL, weight and invalidation policy. This avoids repeated approximate searches
for already-validated exact requests; it does not guarantee a hit on a cold replica.

## The guards against serving the wrong answer

A semantic candidate must pass each applicable check before reuse.

| Check | Requirement | Tradeoff or observation |
|---|---|---|
| Similarity and lexical verification | Clear the similarity threshold and word-overlap floor (`verification-min-overlap`, default `0.4`) | Sampled hits and scores are recorded in `cache_hit_samples` for evaluation |
| Tenant scope | Exact keys and semantic namespaces/query filters bind tenant identity | Reuse cannot cross the admitted tenant scope |
| Redacted content | Derive keys after input guardrail redaction | Requests with different effective content do not share the same exact fingerprint |
| Conversation history | Default context mode includes the prior conversation exactly | Lower multi-turn hit rates protect against reusing another conversation's answer |
| Minimum text length | Prompts under 20 characters skip L2 | `skipped-short`; exact L1 can still serve |
| Sampling policy | Default semantic reuse requires explicit temperature zero | Nonzero or omitted temperature stays off L2 under `exact-only`; L1 can still reuse identical requests |
| Tools | Requests carrying tools skip semantic reuse | Exact L1 reuse remains subject to the request contract |
| Serving contract | Check model permissions, output cap, generation and response policy | A vector match alone is insufficient |

Alternative context modes loosen exact-history matching. Review each mode's tradeoff before using `X-Cache-Context-Mode`. `X-Cache-Temperature-Mode` changes the documented read/semantic policy; it is not a substitute for validating the workload.

## Streaming, and the failure posture

Cached responses can be replayed as SSE. The response must already be complete and pass the normal serving checks.

| Streaming control | Behavior |
|---|---|
| Default replay | One immediate content delta |
| Optional `paced` replay | Word-by-word replay with an 8 ms inter-chunk delay |
| `pointsman.cache.streaming.enabled=false` | Disable streaming cache lookup and write-through; assembly for ledger, counters and reward remains |

Cache failures normally become a miss or a skipped write. Redis commands default to a 50 ms timeout, and the L2 breaker skips work while open. That timeout is **per command**, not a bound on embedding plus the entire multi-command lookup. A cache miss still passes provider admission and budget checks.

Eligible concurrent identical misses can share one provider computation. A follower that exceeds its wait bound proceeds independently; replay still requires the follower's own serving checks.

## Operating it

Use request directives, invalidation and capacity controls for different purposes.

| Control | Scope | Behavior |
|---|---|---|
| `X-Cache-Control: no-cache` | One request | Skip reads on both tiers; allow eligible writes |
| `X-Cache-Control: no-store` | One request | Skip reads, writes and single-flight |
| `DELETE /internal/cache?tenant=...` | One tenant | Invalidate literal tenant ID; glob characters are refused |
| `pointsman.cache.version` | Global key namespace | Changing the salt makes prior keys unreachable |
| TTL | Retention | L2 defaults to 24 hours, extends on hits, and has a 168-hour hard cap; volatile models may use shorter TTLs |

For tenant invalidation, the receiving instance flushes shared L2 and publishes on `illg:cache:invalidate`; subscribers drop local L1. This describes invalidation behavior and does not expand the [qualified deployment scope](release-status.md).

### Bound L2 growth

Unique prompts can grow the index throughout their retention period. Configure both document admission and Redis memory limits.

1. Set `POINTSMAN_CACHE_L2_MAX_DOCUMENTS` (default 200,000). New writes stop at the observed threshold and resume as entries expire or are evicted.
2. Set `POINTSMAN_CACHE_L2_SIZE_CHECK_MS`. Queued writes recheck admission; stale checks close writes.
3. Give **cache-only Redis** a `maxmemory` bound and an `allkeys-lru` or `allkeys-lfu` policy.
4. Use `POINTSMAN_CACHE_L2_STORE_CHECKS=enforce` to stop writes when required store protections are missing. Default `report` exposes the condition without enforcing it.

The document threshold is soft: already-admitted writes, including writes from other replicas, can still land. The memory limit is the hard bound. A skipped cache write is observable and does not refuse a completed client response.

> **Counter Redis must use `noeviction`.** Do not apply the cache eviction policy to a store that holds budget counters.

Inspect `gateway.cache.store.skipped`, `gateway.cache.l2.store_checks_failing` and the `l2WriteAdmission` health detail. Hit rates, hit/miss latency and estimated savings are available at `GET /admin/observe/analytics/cache`; response verdicts use the [cache headers](reference/response-headers.md).

### Hosting your own embedding model

MiniLM is the default because it runs in-process with no key and no network. An operator with a
larger embedding model - self-hosted, or a provider's - can put it behind the cache instead, as
long as it speaks the OpenAI-compatible `/v1/embeddings` shape:

```
POINTSMAN_CACHE_EMBEDDER=hosted
POINTSMAN_CACHE_HOSTED_BASE_URL=https://embeddings.internal      # required
POINTSMAN_CACHE_HOSTED_MODEL=my-embedding-model                  # required
POINTSMAN_CACHE_HOSTED_MAX_INPUT_TOKENS=8191                     # required, see below
POINTSMAN_CACHE_HOSTED_API_KEY_ENV=EMBEDDINGS_API_KEY            # optional: the VARIABLE's name
```

The vector dimension is not configured: the boot probe embeds one string and reads the width
off the answer. If the probe fails, the gateway still starts, the semantic tier misses (the
`gateway.cache.embedder.semantic` gauge reads 0) and the embedder re-probes every 30 seconds
until the endpoint answers.

**`max-input-tokens` is mandatory, and the gateway refuses to start without it.** No embeddings
API discloses its context window, so there is no value the gateway could safely assume. What you
declare here becomes the cache's skip threshold: a prompt estimated to be longer than the window
is not embedded and not cached, and the response says so (`X-Cache-Semantic: skipped-window`).
This is
the setting to raise when a large model should be caching long prompts that MiniLM never could.

### Hosted-window estimation

 The guard is `text.length() / 4 <= maxInputTokens` - four
characters per token, the same divisor the rest of the gateway uses for character-based token
estimates. Dense code or non-Latin text tokenises heavier than that, so a prompt the guard admits
can still exceed the provider's real limit by a margin. Declare the real window, not a rounder
number above it.

The declared window must match the embedding service's real capability.

| Declaration | Consequence |
|---|---|
| Smaller than the real window | Conservative, visible skips for longer prompts |
| Larger than the real window | Provider rejection or silent truncation; a fragment vector can be stored under a full-request fingerprint |

The gateway fingerprint cannot detect a provider silently truncating an input without changing its declared identity. Verify the service's behavior with boundary-length inputs.

### Changing the embedding window

 The declared window is part of the embedder fingerprint
(`text:hosted-<model>:<dim>:w<window>`), alongside the model name and the probed dimension. Change
any of the three and the semantic tier starts cold: old entries are unreachable under the new
scope and age out by their TTL. That is deliberate - vectors computed under one window are not
comparable with vectors computed under another - and it means a window change costs one cache
warm-up, not one silent mismatch.

## Supported embedding scope

The tiers and key namespaces are modality-agnostic - the design carries seams for image,
audio, and document embedders - but only the text embedder ships. If the MiniLM model
cannot load, the embedder falls back to lexical hashing, loudly; deployments where a false
positive is worse than an outage can set `POINTSMAN_CACHE_REQUIRE_SEMANTIC=true` and refuse
to start instead.


### Lexical verification safety boundaries

The default lexical verifier requires stored source text. Legacy entries without it
are misses and are replaced by normal write-through; switching verification on no
longer silently accepts unverifiable old answers. Word matching uses Unicode letters
and numbers, so different non-Latin prompts do not compare as two empty ASCII sets.

Digit-bearing, whitespace-delimited tokens must match literally in order, including
signs, punctuation, identifiers and repetitions. Changing `100` to `1000`, `AB123`
to `AB124`, or `1/2` to `2/1` cannot pass just because most words match. This is a
conservative filter: a valid paraphrase that changes punctuation around a number
can miss. `verification=none` explicitly disables these checks along with lexical
overlap. No new numerical threshold is introduced.

This remains a heuristic, not proof that two questions have interchangeable answers.
It does not establish equivalence of spelled-out numbers, negation, units or other
facts. For workloads requiring literal request matching, disable semantic text reuse
with `pointsman.cache.modalities.text.enabled: false`; the L1 exact cache remains
available. Use `X-Cache-Control: no-store` to bypass both tiers for a request. The
shipped configuration enables semantic text reuse for eligible requests, including
`temperature: 0`. Despite its name, `temperature-mode: exact-only` skips semantic
reuse only for nonzero or omitted temperature; it is not a global L2-off switch.
Evaluate semantic reuse against a labeled workload before relying on it.
The hashing embedder can map distinct texts to identical vectors; unique request text
alone is not a guarantee of a cache miss in a semantic-cache benchmark.