The semantic cache

Pointsman can reuse an exact request or an eligible semantically similar request. This guide explains what permits a hit, what rejects one and which controls an operator can inspect. See the cache architecture for the request flow and cache tuning for configuration steps.

Two tiers, two questions#

• L1 is exact: an in-process cache answering "has this tenant sent the same content and answer contract before?" Its configured entry-equivalent and payload-weight bounds limit retention; the default TTL is 15 minutes. Estimated payload weight is not a measurement of JVM heap. See configuration. It is consulted first, and its hits read X-Cache: HIT-L1.
• L2 is semantic: a Redis vector index answering "has this tenant asked this question in other words?" The prompt is embedded (MiniLM, locally, 384 dimensions), the index searched, and a stored entry whose cosine similarity clears the threshold (0.92 by default, disclosed on every response as X-Cache-Threshold) is a candidate hit: HIT-L2.

L2 hits avoid provider tokens, but approximate reuse can return an unsuitable answer. An approximate index can also omit an existing exact vector; a shared store does not guarantee recall.

A successful L2 replay can warm the serving replica's L1 only when its stored exact-cache fingerprint matches the current request. The controller first checks the current caller's allowed model/output cap and applies output guardrails; it then stores the guarded response in L1. Paraphrases and older entries without that fingerprint are not promoted. The first response still reports HIT-L2; subsequent identical requests can report HIT-L1, under L1's normal TTL, weight and invalidation policy. This avoids repeated approximate searches for already-validated exact requests; it does not guarantee a hit on a cold replica.

The guards against serving the wrong answer#

A semantic candidate must pass each applicable check before reuse.

Alternative context modes loosen exact-history matching. Review each mode's tradeoff before using X-Cache-Context-Mode. X-Cache-Temperature-Mode changes the documented read/semantic policy; it is not a substitute for validating the workload.

Streaming, and the failure posture#

Cached responses can be replayed as SSE. The response must already be complete and pass the normal serving checks.

Cache failures normally become a miss or a skipped write. Redis commands default to a 50 ms timeout, and the L2 breaker skips work while open. That timeout is per command, not a bound on embedding plus the entire multi-command lookup. A cache miss still passes provider admission and budget checks.

Eligible concurrent identical misses can share one provider computation. A follower that exceeds its wait bound proceeds independently; replay still requires the follower's own serving checks.

Operating it#

Use request directives, invalidation and capacity controls for different purposes.

For tenant invalidation, the receiving instance flushes shared L2 and publishes on illg:cache:invalidate; subscribers drop local L1. This describes invalidation behavior and does not expand the qualified deployment scope.

Bound L2 growth#

Unique prompts can grow the index throughout their retention period. Configure both document admission and Redis memory limits.

1. Set POINTSMAN_CACHE_L2_MAX_DOCUMENTS (default 200,000). New writes stop at the observed threshold and resume as entries expire or are evicted.
2. Set POINTSMAN_CACHE_L2_SIZE_CHECK_MS. Queued writes recheck admission; stale checks close writes.
3. Give cache-only Redis a maxmemory bound and an allkeys-lru or allkeys-lfu policy.
4. Use POINTSMAN_CACHE_L2_STORE_CHECKS=enforce to stop writes when required store protections are missing. Default report exposes the condition without enforcing it.

The document threshold is soft: already-admitted writes, including writes from other replicas, can still land. The memory limit is the hard bound. A skipped cache write is observable and does not refuse a completed client response.

Counter Redis must use noeviction. Do not apply the cache eviction policy to a store that holds budget counters.

Inspect gateway.cache.store.skipped, gateway.cache.l2.store_checks_failing and the l2WriteAdmission health detail. Hit rates, hit/miss latency and estimated savings are available at GET /admin/observe/analytics/cache; response verdicts use the cache headers.

Hosting your own embedding model#

MiniLM is the default because it runs in-process with no key and no network. An operator with a larger embedding model - self-hosted, or a provider's - can put it behind the cache instead, as long as it speaks the OpenAI-compatible /v1/embeddings shape:

POINTSMAN_CACHE_EMBEDDER=hosted
POINTSMAN_CACHE_HOSTED_BASE_URL=https://embeddings.internal      # required
POINTSMAN_CACHE_HOSTED_MODEL=my-embedding-model                  # required
POINTSMAN_CACHE_HOSTED_MAX_INPUT_TOKENS=8191                     # required, see below
POINTSMAN_CACHE_HOSTED_API_KEY_ENV=EMBEDDINGS_API_KEY            # optional: the VARIABLE's name

The vector dimension is not configured: the boot probe embeds one string and reads the width off the answer. If the probe fails, the gateway still starts, the semantic tier misses (the gateway.cache.embedder.semantic gauge reads 0) and the embedder re-probes every 30 seconds until the endpoint answers.

max-input-tokens is mandatory, and the gateway refuses to start without it. No embeddings API discloses its context window, so there is no value the gateway could safely assume. What you declare here becomes the cache's skip threshold: a prompt estimated to be longer than the window is not embedded and not cached, and the response says so (X-Cache-Semantic: skipped-window). This is the setting to raise when a large model should be caching long prompts that MiniLM never could.

Hosted-window estimation#

The guard is text.length() / 4 <= maxInputTokens - four characters per token, the same divisor the rest of the gateway uses for character-based token estimates. Dense code or non-Latin text tokenises heavier than that, so a prompt the guard admits can still exceed the provider's real limit by a margin. Declare the real window, not a rounder number above it.

The declared window must match the embedding service's real capability.

The gateway fingerprint cannot detect a provider silently truncating an input without changing its declared identity. Verify the service's behavior with boundary-length inputs.

Changing the embedding window#

The declared window is part of the embedder fingerprint (text:hosted-<model>:<dim>:w<window>), alongside the model name and the probed dimension. Change any of the three and the semantic tier starts cold: old entries are unreachable under the new scope and age out by their TTL. That is deliberate - vectors computed under one window are not comparable with vectors computed under another - and it means a window change costs one cache warm-up, not one silent mismatch.

Supported embedding scope#

The tiers and key namespaces are modality-agnostic - the design carries seams for image, audio, and document embedders - but only the text embedder ships. If the MiniLM model cannot load, the embedder falls back to lexical hashing, loudly; deployments where a false positive is worse than an outage can set POINTSMAN_CACHE_REQUIRE_SEMANTIC=true and refuse to start instead.

Lexical verification safety boundaries#

The default lexical verifier requires stored source text. Legacy entries without it are misses and are replaced by normal write-through; switching verification on no longer silently accepts unverifiable old answers. Word matching uses Unicode letters and numbers, so different non-Latin prompts do not compare as two empty ASCII sets.

Digit-bearing, whitespace-delimited tokens must match literally in order, including signs, punctuation, identifiers and repetitions. Changing 100 to 1000, AB123 to AB124, or 1/2 to 2/1 cannot pass just because most words match. This is a conservative filter: a valid paraphrase that changes punctuation around a number can miss. verification=none explicitly disables these checks along with lexical overlap. No new numerical threshold is introduced.

This remains a heuristic, not proof that two questions have interchangeable answers. It does not establish equivalence of spelled-out numbers, negation, units or other facts. For workloads requiring literal request matching, disable semantic text reuse with pointsman.cache.modalities.text.enabled: false; the L1 exact cache remains available. Use X-Cache-Control: no-store to bypass both tiers for a request. The shipped configuration enables semantic text reuse for eligible requests, including temperature: 0. Despite its name, temperature-mode: exact-only skips semantic reuse only for nonzero or omitted temperature; it is not a global L2-off switch. Evaluate semantic reuse against a labeled workload before relying on it. The hashing embedder can map distinct texts to identical vectors; unique request text alone is not a guarantee of a cache miss in a semantic-cache benchmark.