Caching avoids a provider call only when an earlier answer is eligible for this request. Exact reuse and semantic approximation have different correctness guarantees, so Pointsman separates the tiers and exposes how reuse occurred.
Two tiers#
| Tier | Representation | Benefit and tradeoff |
|---|---|---|
| L1 exact | In-process Caffeine entries keyed by the request contract and scope | Fast deterministic reuse; instance-local, bounded by size/TTL and invalidation |
| L2 semantic | Redis Stack vector index and response bodies, using an eligible embedding | Reuses sufficiently similar text; approximation can produce false positives and requires explicit verification |
The key/namespace binds tenant scope, admitted generation/configuration and the relevant request semantics: model scope, conversation context, temperature and tool-related contract. This prevents a configuration change from silently reusing an answer admitted under a different generation. Omitted temperature is distinct from an explicit value. Output caps are validated for replay rather than assumed satisfied by a key match.
Lookup flow#
- Admission and input guardrails run first. Keys reflect the effective, potentially redacted request.
- Apply cache enablement, request directives, temperature/context policy and reuse eligibility.
- Try L1 exact lookup. A cached body still needs a servability check.
- If eligible, derive the semantic context, embed it, query scoped nearest neighbors and apply similarity plus the configured lexical verifier. Exact-prefix is the conservative conversation default; alternative context modes change the reuse tradeoff.
- Validate serving identity, current allowed models, admitted generation, finish reason/output cap and output policy before serving or promoting a response.
- A miss enters routing. Store only an eligible completed result whose actual serving identity remains bound to the admitted scope.
Tools, short text, an unavailable embedder, excessive context, sampling policy or unsupported content can skip semantic reuse while leaving some exact reuse possible. A miss is not necessarily an error. Use X-Cache and X-Cache-Semantic according to the header reference.
Local embedding work#
The performance candidate uses an explicit ONNX worker budget. Its automatic setting uses half of Java's available processors (rounded down), with a minimum of one and a maximum of four. Java's processor count respects container CPU limits.
| Worker control | Behavior | Operator choice |
|---|---|---|
Automatic thread count (0) |
Half the available processors, bounded to 1–4 | Default candidate setting |
| Positive thread count | Explicit intra-operation worker count | Tune against your workload and CPU limit |
| Active-only spinning | Workers can spin during inference, then park after the last concurrent call returns | Reduces wake-up overhead; may increase active CPU use |
| Fully blocking workers | Spinning disabled | Lower active polling at a possible latency cost |
Model weights, tokenization, mean pooling, normalization, the 512-token window and vector namespace are unchanged. Exact L1 hits still avoid semantic embedding. This does not relax the similarity threshold, lexical verification, scope filter, window check or serving-identity checks. More active workers can increase CPU consumption while reducing latency; measure both at your concurrency and CPU limit. See validation and benchmark scope and worker configuration.
Sharing work and writing results#
Eligible synchronous deterministic misses can share a single in-flight computation. Sharing is scoped to the admitted generation; each follower must still pass its own permissions, output-cap and response-policy checks. A leader's successful call is not blanket permission to replay it to every waiter.
L1 insertion is local. L2 writes use a bounded asynchronous path; overflow or a timeout can lose an insertion without failing a completed client response. Live streams use a completion/tee path with its own eligibility, while cache SSE replay starts from a complete validated response. Partial, conflicting or unbound serving results must not become reusable entries.
Failure and isolation#
Cache failure normally becomes a miss or a skipped write. The cache has its own operation bounds and breaker, separate from Redis control counters. A cache timeout therefore does not imply budgets should fail open. A miss can increase provider traffic and spend; reservations remain relevant.
Tenant scope is part of the exact digest and semantic namespace/query filter. That is a structural boundary, but semantic similarity is not a proof of equivalent meaning. The lexical verifier rejects some unsafe matches; it does not prove factual correctness, preserve every negation or make approximate reuse suitable for every application.
Controls and observations#
no-cache bypasses reads while allowing eligible writes; no-store bypasses reuse and storage. Keep deterministic exact caching distinct from opting into reuse of stochastic requests. Invalidation must cover local and shared tiers; expiry and capacity policies bound retention. Hosted embeddings or optional context processing introduce their own service dependencies and content transfers.
The accepted 24-hour run retained 29 lookup and 19 asynchronous store Redis timeouts at the 50 ms Redis command timeout. These did not create unexpected client outcomes or accounting discrepancies, but they are not evidence of zero cache errors. See release status.
Implementation anchors: cache/CacheService, CacheKeys, ExactMatchCache, SemanticCache, ContextResolver, LexicalOverlapHitVerifier, generation scopes and the controller's replay checks. See cache behavior, cache tuning and configuration.