Tune semantic reuse against a representative workload. Measure hit rate, inspect whether reused answers are appropriate, and retain the same tenant and serving checks throughout the comparison. Similarity thresholds and lexical verification reduce false matches; neither proves answer equivalence.
For tier behavior and architecture, see the semantic cache and cache design.
Read cache outcomes#
Inspect the cache verdict on each response:
X-Cache: HIT-L1
X-Cache-Semantic: not-consulted
X-Cache-Threshold: 0.92
| Signal | Meaning | Next check |
|---|---|---|
X-Cache: HIT-L1 |
Exact, in-process reuse | Confirm the request and answer contract match your intent |
X-Cache: HIT-L2 |
Semantic reuse from the shared index | Evaluate sampled matches and answer suitability |
X-Cache: MISS |
No response reused | Read X-Cache-Semantic for the reason |
skipped-temperature |
Sampling policy excludes semantic reuse | Default exact-only requires explicit temperature: 0 |
replay-refused |
Stored serving identity or contract is unsuitable | The request proceeds as a miss through normal routing |
skipped-mutable-image |
Remote image bytes can change | The mutable-image request is not cached |
X-Cache-Threshold |
Current similarity floor | Compare with the intended configuration |
Omitting temperature is distinct from declaring zero. Under the default policy, omitted or nonzero temperature stays off semantic L2; eligible exact requests can still use L1. Other temperature modes change this tradeoff. See the complete header reference.
Read the aggregate view before changing configuration:
curl -s -u "$POINTSMAN_ADMIN_USERNAME:$POINTSMAN_ADMIN_PASSWORD" \
"localhost:8080/admin/observe/analytics/cache?org=$ORG"
| Field | Use |
|---|---|
hits, misses, hit_rate_pct |
Measure reuse for this workload |
hit_mean_latency_ms, miss_mean_latency_ms |
Compare observed paths; provider time and workload can differ |
saved_estimate_usd |
Inspect estimated avoided provider cost; this is not invoice verification |
Choose a similarity threshold#
pointsman.cache.modalities.text.threshold defaults to 0.92. It is the cosine-similarity floor for considering a semantic candidate; the other verification and serving checks still apply.
| Change | Likely effect | Risk to evaluate |
|---|---|---|
| Lower the threshold | More candidate matches | More unsuitable reuse, including superficially similar questions |
| Raise the threshold | Fewer candidate matches | Lower hit rate; false matches are still possible |
| Disable semantic text reuse | Exact L1 remains available | Provider traffic can increase |
The threshold can be retuned on a running gateway:
curl -s -X POST localhost:8080/actuator/env \
-u "$POINTSMAN_ADMIN_USERNAME:$POINTSMAN_ADMIN_PASSWORD" \
-H 'Content-Type: application/json' \
-d '{"name":"pointsman.cache.modalities.text.threshold","value":"0.95"}'
Check X-Cache-Threshold on subsequent responses. This changes the running process's environment; persist the chosen value in deployment configuration so it survives a restart.
Evaluate sampled matches#
POINTSMAN_CACHE_DEBUG_SAMPLE_RATE defaults to 0.02 and samples semantic hits into durable cache_hit_samples rows.
- Record the current threshold, hit rate and workload.
- Collect sampled matches at that configuration.
- Evaluate whether the matched prompts can safely reuse the same answer. Include negation, quantities, identifiers and domain-specific distinctions.
- Change one setting and repeat the evaluation on comparable traffic.
- Persist the chosen setting and continue monitoring both hit rate and unsuitable matches.
Prompt retention: samples contain scores and metadata by default. Setting
POINTSMAN_CACHE_DEBUG_SAMPLE_CONTENT=truealso stores prompt text durably and emits a startup warning. Apply your data-retention policy before enabling it.
The lexical verifier provides an additional filter. POINTSMAN_CACHE_VERIFICATION_MIN_OVERLAP defaults to 0.4; a vector match whose words barely overlap is rejected. The verifier also applies the documented literal checks for digit-bearing tokens. These checks remain heuristic; see verification boundaries.
Check eligibility and bypass directives#
Different conditions skip different parts of the cache.
| Condition | Read behavior | Write or scope behavior |
|---|---|---|
Prompt shorter than min-prompt-chars (default 20) |
Skip L2 with skipped-short; L1 can still serve |
Do not treat this as a global cache-off switch |
X-Cache-Control: no-cache |
Skip both tiers | Eligible writes remain enabled |
X-Cache-Control: no-store |
Skip both tiers | Skip writes and single-flight |
| Another tenant's entry | Cannot be reused across the admitted tenant scope | Tenant isolation is mandatory |
Use no-store when a request must neither read nor populate the cache. Use no-cache when you want a fresh provider response while still allowing normal write-through.
Lifetimes#
Set retention independently from similarity.
| Setting | Default or behavior | Operator decision |
|---|---|---|
| L1 TTL | 15 minutes | Balance local reuse against freshness |
| L1 capacity | 10,000 entry-equivalents plus configured payload-weight bounds | Size for response lengths; entry count is not a heap-memory measurement |
| L2 TTL | 24 hours | Shorten for volatile answers |
refresh-on-hit |
Extend an entry's lifetime on reuse | Account for frequently accessed entries |
max-age-hours |
168 hours | Cap how long an entry can remain reusable |
model-ttl-hours |
Per-model overrides | Give volatile models shorter retention |
pointsman.cache.version |
Cache namespace salt | Changing it makes previous entries unreachable |
Invalidate one literal tenant ID with:
curl -X DELETE "localhost:8080/internal/cache?tenant=$TENANT_ID" \
-u "$POINTSMAN_ADMIN_USERNAME:$POINTSMAN_ADMIN_PASSWORD"
See invalidation and capacity behavior for shared L2 and replica-local L1 handling.
When the cache degrades#
Cache lookup failures normally become misses. A failed Redis operation can produce:
X-Cache: MISS
X-Cache-Semantic: error
POINTSMAN_CACHE_L2_OP_TIMEOUT_MS defaults to 50 ms per Redis command. It does not bound embedding plus the entire multi-command lookup. The L2 circuit breaker can skip work while open; provider admission and budget checks still govern the resulting miss.
| Observation | Investigate |
|---|---|
| Occasional timeout | Command duration, scheduling and network jitter |
Sustained error |
Redis availability, command cost, network distance and resource pressure |
| Low semantic hit rate without errors | Eligibility, threshold, corpus, occupancy and verification refusals |
Use logs and metrics to establish the cause before changing the timeout. A larger timeout can increase foreground waiting. The release observations include bounded cache timeouts; qualification does not imply zero cache errors.
When the cache stops accepting writes#
L2 write admission is separate from lookup. Reads can continue while new entries are refused; hit rate may decline as old entries expire. Inspect gateway.cache.store.skipped, gateway.cache.l2.store_checks_failing and l2WriteAdmission health details.
| Reason | What it means | What to do |
|---|---|---|
open |
Writes accepted | Monitor occupancy and retention |
full |
Observed document count reached POINTSMAN_CACHE_L2_MAX_DOCUMENTS |
Review retention and document/memory capacity |
size_unknown |
Document count could not be read | Inspect Redis command errors and timeout logs |
store_checks |
Required cache memory/eviction checks failed or were unreadable | Verify cache-only Redis configuration and connectivity |
*_tolerated |
A bounded unreadable observation is tolerated | Watch for persistence and stale checks |
Set retention and capacity#
- Estimate retained unique entries from write rate and TTL, allowing for refresh and eviction.
- Set the document cap above the expected retained population with headroom.
- Measure payload sizes and Redis usage, then set a compatible memory bound.
- Configure cache-only Redis with
maxmemoryandallkeys-lruorallkeys-lfu; enable enforced store checks for the intended deployment profile.
The document cap is soft because admitted writes can still land. Redis memory is a separate bound. Counter Redis requires noeviction and must not inherit cache eviction settings.
Diagnose write-admission pauses#
Admission checks run every POINTSMAN_CACHE_L2_SIZE_CHECK_MS (default 10 seconds). A short closure alone does not identify the cause: inspect the recorded reason and underlying command result.
POINTSMAN_CACHE_L2_SIZE_UNKNOWN_TOLERANCE defaults to three unreadable readings. Bounded tolerance avoids reacting to every isolated missing reading; persistent or stale observations still close writes. Do not increase tolerance to conceal an unknown store state.
Verify the tuning result#
Compare both outcomes after each change:
- Reuse benefit: hit rate, observed latency and estimated avoided provider calls.
- Reuse quality: whether sampled matches and outputs remain suitable for the application.
A higher hit rate alone is not a successful tuning result.