# Tune the semantic cache

Tune semantic reuse against a representative workload. Measure hit rate, inspect whether reused answers are appropriate, and retain the same tenant and serving checks throughout the comparison. Similarity thresholds and lexical verification reduce false matches; neither proves answer equivalence.

For tier behavior and architecture, see [the semantic cache](../semantic-cache.md) and [cache design](../design/cache.md).

<a id="the-instrument-first"></a>

## Read cache outcomes

Inspect the cache verdict on each response:

```http
X-Cache: HIT-L1
X-Cache-Semantic: not-consulted
X-Cache-Threshold: 0.92
```

| Signal | Meaning | Next check |
|---|---|---|
| `X-Cache: HIT-L1` | Exact, in-process reuse | Confirm the request and answer contract match your intent |
| `X-Cache: HIT-L2` | Semantic reuse from the shared index | Evaluate sampled matches and answer suitability |
| `X-Cache: MISS` | No response reused | Read `X-Cache-Semantic` for the reason |
| `skipped-temperature` | Sampling policy excludes semantic reuse | Default `exact-only` requires explicit `temperature: 0` |
| `replay-refused` | Stored serving identity or contract is unsuitable | The request proceeds as a miss through normal routing |
| `skipped-mutable-image` | Remote image bytes can change | The mutable-image request is not cached |
| `X-Cache-Threshold` | Current similarity floor | Compare with the intended configuration |

Omitting temperature is distinct from declaring zero. Under the default policy, omitted or nonzero temperature stays off semantic L2; eligible exact requests can still use L1. Other temperature modes change this tradeoff. See the complete [header reference](../reference/response-headers.md).

Read the aggregate view before changing configuration:

```bash
curl -s -u "$POINTSMAN_ADMIN_USERNAME:$POINTSMAN_ADMIN_PASSWORD" \
  "localhost:8080/admin/observe/analytics/cache?org=$ORG"
```

| Field | Use |
|---|---|
| `hits`, `misses`, `hit_rate_pct` | Measure reuse for this workload |
| `hit_mean_latency_ms`, `miss_mean_latency_ms` | Compare observed paths; provider time and workload can differ |
| `saved_estimate_usd` | Inspect estimated avoided provider cost; this is not invoice verification |

<a id="the-one-dial-that-matters"></a>

## Choose a similarity threshold

`pointsman.cache.modalities.text.threshold` defaults to `0.92`. It is the cosine-similarity floor for considering a semantic candidate; the other verification and serving checks still apply.

| Change | Likely effect | Risk to evaluate |
|---|---|---|
| Lower the threshold | More candidate matches | More unsuitable reuse, including superficially similar questions |
| Raise the threshold | Fewer candidate matches | Lower hit rate; false matches are still possible |
| Disable semantic text reuse | Exact L1 remains available | Provider traffic can increase |

The threshold can be retuned on a running gateway:

```bash
curl -s -X POST localhost:8080/actuator/env \
  -u "$POINTSMAN_ADMIN_USERNAME:$POINTSMAN_ADMIN_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"name":"pointsman.cache.modalities.text.threshold","value":"0.95"}'
```

Check `X-Cache-Threshold` on subsequent responses. This changes the running process's environment; persist the chosen value in deployment configuration so it survives a restart.

<a id="tune-from-evidence-not-vibes"></a>

## Evaluate sampled matches

`POINTSMAN_CACHE_DEBUG_SAMPLE_RATE` defaults to `0.02` and samples semantic hits into durable `cache_hit_samples` rows.

1. Record the current threshold, hit rate and workload.
2. Collect sampled matches at that configuration.
3. Evaluate whether the matched prompts can safely reuse the same answer. Include negation, quantities, identifiers and domain-specific distinctions.
4. Change one setting and repeat the evaluation on comparable traffic.
5. Persist the chosen setting and continue monitoring both hit rate and unsuitable matches.

> **Prompt retention:** samples contain scores and metadata by default. Setting `POINTSMAN_CACHE_DEBUG_SAMPLE_CONTENT=true` also stores prompt text durably and emits a startup warning. Apply your data-retention policy before enabling it.

The lexical verifier provides an additional filter. `POINTSMAN_CACHE_VERIFICATION_MIN_OVERLAP` defaults to `0.4`; a vector match whose words barely overlap is rejected. The verifier also applies the documented literal checks for digit-bearing tokens. These checks remain heuristic; see [verification boundaries](../semantic-cache.md#lexical-verification-safety-boundaries).

<a id="what-never-caches-and-why"></a>

## Check eligibility and bypass directives

Different conditions skip different parts of the cache.

| Condition | Read behavior | Write or scope behavior |
|---|---|---|
| Prompt shorter than `min-prompt-chars` (default 20) | Skip L2 with `skipped-short`; L1 can still serve | Do not treat this as a global cache-off switch |
| `X-Cache-Control: no-cache` | Skip both tiers | Eligible writes remain enabled |
| `X-Cache-Control: no-store` | Skip both tiers | Skip writes and single-flight |
| Another tenant's entry | Cannot be reused across the admitted tenant scope | Tenant isolation is mandatory |

Use `no-store` when a request must neither read nor populate the cache. Use `no-cache` when you want a fresh provider response while still allowing normal write-through.

## Lifetimes

Set retention independently from similarity.

| Setting | Default or behavior | Operator decision |
|---|---|---|
| L1 TTL | 15 minutes | Balance local reuse against freshness |
| L1 capacity | 10,000 entry-equivalents plus configured payload-weight bounds | Size for response lengths; entry count is not a heap-memory measurement |
| L2 TTL | 24 hours | Shorten for volatile answers |
| `refresh-on-hit` | Extend an entry's lifetime on reuse | Account for frequently accessed entries |
| `max-age-hours` | 168 hours | Cap how long an entry can remain reusable |
| `model-ttl-hours` | Per-model overrides | Give volatile models shorter retention |
| `pointsman.cache.version` | Cache namespace salt | Changing it makes previous entries unreachable |

Invalidate one literal tenant ID with:

```bash
curl -X DELETE "localhost:8080/internal/cache?tenant=$TENANT_ID" \
  -u "$POINTSMAN_ADMIN_USERNAME:$POINTSMAN_ADMIN_PASSWORD"
```

See [invalidation and capacity behavior](../semantic-cache.md#operating-it) for shared L2 and replica-local L1 handling.

## When the cache degrades

Cache lookup failures normally become misses. A failed Redis operation can produce:

```http
X-Cache: MISS
X-Cache-Semantic: error
```

`POINTSMAN_CACHE_L2_OP_TIMEOUT_MS` defaults to 50 ms **per Redis command**. It does not bound embedding plus the entire multi-command lookup. The L2 circuit breaker can skip work while open; provider admission and budget checks still govern the resulting miss.

| Observation | Investigate |
|---|---|
| Occasional timeout | Command duration, scheduling and network jitter |
| Sustained `error` | Redis availability, command cost, network distance and resource pressure |
| Low semantic hit rate without errors | Eligibility, threshold, corpus, occupancy and verification refusals |

Use logs and metrics to establish the cause before changing the timeout. A larger timeout can increase foreground waiting. The [release observations](../release-status.md#cache-timeouts) include bounded cache timeouts; qualification does not imply zero cache errors.

## When the cache stops accepting writes

L2 write admission is separate from lookup. Reads can continue while new entries are refused; hit rate may decline as old entries expire. Inspect `gateway.cache.store.skipped`, `gateway.cache.l2.store_checks_failing` and `l2WriteAdmission` health details.

| Reason | What it means | What to do |
|---|---|---|
| `open` | Writes accepted | Monitor occupancy and retention |
| `full` | Observed document count reached `POINTSMAN_CACHE_L2_MAX_DOCUMENTS` | Review retention and document/memory capacity |
| `size_unknown` | Document count could not be read | Inspect Redis command errors and timeout logs |
| `store_checks` | Required cache memory/eviction checks failed or were unreadable | Verify cache-only Redis configuration and connectivity |
| `*_tolerated` | A bounded unreadable observation is tolerated | Watch for persistence and stale checks |

### Set retention and capacity

1. Estimate retained unique entries from write rate and TTL, allowing for refresh and eviction.
2. Set the document cap above the expected retained population with headroom.
3. Measure payload sizes and Redis usage, then set a compatible memory bound.
4. Configure cache-only Redis with `maxmemory` and `allkeys-lru` or `allkeys-lfu`; enable enforced store checks for the intended deployment profile.

The document cap is soft because admitted writes can still land. Redis memory is a separate bound. **Counter Redis requires `noeviction`** and must not inherit cache eviction settings.

### Diagnose write-admission pauses

Admission checks run every `POINTSMAN_CACHE_L2_SIZE_CHECK_MS` (default 10 seconds). A short closure alone does not identify the cause: inspect the recorded reason and underlying command result.

`POINTSMAN_CACHE_L2_SIZE_UNKNOWN_TOLERANCE` defaults to three unreadable readings. Bounded tolerance avoids reacting to every isolated missing reading; persistent or stale observations still close writes. Do not increase tolerance to conceal an unknown store state.

<a id="the-honest-loop"></a>

## Verify the tuning result

Compare both outcomes after each change:

- **Reuse benefit:** hit rate, observed latency and estimated avoided provider calls.
- **Reuse quality:** whether sampled matches and outputs remain suitable for the application.

A higher hit rate alone is not a successful tuning result.