Learning history: window, maintenance and recovery (learner)

This runbook covers the learner's durable learning history: the record that lets it deduplicate retried rewards and revise ratings. The ts2 history is bounded by a supported window and compacted in the background. Legacy ts1 rows retain a separate cap, described below.

See policy service design for event identity, commit order and failure behavior. Implementation anchors are BanditStore.compact_ts2 and ThompsonRegistry.maintain / _check_history_window.

What is kept, what is compacted#

• The arm state is the checkpoint. It holds alpha, beta, pulls, evidence mass, epoch and the persisted change detector. Maintenance never changes it, so /policy/parameters publishes the same posteriors before and after a pass.
• The history is the per-decision record. For each decision it holds:
• the event identities with their payload hashes and original acknowledgements;
• each observation's immutable context, fixed uniform draw and the mass it was applied with;
• pending ratings.

A decision older than the window is compacted: its rows are deleted, and its contribution stays in the arm state. - The watermark is committed with the deletion. Every pass is one transaction: it advances ts2_history_watermark_ms and deletes every ts2 row below it, or does neither. A crash, a disk error or a kill at any point leaves the fence and the rows consistent, and the next pass completes the work. - An event for a decision outside the window is refused. The test is: older than now - W - skew, or below the watermark. The response is 410 outside_history_window, and nothing is applied. Its identity may already be gone, so applying it could apply it twice.

The window decides, not whether maintenance has run, so the answer never depends on maintenance timing. A decision dated more than the skew allowance in the future is refused 422 decision_time_in_future. - The decision time is the gateway's. It is the call's start, minted when the decision is made. For a rating it is carried in the HMAC-signed decision envelope (v3, its own field beside the seal time), or in the deciding instance's own memo; both routes send the same value, so a retried rating is a duplicate wherever it lands. A feedback caller cannot supply or alter it. - ts1 rows are never compacted. A legacy (pre-ts2) event carries no decision time, so its identity cannot be fenced. ts1 rows therefore keep their deduplication forever under BANDIT_REWARD_EVENT_LIMIT, which now caps ts1 rows only. At that cap ts1 learning is refused 503 history_full_ts1, as before. No current gateway writes ts1; only a rolled-back gateway would.

The window and its floor#

W = BANDIT_HISTORY_WINDOW_S (default 7,200 s). The learner refuses to start when W is below:

BANDIT_HISTORY_ENVELOPE_TTL_S (900)  - the gateway's decision-envelope validity, pointsman.bandit.decision-memory-ttl-min x 60
+ BANDIT_HISTORY_RETRY_HORIZON_S (1800) - the longest a gateway holds a reward: bounded queue x bounded retries
+ BANDIT_HISTORY_SKEW_S (300)        - clock skew between gateway and learner
= 3,000 s at the defaults

The floor exists so that accepted feedback is never shortened to make storage fit. A gateway that honours longer envelopes than the learner assumes logs a warning and sets the gauge gateway.bandit.learner.history.window_short to 1. The gateway reads the learner's assumption from the published history block.

A rating can arrive up to the request's own duration plus the envelope validity after its decision time: the memo and the envelope are written once the provider has answered. A stream has no total duration bound (only an idle timeout), so the gateway also checks each completed decision: when its duration plus the decision-memory TTL exceeds the learner's window, it counts gateway.bandit.feedback.horizon_beyond_window and logs a warning on the change of state. A late rating for such a decision will be refused outside_history_window; raise W if long requests are normal.

Capacity#

BANDIT_HISTORY_MAX_EVENTS (default 1,000,000) bounds the retained ts2 event rows. The steady state is events per second x W. An event is a completion, a failed attempt or a rating revision; a duplicate is not an event.

At the default W of 2 h, a budget of 1,000,000 supports a sustained ~138 events/s. The reviewed 50-RPS endurance profile produces fewer than 60 events/s, including failed attempts and ratings.

Size the learner volume for the configured event budget, SQLite indexes and WAL, and leave headroom for maintenance. Bytes per event and durable transaction throughput depend on the host, event mix and history; the arithmetic above is a capacity requirement, not a throughput guarantee.

Compaction releases SQLite pages for reuse. It need not shrink the database file on disk. GET /policy/history reports current file sizes in storage_bytes; monitor them alongside event counts, reward refusals and maintenance lock time. Verify apply throughput and compaction behavior on the intended deployment before increasing the event budget.

When the budget is reached, the learner first compacts, inside the arriving event's own transaction, every row that is already outside the window but was waiting for the next maintenance run (counted as maintenance.inline_compacted_events). Only when decisions still inside the window fill the budget does it refuse new ts2 events with 507 history_full_within_window. Nothing inside the window is evicted. The gateway drops these under gateway.bandit.rewards.dropped{reason=refused_history_full_within_window}; routing continues on the last published parameters. Learning resumes by itself once the window moves past the oldest decisions and maintenance compacts them. To stop the refusals sooner, do one of these:

1. raise BANDIT_HISTORY_MAX_EVENTS (storage grows by the bytes-per-event figure above);
2. reduce the arrival rate (fewer rated or retried events);
3. if, and only if, the gateway's own envelope validity and retry horizon really are shorter, lower W within its floor.

Never delete rows or the database to make room. That drops deduplication for decisions a retry may still present.

What to watch#

• GET /policy/history (learner) reports:
• window_ms, skew_ms, envelope_ttl_ms, watermark_ms;
• ts2_events / ts2_limit / utilization, and ts1_events / ts1_limit;
• maintenance: runs, passes, compacted events, last run, last_error, and inline_compacted_events (rows compacted by an arriving event because the budget was reached before maintenance ran);
• refused: counts of typed refusals by signal kind and reason;
• storage_bytes.

The same summary rides /policy/parameters as history. - The learner log has a WARNING once per crossing of 80% utilization, INFO for each compaction, and an ERROR with traceback for a failed maintenance pass. The next tick retries a failed pass, which changed nothing. - Gateway gauges: - gateway.bandit.learner.history.utilization (-1 before a learner publishes it); - gateway.bandit.learner.history.events; - gateway.bandit.learner.history.window_short; - gateway.bandit.params.age.seconds (a stalled learner).

The gateway's drop counter carries one reason per refusal: refused_outside_history_window, refused_history_full_within_window, refused_conflict, refused_composition_mismatch, refused_decision_time_in_future and refused_invalid. The learner also refuses 422 semantics_mismatch for a ts1 event naming a key tagged with another learning semantics. A current gateway never sends one; it guards against a hand-made post and against a pre-ts2 gateway, which since envelope v3 cannot open a ts2 decision's envelope at all. An unreachable learner is still retries_exhausted. A refusal is never retried and never counts toward opening the breaker.

Known limit: the wall clock#

The window, the watermark and the future-time refusal all read the learner's wall clock. A large clock step is not detected: - Forward (more than the skew): maintenance compacts decisions that are really still inside the window. A retry of one of them is then refused 410 outside_history_window instead of answered as a duplicate. Nothing is applied twice; the retry's evidence is lost. - Backward (more than the skew): new decisions are refused 422 decision_time_in_future until the clock catches up. The watermark never moves back, so nothing already compacted is re-admitted.

Run NTP on the learner and gateway hosts; the default skew allowance (300 s) is far above normal NTP error.

Recovery#

• Learner restart. It reloads the arm states, the watermark and the history, then catches up maintenance on the first tick. Retries inside the window still get their original acknowledgements.
• Store unavailable (locked past BANDIT_DB_TIMEOUT_SECS, disk full). The answer is 503 store_unavailable, and nothing moved. The gateway retries within its bound, then drops and counts.
• Restore from a backup. Take the backup with SQLite's online backup API (or stop the learner and copy the file). On restore, the state is exactly the backup's.
• Retries of events acknowledged before the backup return their original acknowledgements.
• Decisions compacted before the backup are still refused.
• Events acknowledged AFTER the backup are gone; the gateway does not resend them. That is the backup's recovery point. Nothing is applied twice.
• Interrupted maintenance. Every pass either committed (fence and deletion) or did not happen. Restart and let the next tick finish.

Upgrade and rollback#

The storage changes are additive: a decision_at_ms column on reward_events with its partial index, and the watermark in schema_meta. An older learner opening the store ignores them.

A ts2 gateway needs a ts2 learner: upgrade the learner first. Rolling the gateway back returns it to the frozen ts1 keys and the ts1 cap.