# Journal epoch directories

The durable journal stores each request identity in one bounded epoch directory. Its records stay together so existing obligations can settle even when new admission is full. This page explains the storage protocol, configuration bounds and recovery behavior.

The Q=32 single-instance profile on `e71e5da8` passed the documented [release qualification](../release-status.md). Qualification of later runtime changes is separate.

## What changes

### Durable record protocol

Every record follows the same order:

1. Write the file.
2. Fsync the file.
3. Atomically rename it into place.
4. Fsync the containing directory before acknowledging durability.

A step that replaces one record with another - the settled dispatch record replacing its intent, the request's row
replacing its request intent - runs as one unit: the new record is written, fsynced and renamed, the directory is fsynced once, and
the replaced record is unlinked after that fsync, so the replacement is durable before the old name goes
without relying on the filesystem persisting directory operations in issue order. The request intent and the first dispatch intent
are written in one unit the same way, immediately before the send. Each step therefore fsyncs its directory once
(four times over a request's lifecycle, where the earlier protocol fsynced seven times).
The unlink has no additional directory fsync: its persistence can wait because the replacement is already durable.
A crash may leave the old intent beside the settled record; recovery keeps the settled record and delivers it once.
Before the fused begin's directory fsync completes, a crash may leave either intent alone. No provider send has
occurred at that point; surviving intents recover as `UNKNOWN`, including a dispatch intent without a request intent.

Replay serializes with the identity's writer. It skips a busy identity without blocking unrelated work. Snapshots carry directory-incarnation objects so reusing a slot name cannot validate an old snapshot.

### Epoch lifecycle

The ring uses `s00` through the configured last hexadecimal slot.

| Transition | Required evidence | Failure behavior |
|---|---|---|
| Prepare a directory | Fsync the journal root before use | A timed-out waiter does not cancel creation or add another worker |
| Assign identities | Remain within the epoch's cumulative capacity | Rotate only to a prepared durable directory |
| Retire a directory | No mapped identities or in-flight pins; empty listing; successful removal and root fsync | Retain the slot on failure |
| Ring exhausted | No reusable slot | Refuse new accounting identities; do not create directories without bound |

Only one preparation and one creation operation may exist. Creating/removing states count toward the retained-epoch limit. Held records and failed cleanup can therefore eventually close new admission.

### Record replacement failures

| Failure point | Retained state |
|---|---|
| Before atomic rename | Previous PRESENT record remains |
| Fused begin before its directory fsync | Neither, either or both intents may survive; no provider send yet; surviving intents recover as UNKNOWN |
| Directory fsync after rename | New record remains indexed for replay |
| After a record's directory fsync, before the unlink of the record it replaces (one unit) | Both records remain and stay indexed; boot recovery keeps the settled record and discards its intent; replay delivers it once |
| Uncertain quarantine rename | Retain the identity for restart/reconciliation; do not report delivery |

The request-side index avoids speculative probes or unlinks for nonexistent intent files.

## Configuration

All settings below have the `POINTSMAN_USAGE_JOURNAL_` environment prefix, are documented in
`docker/.env.example`, and are forwarded by the standard Docker Compose service.

| Suffix | Default | Meaning |
|---|---:|---|
| `EPOCH_IDENTITIES` | 1024 | Maximum cumulative new identities assigned to one epoch |
| `MAX_RETAINED_EPOCHS` | 32 | Ring directory limit, 2–256; every lifecycle state counts |
| `EPOCH_MAX_AGE_MS` | 60000 | Soft age rotation when another durable directory is ready |
| `EPOCH_CREATE_WAIT_MS` | 250 | Admission waiter deadline; not a filesystem I/O cancellation guarantee |
| `NAME_BUDGET_BYTES` | 268435456 | Startup ceiling for the configured namespace-memory estimate |

The startup estimate is `E × R × (3 + 2D) × 320`, with D equal to fallback attempts multiplied by the
largest retry count in the initialized retry registry (including its default). At D=10 this is 241,172,480
bytes (230 MiB). The coefficient is a Linux allocation estimate, **not a portable memory guarantee**.
Legacy files, quarantine, positive records, JVM, filesystem and other kernel memory still consume resources.
Changing provider retry configurations dynamically would require revalidating this estimate; the release
configuration uses the registry's startup instances. The configured limits are ignored when the journal is disabled.

The maintenance cadence is 200 ms; it affects cleanup latency, not the number of directories permitted.
Shutdown waits up to 10 s for the worker. If filesystem work or record leases remain active, shutdown refuses
to release ownership. The process must fully exit before a successor opens its volume.

## Bounded journal I/O

Every journal filesystem unit - a record's open, write, fsync, atomic rename and directory fsync; an unlink or
rename with its directory fsync; a quarantine; a replay read - holds one of `IO_CONCURRENCY` permits (default 16)
from its first filesystem call until the last one has returned or thrown. Nothing inside the unit changed: the
protocol, its ordering and its acknowledgement boundaries are the ones above. Units beyond the bound park in
arrival order (a fair queue, so replay is not starved) without holding a carrier thread or a descriptor. A wait
is never timed out and is not ended by an interrupt; a permit is never released while its unit can still mutate
the journal.

A NEW request identity - a request's first record, a cache hit's row, a pre-dispatch failure row - is refused
while `IO_ADMISSION_WAITING` units (default 64) are already waiting: 503 `accounting_unavailable`, counted as
`admission_refused{reason="io_saturated"}`, decided under the index lock before any file, index entry or epoch
admission exists, and before any provider send. This is the same boundary `journal-max-entries` refuses at.
Settlements, dispatch records, deletions and replay are never refused; they wait. A request that settles without a
provider send (a cache hit, a coalesced delivery) first secures its journal identity with a request intent, before the
hit is announced and before anything settles: a refusal there is a clean 503 whose reservation is released, never a
settled admission whose record could not land. This costs a cache hit one more record and one more unlink.

### Clean refusals and existing obligations

A clean journal-admission refusal has no optional usage-ledger failure row. It has incurred no provider obligation and must not wait behind the same stalled storage it just refused.

| Request state | Treatment |
|---|---|
| Direct request refused at first record | Return the clean refusal; release reservation and concurrency |
| Cache hit or coalesced delivery refused before settlement | Do not announce the hit; release its reservation |
| Stream refused before provider stream starts | No provider dispatch; clean refusal |
| Coalesced follower already settling an output-policy block | Preserve the 400 outcome |
| Partially written record, provider bill or already-owed settlement | Keep existing failure-row and recovery handling |

`X-Pointsman-Request-ID` joins the RELEASED reservation rows and refusal metrics; a clean refusal has no dispatch or charge. An absent ledger row does not make the refusal unobservable.

### What each bound covers

The I/O settings bound filesystem units, not complete requests.

| Bound | Covers | Does not cover |
|---|---|---|
| `IO_CONCURRENCY` | Journal units currently holding a filesystem permit | Other blocking work or the complete JDK carrier pool |
| `IO_ADMISSION_WAITING` | Queue pressure at which new identities refuse | A hard ceiling on all queued units; existing obligations still join |
| `EPOCH_CREATE_WAIT_MS` | Time a new identity waits for directory creation before I/O admission | Number of those waiters |
| `journal-max-entries` | Admitted journal identities | An HTTP request concurrency limit |
| `pointsman.intake.max-in-flight` | Concurrent `/v1` HTTP request lifecycles (default 256) | Settlement work continuing after timeout or disconnect |

The intake gate returns `503 overloaded`. Journal admission returns `503 accounting_unavailable`, including `io_saturated` or `epoch_create_timeout` reasons as applicable.

| Suffix | Default | Meaning |
|---|---:|---|
| `IO_CONCURRENCY` | 16 | Journal filesystem units running at once, 1-256; about two descriptors per unit. It limits how many journal units block in the kernel at once; it does not bound the JDK carrier pool, which other blocking work also grows and which keeps compensating carriers after a burst |
| `IO_ADMISSION_WAITING` | 64 | Waiting units at which new identities are refused, 1-100000 |

### Qualified profile and storage-pressure observation

The v1 overlay sets `IO_ADMISSION_WAITING=32`; the gateway default remains 64 and `IO_CONCURRENCY` remains 16. A smaller admission depth refuses new work sooner during a storage stall.

One diagnostic per depth added 50 ms write latency for 60 seconds at 50 RPS (30 September 2026):

| Observation | Depth 64 | Depth 32 |
|---|---:|---:|
| Descriptor rise | +71 | +39 |
| Journal queue plateau | 64 | 32 |
| Refused requests | 2,685 | 2,706 |
| Successful requests during stall | 313 | 291 |
| Successful-request p99 during stall | 18.2 s | 11.5 s |

All refusals were clean: no ledger row or provider send, reservation released. These single observations do not establish healthy-burst tolerance or a general performance guarantee. Slow storage still degrades service.

The Q=32 profile later passed the frozen 12-hour and 24-hour qualification on `e71e5da8`. See [release status](../release-status.md) for scope; this does not change the library default or qualify later runtime changes.

### Work outside the permit gate

Epoch maintenance, replay directory listing/per-record `stat`, and the quarantine gauge each run on one thread outside the I/O permit gate. Epoch creation stays outside the saturated queue so admissions can obtain a directory without waiting behind that queue. Group commit is not implemented.

## Upgrade, corruption and downgrade

Use one durable volume per replica. Stop the old owner before starting this candidate. Root-level legacy
records remain readable and replayable; new identities use epochs. Existing legacy identities settle in
place. No bulk flat-layout conversion occurs. Boot plans recovery before deleting any competing record.
A valid settled record supersedes its intent only after durability validation. Identical settlements are
deduplicated. Conflicting bytes, malformed competing settlements or an identity split between surviving
directories refuse startup with the originals unchanged. Conflicting accounting evidence must be reconciled before startup.

Malformed standalone replay records move intact to `q/` with collision-resistant names. Quarantine is
never automatically erased. Keep the volume and reconcile it against the ledger and reservation manifest.
A zero pending gauge only means no replayable records; it does not resolve quarantine. Unknown files and
unreadable inventories must not be interpreted as an empty journal.

Held settlement records (EVIDENCE_HELD: a restored database lost their rows) stay in their epoch, retried every
minute, counted by `gateway_usage_journal_held` and alerted by BudgetEvidenceHeld, until an attestation includes them
and an operator archives them (`POST /admin/budget/held/{event_id}/archive`; `docs/operating-notes.md`). An archived
record moves intact to `a/` as `<event>.<sha256>.held.json`: never replayed, never deleted by the gateway, and its
epoch slot is released. The gateway and `journal-records.py` (selector `archived`) accept `a/`; a drain refuses it
like quarantine. The archive is still read (read-only) as recovery evidence after a restore to before the attestation
that included it (`docs/operating-notes.md`), so keep it on the volume while any restorable backup predates that
attestation; move it to the incident record before a downgrade only after that, since an older reader refuses the
unknown directory.

Downgrading to a root-only reader is safe only after stopping the gateway and verifying a drained volume:

```bash
bash load-test/journal-drain-check.sh /absolute/path/to/stopped-journal
```

This uses a nonblocking POSIX record lock compatible with Java FileChannel (not `flock`). It refuses a live
owner, missing lock, symlink, unknown entry, temporary record, carried reservation, quarantine or archived held
evidence.
Keep the gateway stopped between this check and the downgrade. Empty known epoch directories can remain.
If any obligation remains, use the newer reader to reconcile/drain it; do not flatten or delete the volume.

## Observe and investigate

- `gateway_usage_journal_epochs{state}` and `gateway_usage_journal_retained_limit`: capacity and lifecycle.
- `gateway_usage_journal_admission_refused_total{reason}`: capacity, exhausted ring, creation failure/timeout.
- `gateway_usage_journal_epoch_removals_total` / `epoch_removal_failures_total{reason}`: reclamation progress.
- `gateway_usage_journal_index_repairs_total{kind}`: adopted or vanished externally observed records.
- `gateway_usage_journal_quarantine_records`: unresolved evidence; negative means inventory unavailable.
- `gateway_usage_journal_io_active`, `_io_queued`, `_io_permits`: permits in use, units waiting, the bound.
- `gateway_usage_journal_admission_duration_seconds{outcome,reason}`: how long a NEW identity's admission decision took
  (the index lock and any wait for epoch creation), admitted or refused. Diagnostic, fixed buckets.
- `gateway_journal_refusal_stage_seconds{stage,path,outcome}`: where a request refused at journal admission spent its
  time - `to_refusal` (receipt to the refusal surfacing), `abandon` (releasing its admission), `failure_row` (its
  failure row; `outcome` is `skipped` for a clean journal refusal - the row is intentionally not attempted - and
  written/refused/failed where one was attempted) and `total`. Refusals slower than 1 s
  are also sampled to the log at most once a second, with the `X-Pointsman-Request-ID` value on the line, never as a
  label.
- `gateway_usage_journal_io_wait_seconds` and `gateway_usage_journal_io_duration_seconds{op}` (`record`, `delete`,
  `move`, `quarantine`, `replay_read`): time waiting for a permit and time holding one.
- `gateway_usage_journal_identities`: identities holding records or leases.
- `gateway_requests_accepted_outstanding` and `gateway_stream_settlements_in_flight`: accepted requests not yet
  settled, and settlements running. Both are observation gauges, not admission caps: nothing refuses a request
  because of them.
- Existing pending, delivery, replay failure and held-settlement metrics remain relevant.

Prometheus rules warn when half the ring stays retained for ten minutes, warn on `io_saturated` refusals
(`JournalIoSaturated`: journal storage slower than the arrival rate), and alert on other non-capacity admission
refusals or quarantine. Inspect held reservations, delivery failures, the volume's permissions and free space.
Do not relieve pressure by deleting accounting evidence. A failed tmp write may retain a nonempty epoch
until boot recovery removes its incomplete temporary file; that consumes a bounded slot and must be investigated.

`load-test/journal-records.py` supports strict root/epoch inventory, selectors, checksums and the drain check.
Live counts are observations, not atomic accounting snapshots. Freeze or stop writers for evidence capture.
The reconciler reads both layouts and refuses unknown directories or hidden quarantine obligations.

## Quarantine and journal storage: monitoring and the operator procedure

Quarantine is never erased automatically. That is deliberate: a record the gateway cannot read is evidence, and
deleting it silently would turn an accounting question into an accounting loss. Its storage is therefore **indefinite
until an operator acts**, not bounded. Two signals cover it:

- `gateway_usage_journal_quarantine_records` and the critical alert **JournalQuarantineEvidence** (`!= 0`). This
  fires on any quarantined record, and also on `-1`, which means the inventory is unreadable.
- `gateway_usage_journal_volume_usable_bytes` and the critical alert **JournalVolumeLowSpace**. This fires below
  1 GiB usable for 5 minutes, or on `-1`, which means the file store is unreadable. A full volume refuses new request
  identities with 503 `accounting_unavailable`; this alert says so first.

To resolve quarantine, with the gateway stopped (or the volume mounted read-only elsewhere):

1. **Inventory:** `python3 load-test/journal-records.py <journal-root> paths --selector quarantine`.
2. **Manifest:** `python3 load-test/journal-records.py <journal-root> manifest | grep '  q/' > quarantine.sha256`.
   The `manifest` mode hashes every record with paths relative to the root, and quarantine lives under `q/`.
3. **Copy off the volume and verify the copy:** copy `q/` with its relative layout, then run
   `sha256sum -c quarantine.sha256` from the copy's root.
4. **Reconcile** each record against the usage ledger and the reservation manifest. A record that still describes
   unsettled usage is an accounting question, not a storage one.
5. **Remove** the verified, reconciled files by hand. Record what was removed, and where the verified copy lives.

`drain` refuses while quarantine evidence exists, so a non-zero quarantine can only reach zero through this
procedure. Never relieve journal-volume pressure by deleting accounting evidence: grow the volume instead.