Journal epoch directories

The durable journal stores each request identity in one bounded epoch directory. Its records stay together so existing obligations can settle even when new admission is full. This page explains the storage protocol, configuration bounds and recovery behavior.

The Q=32 single-instance profile on e71e5da8 passed the documented release qualification. Qualification of later runtime changes is separate.

What changes#

Durable record protocol#

Every record follows the same order:

1. Write the file.
2. Fsync the file.
3. Atomically rename it into place.
4. Fsync the containing directory before acknowledging durability.

A step that replaces one record with another - the settled dispatch record replacing its intent, the request's row replacing its request intent - runs as one unit: the new record is written, fsynced and renamed, the directory is fsynced once, and the replaced record is unlinked after that fsync, so the replacement is durable before the old name goes without relying on the filesystem persisting directory operations in issue order. The request intent and the first dispatch intent are written in one unit the same way, immediately before the send. Each step therefore fsyncs its directory once (four times over a request's lifecycle, where the earlier protocol fsynced seven times). The unlink has no additional directory fsync: its persistence can wait because the replacement is already durable. A crash may leave the old intent beside the settled record; recovery keeps the settled record and delivers it once. Before the fused begin's directory fsync completes, a crash may leave either intent alone. No provider send has occurred at that point; surviving intents recover as UNKNOWN, including a dispatch intent without a request intent.

Replay serializes with the identity's writer. It skips a busy identity without blocking unrelated work. Snapshots carry directory-incarnation objects so reusing a slot name cannot validate an old snapshot.

Epoch lifecycle#

The ring uses s00 through the configured last hexadecimal slot.

Only one preparation and one creation operation may exist. Creating/removing states count toward the retained-epoch limit. Held records and failed cleanup can therefore eventually close new admission.

Record replacement failures#

The request-side index avoids speculative probes or unlinks for nonexistent intent files.

Configuration#

All settings below have the POINTSMAN_USAGE_JOURNAL_ environment prefix, are documented in docker/.env.example, and are forwarded by the standard Docker Compose service.

The startup estimate is E × R × (3 + 2D) × 320, with D equal to fallback attempts multiplied by the largest retry count in the initialized retry registry (including its default). At D=10 this is 241,172,480 bytes (230 MiB). The coefficient is a Linux allocation estimate, not a portable memory guarantee. Legacy files, quarantine, positive records, JVM, filesystem and other kernel memory still consume resources. Changing provider retry configurations dynamically would require revalidating this estimate; the release configuration uses the registry's startup instances. The configured limits are ignored when the journal is disabled.

The maintenance cadence is 200 ms; it affects cleanup latency, not the number of directories permitted. Shutdown waits up to 10 s for the worker. If filesystem work or record leases remain active, shutdown refuses to release ownership. The process must fully exit before a successor opens its volume.

Bounded journal I/O#

Every journal filesystem unit - a record's open, write, fsync, atomic rename and directory fsync; an unlink or rename with its directory fsync; a quarantine; a replay read - holds one of IO_CONCURRENCY permits (default 16) from its first filesystem call until the last one has returned or thrown. Nothing inside the unit changed: the protocol, its ordering and its acknowledgement boundaries are the ones above. Units beyond the bound park in arrival order (a fair queue, so replay is not starved) without holding a carrier thread or a descriptor. A wait is never timed out and is not ended by an interrupt; a permit is never released while its unit can still mutate the journal.

A NEW request identity - a request's first record, a cache hit's row, a pre-dispatch failure row - is refused while IO_ADMISSION_WAITING units (default 64) are already waiting: 503 accounting_unavailable, counted as admission_refused{reason="io_saturated"}, decided under the index lock before any file, index entry or epoch admission exists, and before any provider send. This is the same boundary journal-max-entries refuses at. Settlements, dispatch records, deletions and replay are never refused; they wait. A request that settles without a provider send (a cache hit, a coalesced delivery) first secures its journal identity with a request intent, before the hit is announced and before anything settles: a refusal there is a clean 503 whose reservation is released, never a settled admission whose record could not land. This costs a cache hit one more record and one more unlink.

Clean refusals and existing obligations#

A clean journal-admission refusal has no optional usage-ledger failure row. It has incurred no provider obligation and must not wait behind the same stalled storage it just refused.

X-Pointsman-Request-ID joins the RELEASED reservation rows and refusal metrics; a clean refusal has no dispatch or charge. An absent ledger row does not make the refusal unobservable.

What each bound covers#

The I/O settings bound filesystem units, not complete requests.

The intake gate returns 503 overloaded. Journal admission returns 503 accounting_unavailable, including io_saturated or epoch_create_timeout reasons as applicable.

Qualified profile and storage-pressure observation#

The v1 overlay sets IO_ADMISSION_WAITING=32; the gateway default remains 64 and IO_CONCURRENCY remains 16. A smaller admission depth refuses new work sooner during a storage stall.

One diagnostic per depth added 50 ms write latency for 60 seconds at 50 RPS (30 September 2026):

All refusals were clean: no ledger row or provider send, reservation released. These single observations do not establish healthy-burst tolerance or a general performance guarantee. Slow storage still degrades service.

The Q=32 profile later passed the frozen 12-hour and 24-hour qualification on e71e5da8. See release status for scope; this does not change the library default or qualify later runtime changes.

Work outside the permit gate#

Epoch maintenance, replay directory listing/per-record stat, and the quarantine gauge each run on one thread outside the I/O permit gate. Epoch creation stays outside the saturated queue so admissions can obtain a directory without waiting behind that queue. Group commit is not implemented.

Upgrade, corruption and downgrade#

Use one durable volume per replica. Stop the old owner before starting this candidate. Root-level legacy records remain readable and replayable; new identities use epochs. Existing legacy identities settle in place. No bulk flat-layout conversion occurs. Boot plans recovery before deleting any competing record. A valid settled record supersedes its intent only after durability validation. Identical settlements are deduplicated. Conflicting bytes, malformed competing settlements or an identity split between surviving directories refuse startup with the originals unchanged. Conflicting accounting evidence must be reconciled before startup.

Malformed standalone replay records move intact to q/ with collision-resistant names. Quarantine is never automatically erased. Keep the volume and reconcile it against the ledger and reservation manifest. A zero pending gauge only means no replayable records; it does not resolve quarantine. Unknown files and unreadable inventories must not be interpreted as an empty journal.

Held settlement records (EVIDENCE_HELD: a restored database lost their rows) stay in their epoch, retried every minute, counted by gateway_usage_journal_held and alerted by BudgetEvidenceHeld, until an attestation includes them and an operator archives them (POST /admin/budget/held/{event_id}/archive; docs/operating-notes.md). An archived record moves intact to a/ as <event>.<sha256>.held.json: never replayed, never deleted by the gateway, and its epoch slot is released. The gateway and journal-records.py (selector archived) accept a/; a drain refuses it like quarantine. The archive is still read (read-only) as recovery evidence after a restore to before the attestation that included it (docs/operating-notes.md), so keep it on the volume while any restorable backup predates that attestation; move it to the incident record before a downgrade only after that, since an older reader refuses the unknown directory.

Downgrading to a root-only reader is safe only after stopping the gateway and verifying a drained volume:

bash load-test/journal-drain-check.sh /absolute/path/to/stopped-journal

This uses a nonblocking POSIX record lock compatible with Java FileChannel (not flock). It refuses a live owner, missing lock, symlink, unknown entry, temporary record, carried reservation, quarantine or archived held evidence. Keep the gateway stopped between this check and the downgrade. Empty known epoch directories can remain. If any obligation remains, use the newer reader to reconcile/drain it; do not flatten or delete the volume.

Observe and investigate#

• gateway_usage_journal_epochs{state} and gateway_usage_journal_retained_limit: capacity and lifecycle.
• gateway_usage_journal_admission_refused_total{reason}: capacity, exhausted ring, creation failure/timeout.
• gateway_usage_journal_epoch_removals_total / epoch_removal_failures_total{reason}: reclamation progress.
• gateway_usage_journal_index_repairs_total{kind}: adopted or vanished externally observed records.
• gateway_usage_journal_quarantine_records: unresolved evidence; negative means inventory unavailable.
• gateway_usage_journal_io_active, _io_queued, _io_permits: permits in use, units waiting, the bound.
• gateway_usage_journal_admission_duration_seconds{outcome,reason}: how long a NEW identity's admission decision took (the index lock and any wait for epoch creation), admitted or refused. Diagnostic, fixed buckets.
• gateway_journal_refusal_stage_seconds{stage,path,outcome}: where a request refused at journal admission spent its time - to_refusal (receipt to the refusal surfacing), abandon (releasing its admission), failure_row (its failure row; outcome is skipped for a clean journal refusal - the row is intentionally not attempted - and written/refused/failed where one was attempted) and total. Refusals slower than 1 s are also sampled to the log at most once a second, with the X-Pointsman-Request-ID value on the line, never as a label.
• gateway_usage_journal_io_wait_seconds and gateway_usage_journal_io_duration_seconds{op} (record, delete, move, quarantine, replay_read): time waiting for a permit and time holding one.
• gateway_usage_journal_identities: identities holding records or leases.
• gateway_requests_accepted_outstanding and gateway_stream_settlements_in_flight: accepted requests not yet settled, and settlements running. Both are observation gauges, not admission caps: nothing refuses a request because of them.
• Existing pending, delivery, replay failure and held-settlement metrics remain relevant.

Prometheus rules warn when half the ring stays retained for ten minutes, warn on io_saturated refusals (JournalIoSaturated: journal storage slower than the arrival rate), and alert on other non-capacity admission refusals or quarantine. Inspect held reservations, delivery failures, the volume's permissions and free space. Do not relieve pressure by deleting accounting evidence. A failed tmp write may retain a nonempty epoch until boot recovery removes its incomplete temporary file; that consumes a bounded slot and must be investigated.

load-test/journal-records.py supports strict root/epoch inventory, selectors, checksums and the drain check. Live counts are observations, not atomic accounting snapshots. Freeze or stop writers for evidence capture. The reconciler reads both layouts and refuses unknown directories or hidden quarantine obligations.

Quarantine and journal storage: monitoring and the operator procedure#

Quarantine is never erased automatically. That is deliberate: a record the gateway cannot read is evidence, and deleting it silently would turn an accounting question into an accounting loss. Its storage is therefore indefinite until an operator acts, not bounded. Two signals cover it:

• gateway_usage_journal_quarantine_records and the critical alert JournalQuarantineEvidence (!= 0). This fires on any quarantined record, and also on -1, which means the inventory is unreadable.
• gateway_usage_journal_volume_usable_bytes and the critical alert JournalVolumeLowSpace. This fires below 1 GiB usable for 5 minutes, or on -1, which means the file store is unreadable. A full volume refuses new request identities with 503 accounting_unavailable; this alert says so first.

To resolve quarantine, with the gateway stopped (or the volume mounted read-only elsewhere):

1. Inventory: python3 load-test/journal-records.py <journal-root> paths --selector quarantine.
2. Manifest: python3 load-test/journal-records.py <journal-root> manifest | grep ' q/' > quarantine.sha256. The manifest mode hashes every record with paths relative to the root, and quarantine lives under q/.
3. Copy off the volume and verify the copy: copy q/ with its relative layout, then run sha256sum -c quarantine.sha256 from the copy's root.
4. Reconcile each record against the usage ledger and the reservation manifest. A record that still describes unsettled usage is an accounting question, not a storage one.
5. Remove the verified, reconciled files by hand. Record what was removed, and where the verified copy lives.

drain refuses while quarantine evidence exists, so a non-zero quarantine can only reach zero through this procedure. Never relieve journal-volume pressure by deleting accounting evidence: grow the volume instead.