Latency and benchmark breakdown

Pointsman processes requests through authentication, model authorization, budgets, guardrails, caching, routing and durable usage accounting. Latency depends on the path: an exact cache hit avoids provider dispatch and semantic embedding, a semantic miss performs cache lookup before forwarding, and an explicit cache bypass skips cache lookup. This page reports those paths separately and explains the work measured by each stage.

Use the path that matches your application, and keep three measurements separate:

For streaming, time to first token and time to completion are separate measurements. A slow reader can increase completion time without increasing time to first token. The short synchronous benchmark below measures neither streaming TTFT nor real-provider generation time.

Usage-journal unit fusion comparison#

29 October 2026: candidate runtime c8de5d86 against the optimized runtime 2981c99a, same PostgreSQL profile, model, CPU and memory in all four runs. The candidate changes one thing: how often the durable usage journal fsyncs its directory. Each request used to force the journal directory seven times (after each of four record renames and after each of three unlinks). The candidate performs a record write and the unlink of the record it replaces as one unit: the new record is written, fsynced and atomically renamed, the directory is fsynced once, and only then is the replaced record unlinked, so the replacement is durable before the old name is removed on any filesystem whose directory fsync persists the operations issued before it. The request intent is written together with the first dispatch intent in one unit immediately before the send. A request therefore forces the directory four times. Both intents must be durable before a provider send. A replacement is durable before its old intent is unlinked; the unlink itself has no additional directory fsync, so a crash may leave both names and recovery keeps the settled record. A crash before the fused begin's directory fsync may leave either intent alone; no provider send has occurred at that point, and surviving intents recover as UNKNOWN. PostgreSQL fsync, synchronous_commit and full_page_writes stayed on; no SQL, lock, transaction boundary or acknowledgement boundary changed.

The candidate was measured twice. Its first form, e6f4d43b, unlinked the replaced record before the unit's directory fsync and passed the same rule (38.21 / 38.11 ms against 41.66 / 41.82 ms). A review then found that this order made the replacement's durability before the old name's removal depend on the filesystem persisting directory operations in issue order, which ext4 and XFS do, POSIX does not promise, and the journal volume check does not require. c8de5d86 moves the unlink after the fsync, with the same four forces per request, and the comparison was repeated on it in full. The tables below are that second run; the first run's data remain downloadable at the end of this section.

The acceptance rule was fixed before the run: both candidate read-bypass p99 values below both control values. It passed: 37.05 / 37.70 ms against 41.31 / 40.22 ms. Medians fell on every gateway path. Rarer tails (p99.9, maximum) moved less, in both directions, and remain storage-sensitive, as the sections below explain.

Each path uses 60 seconds of discarded warmup and 120 seconds measured at 50 requests/second against a zero-delay local provider, ordered candidate → control → control → candidate with fresh stores for every run. / separates independent repetitions; these are total client response times, not pooled percentiles.

Read-bypass tail, stages and journal units#

provider.attempt includes the request and dispatch intents written before the send and the settled dispatch record written after it; ledger.write includes the row's journal record, the settlement transaction and the record's removal. Both stages lost directory fsyncs. Stage values are inclusive means per invocation and are not additive.

A candidate record unit may write two files, or write one, fsync the directory and then unlink another, which is why its mean is higher while there are fewer of them. Unlinking after the fsync did not lengthen the unit: 3.41 / 3.53 ms here against 3.58 / 3.55 ms in the first run. One delete per request in both variants belongs to the background replay of the dispatch record, not to the request thread. The manifest transactions (prepare, transition, settle) were unchanged within 0.15 ms.

CPU and verification#

JVM process CPU includes background work during each measured window; it excludes PostgreSQL and other services. No CPU change is claimed.

All 96,000 measured requests completed with HTTP 200 (72,000 through the gateway, 24,000 directly to the fixture), exactly 6,000 per path in every run, with zero measured timeout, transport failure, dropped launch or outstanding request. In every run the ledger rows equal the successful gateway responses and the distinct event ids, each carries the fixture's two settled budget charges with matching amounts and evidence, every dispatch record was delivered, and journal pending, held, failure and replay-failure counts are zero. No container restart or OOM was observed; all owned run containers stopped. In the last run one semantic-miss request's index lookup window was skipped; it was served as a miss with HTTP 200 and a ledger row, and the harness's consultation gate (at least 99 percent) passed.

The candidate JAR differs from the control JAR only in the usage-journal, journal epoch store and router classes. Before the comparison, 26 targeted test classes (270 tests: journal protocol and crash-state tests, settlement transaction atomicity, budget restore and recovery, stream settlement and the router) passed on the benchmark host with real PostgreSQL containers. Six new journal tests pin the fused protocol and its crash states, the rename-fsync-unlink order, and that a dispatch intent found without its request intent boots and replays as an UNKNOWN send. This short comparison does not transfer any 12-hour/24-hour qualification to the candidate.

The shipped whole-stack crash-resolution drill was run on 29 October against both forms of the candidate, e6f4d43b and c8de5d86, and passed both times (45 checks each, none failed) with the same recovered figures: the gateway was killed with SIGKILL while 20 dispatched requests were unanswered, restarted on the same journal, recovered each request's intent and its dispatch intent as UNKNOWN (20 interrupted ledger rows, 20 unresolved dispatches, 20 admitted reservations), held the budget counter at settled plus held through the alert's 15-minute window, raised and then resolved the unresolved-reservation alert after one accepted attestation, refused the under-floor and stale attestations without writing, and served, charged and forwarded nothing during the attestation race. One crash shape was exercised, not power loss, and a process kill cannot reach the reordering state that c8de5d86 removes. The endurance run remains to be done before release.

Download client timings and attempt counts, stage, transaction and journal-unit measurements, and JVM CPU observations of this run; the first run's client timings, stages, CPU and chart are kept.

WAL segment-size comparison#

29 October 2026: same application runtime 2981c99a in all four runs. This comparison isolates PostgreSQL WAL segment size: 1 MiB → 16 MiB → 16 MiB → 1 MiB, with fresh stores for each repetition. Both settings retain 4 MiB WAL buffers, eager initialization, recycling, fdatasync, fsync, synchronous commits and full-page writes. Gateway journal, SQL, model, CPU and memory settings are identical.

The 1 MiB setting was not adopted. It did not consistently improve read-bypass p99: the acceptance rule required both candidate repetitions to be lower than both 16 MiB controls. Lower p99.9 values alone do not establish a p99 fix. The shipped PostgreSQL profile remains unchanged, and the runtime remains pending separate 12-hour/24-hour qualification.

Each path uses 60 seconds of discarded warmup and 120 seconds measured at 50 requests/second against a zero-delay local provider. / separates independent repetitions; these are total client response times, not pooled percentiles.

Read-bypass tail and accounting stages#

Stage values are inclusive means per invocation and are not additive. The complete CSV preserves all measured stages, transaction components, journal operations and histogram bounds for all four paths.

CPU and verification#

JVM process CPU includes background work during each measured window; it excludes PostgreSQL and other services and is not total deployment CPU.

All 96,001 measured requests completed with HTTP 200 (72,001 through the gateway, 24,000 directly to the fixture), with zero measured timeout, transport failure, dropped launch or outstanding request. The downloadable CSV reports actual attempt counts and per-run latency summaries; no response was excluded. The final semantic-miss window included one extra scheduler-boundary request (6,001); it is included under the predeclared time-window population rule. Final ledger rows match successful gateway responses, each has the fixture's two settled budget charges, charge amounts and evidence match, and journal pending/held counts are zero. No container restart or OOM was observed; all owned run containers stopped.

Semantic consultation retains the ≥99% gate. No cache-error counter or cache-timeout log was observed in these short runs; the accepted long-run cache observations remain disclosed in release status.

Download client timings and attempt counts, stage and accounting component measurements, and JVM CPU observations.

Performance candidate#

28 October 2026: candidate runtime 2981c99a, compared with qualified runtime e71e5da8 in four fresh runs ordered candidate → baseline → baseline → candidate. Both variants used the same durable profile with the intentional worker-policy change described below.

Values separated by / show independent repetitions, not pooled percentiles or confidence intervals. These are total client-observed response times.

Client response times#

Read the result by path:

• Semantic miss: both candidate medians and p99s were lower than both baseline repetitions.
• Read bypass: both medians improved, but the second candidate p99 rose to 51.58 ms, above both baselines. The comparison does not establish a consistent tail-latency improvement for this path.
• Exact hit: medians improved modestly; the p99 ranges overlap.

Stage breakdown#

The instrumented semantic-miss stages changed as follows. These are means per invocation, with inclusive boundaries; do not add them to reconstruct a response or treat a provider-attempt span as pure model inference.

CPU cost#

The worker policy can trade more active CPU for lower latency. The following is JVM process CPU time per completed request, including background work during each measured window; it is not wall time or total deployment CPU. The gateway retains the same 8-CPU allowance. The read-bypass requests use no-cache, so eligible asynchronous stores and their embedding work still run. no-store is a different path and was not measured here. The direct-fixture window's idle/background JVM CPU is retained in the downloadable CPU CSV.

Durable accounting components#

Manifest transactions and journal I/O explain some of the remaining accounting cost:

A record operation includes writing, file fsync, atomic rename and directory fsync. A delete operation includes unlink and directory fsync. A request can perform several operations; these per-operation means are nested observations, not extra stages to sum with the table above. In this runtime a request that reaches a provider performs four record and three delete operations on its own thread; the journal unit fusion candidate performs three and one. The downloadable CSV retains their counts and journal-admission/I/O-wait observations.

Cache execution and accounting durability#

• Cache lookup: size ONNX workers to half the JVM container CPU count (one to four workers), permit spinning during active inference, and park workers when the last concurrent inference returns. In this 8-CPU rig the candidate has four intra-op workers; the baseline uses automatic ONNX sizing with fully blocking workers. These are intentional worker-policy changes alongside the runtime update. Model weights, tokenizer, input window, pooling, normalization, similarity thresholds, lexical verification, scopes and replay checks are unchanged. This does not change the 50 ms Redis timeout or establish that earlier long-run cache timeouts are resolved.
• Durable accounting: batch reservation transitions and settlement writes; verify charge amounts in one query; remove an unused prepare read. Reservation locks, the budget-trust lock, conflict checks and the single ledger/charge/state transaction remain. Replay retains the original evidence and operation identity. No file fsync, directory fsync, atomic rename, durable deletion ordering or acknowledgement boundary was removed. PostgreSQL fsync, synchronous_commit and full_page_writes were all on in both variants.

The candidate JAR differs in four entries: the embedding adapter, its configuration wiring, shipped application settings and reservation-manifest class. Packaged libraries, model/tokenizer artifacts and journal classes are byte-identical. The candidate image is b0c1d3a43ab3; the baseline image is a294897dcb30.

Verification and remaining scope#

All 96,000 measured requests completed with HTTP 200: 72,000 through the gateway and 24,000 directly to the fixture. Each path passed its cache-header checks, with zero measured timeouts, transport failures, dropped launches or outstanding requests. Semantic consultation retains the original ≥99% gate; conservative window-check bypasses are disclosed rather than counted as semantic lookups:

Final captures matched successful gateway responses to unique ledger rows, matched charges to settled reservations, found no charge/amount/evidence mismatch or unresolved settlement, and verified journal drain. All owned containers stopped with no observed restart or OOM. No cache error counter or cache-timeout log was observed in these short runs; the longer-run observations remain disclosed in release status.

183 distinct targeted Java tests passed with no skips, plus six offline analysis tests, including real PostgreSQL JPA/JDBC rollback and replay, conflicting charges, missing siblings, journal failure/recovery bounds and cache checks. Real MiniLM vectors matched the upstream implementation at the existing 1e-6 component tolerance across empty, Unicode, short, mixed-batch and window-boundary inputs, including overlapping calls and parked-worker reuse.

Qualification boundary: this performance candidate has not repeated the 12-hour/24-hour qualification. The accepted release evidence belongs to e71e5da8.

The short named-model comparison does not establish:

• Adaptive-routing performance or superiority.
• Long-prompt or streaming performance.
• Maximum production capacity or a production latency SLO.
• Semantic-hit answer quality.

Durable filesystem and database work still contributes to latency. The response acknowledgement retains its existing durability boundary.

Download the per-run client timings and attempt counts, stage/component timings with counts and histogram bounds, and CPU observations.

Storage-sensitive tail latency#

Cache read bypass skips cache lookup, but still performs rate-limit admission, provider dispatch and durable accounting. X-Cache-Control: no-cache also permits eligible background cache stores; no-store disables both cache reads and writes. Neither directive disables accounting or changes the response acknowledgement boundary.

A separate instrumented diagnostic on runtime 2981c99a measured 6,000 requests on each of three paths. All 18,000 completed with HTTP 200 and reconciled exactly.

PostgreSQL commit waits coincided with repeated response-time spikes even with no-store. Twenty commit-stall clusters began within 2.6 ms of new 16 MiB WAL-file creation. No measured virtual-thread pinning or long GC pause explained those clusters. These are temporal correlations from a JFR-instrumented diagnosis, not per-request causal traces or a reproduction of every earlier outlier.

The commit-stall clusters account for the responses above 50 ms, roughly 0.3% of requests, and not for the p99 itself: both storage experiments below moved p99.9 and the maximum and left p99 in place. The same diagnostic recorded about nine journal fsyncs of a millisecond or more per read-bypass request, which the journal unit fusion comparison addressed.

PostgreSQL creates and flushes WAL segments before using them. Eager initialization can produce a foreground I/O burst until enough files can be recycled. Disabling zero-fill still requires a flush and can shift allocation work into ordinary writes; it did not consistently improve p99 on this test rig. All runs retained database flush, synchronous-commit and full-page-write protections and the gateway journal barriers.

Changing segment size is a separate storage-profile choice: PostgreSQL fixes it when a cluster is initialized. Existing databases do not change segment size when an initialization environment variable changes. Smaller segments also increase rotation frequency and file count; archival and replication workloads require their own validation. See PostgreSQL WAL configuration and cluster initialization.

Benchmark environment#

• One gateway container per run, Java 21 with generational ZGC, 256–640 MiB heap and a 2.5 GiB container limit. The application comparison uses qualified baseline image a294897dcb30 and candidate b0c1d3a43ab3; the storage comparison uses b0c1d3a43ab3 in every run.
• PostgreSQL 16.15 on an ext4 filesystem backed by a nonrotational virtual disk. WAL buffers are 4 MiB; the storage comparison explicitly records 1 or 16 MiB segments. These results do not characterize every storage device or topology.
• Shared 16-vCPU Linux KVM host. Gateway affinity: eight vCPUs; load generator: three; stores, provider fixture and learner: four. Runs are sequential and use fresh stores, with no unrelated containers active during the comparison.
• Durable journal, enforced counter/cache-store checks, reservation manifests and transaction-first settlement enabled; intake limit 32. Authentication and local guardrails enabled. INFO logging; tracing export disabled.
• Local MiniLM and Redis semantic cache enabled with identical model, tokenizer and native-library artifacts. The intentional worker-policy difference is documented above. Tokenizer accounting uses cl100k_base.
• Short synthetic user prompts: soak plus ten pseudorandom vocabulary items, temperature: 0, nonstreaming, no tools/images, one named model and one provider deployment. Exact hits use a prepopulated set of 20 prompts.
• Internal Docker HTTP network with no external connectivity. WireMock injects zero provider delay. No real-provider, external guardrail, judge or optional adaptive-routing call is part of the measured request.

What each section does#

The request lifecycle explains ordering and failure handling. These timer names let an operator connect a latency observation to that design.

gateway.cache.l2.op and pointsman.budget.manifest provide additional component timers. L2 lookup timing can include embedding; it is not a Redis-only network measurement. Asynchronous cache storage is background work and must not be added to the foreground response latency. Manifest timers are nested within broader operations.

Policy authorization and the policy service#

The scope/grant authorization core is enabled by default, while request-path tree composition is disabled by default. This benchmark keeps those defaults. It measures virtual-key authentication and the fixed request allow-set path; it does not assign a millisecond cost to administrative grant changes, deep scope inheritance or opt-in tree composition.

The Python policy service is a learner, separate from authorization. Normal adaptive selection uses a gateway-local parameter snapshot; reward delivery and parameter refresh happen in the background. This named-model test does not exercise adaptive choices and does not establish an adaptive-routing latency number. Optional external judges, cascades and context processing need separate measurements.

Read the stage numbers correctly#

The boundary scrapes bracket the client measurement and the harness's own metric reads. Their populations can differ slightly from the client-window population because requests cross those boundaries. Raw counts are retained; the tables do not claim exact per-request tracing.

When comparing results:

• Keep stage p50s and p99s separate: they describe different observations, and some timers nest.
• Allow for uninstrumented work, HTTP encoding/decoding, scheduling, socket transfers and response delivery. A residual from unrelated percentiles cannot identify “network time.”
• Use total response latency for cache hits. A hit avoids the provider call, so subtracting a direct-provider distribution does not measure forwarding overhead.

Scope and limitations#

This is a short, single-instance, warm, fixed-arrival-rate benchmark with a local zero-delay fixture and short synthetic prompts. It is separate from the accepted 12-hour and 24-hour reliability runs. Their latency distributions include simulated provider delay, long prompts, streaming, deliberate cancellations and slow readers; they must not be presented as gateway overhead.

These measurements do not establish a production p99 SLO, maximum capacity, real semantic-hit quality or a comparison with another gateway.

Reproduce and measure your deployment#

Use the repository's standalone load-test/OverheadScenario on disposable, isolated stores and a fake provider.

1. Start an isolated rig. The harness targets an already running environment; it does not provision isolation for an arbitrary target.
2. Record runtime/image identity, effective configuration, CPU/memory limits, native model/library hashes and corpus.
3. Set endpoint and output variables explicitly, using a fresh output directory for each repetition.
4. Run direct, miss, read-bypass and hit paths with the same client/load shape. Separate warmup from measurement.
5. Retain every attempt outcome, cache header and start/end metric. Report timeouts, transport failures and dropped launches beside percentiles.
6. Drain pending work, then verify final accounting and resource cleanup.

Harness configuration#

Report an absent stage as “not exercised” or “unavailable.” To measure an optional feature, change only that feature in a matched comparison and inspect both client distributions and stage timers. A difference in p99s remains a distribution difference, not a per-request overhead percentile.