Pointsman is a gateway for a documented subset of Chat Completions. Applications send a virtual key and an OpenAI-shaped request; the gateway applies access controls, reservations, guardrails and routing before calling a configured provider. The dashboard is its administration interface. It does not execute an application's tools or provide an agent runtime.
These guides describe the qualified implementation at e71e5da8 and its documentation successors. They explain responsibilities and failure boundaries, not a promise that every optional feature is enabled in the shipped profile.
Start with the system boundary#
| Part | Owns | State and trust boundary |
|---|---|---|
| Gateway | Authentication, admission, cache decisions, model selection, provider calls, streaming and settlement | Per-request state, local caches and routing snapshots; optional durable local usage journal |
| Dashboard | Operator workflows and configuration review | Browser session; server-side authorization remains authoritative |
| PostgreSQL | Identities, configuration, usage, decisions and audit records | Durable administrative and accounting store |
| Redis / Redis Stack | Atomic limits and reservation counters; optional semantic cache | Counter and cache clients have different timeouts and failure policies |
| Providers | Model inference and provider-reported usage | External credentials and billing; the gateway's estimates are not an invoice |
| Policy service | Background reward processing and published learner parameters | One Python worker with persistent SQLite; private network only |
A static documentation website is outside this runtime. Hosting docs.pointsman.dev does not require running the gateway, PostgreSQL, Redis or the learner.
Follow a request#
- Bound intake and authenticate. Apply request/body and in-flight bounds, validate the virtual key, resolve its identity, and parse supported controls.
- Resolve and reserve. Determine allowed models and capabilities, bound context/output requirements, reserve limits and spend, and acquire applicable concurrency slots.
- Apply input policy. Run enabled input guardrails; enforced redaction changes the working request before its cache key is built.
- Consider reuse. Check exact and eligible semantic cache entries against this request's admitted generation, permissions and response contract. A hit still passes response policy.
- Select an allowed route. Filter candidates, apply the configured strategy, and keep fallback inside the same governance boundary.
- Call the provider. Bind the attempted deployment and credential; translate the supported contract, collect usage and classify failures.
- Handle the response. Apply complete-response output checks or choose an eligible streaming path. Only reusable, bound results may populate the cache.
- Settle each terminal path. Record observed or estimated usage, reconcile reservations, release permits and submit eligible learning signals. A failure after an upstream call does not erase incurred spend.
Live streaming overlaps steps 6–8. Before response commitment, a refusal can be an HTTP error. After commitment, a terminal failure is an SSE error and an incomplete stream; HTTP 200 alone is not success. Enforcing whole-response output policy may require buffering and replay, which delays the first byte.
For measured costs and timer boundaries, see the latency and benchmark breakdown.
Read each design#
| Stage | Questions answered |
|---|---|
| Authentication and identity | How are keys resolved, revoked and isolated? How does admin login differ? |
| Policy-based authorization | How do scopes, grants, delegation and model-policy inheritance work? Which path is enabled? |
| Rate limiting, budgets and settlement | Why reserve before execution? What is atomic? What happens on counter loss? |
| Guardrails | How do monitor, enforce, redaction and streaming interact? |
| Exact and semantic cache | What may be reused, under whose identity, and when must it become a miss? |
| Routing and adaptive selection | Which decisions are deterministic, which learn, and what does quality mean? |
| Policy service | How are reward revisions persisted, deduplicated and delivered back to the router? |
Design principles and tradeoffs#
Admit before spending. Permission, capability and budget decisions constrain every attempt; fallback does not create a second ungoverned route.
Separate an optimization from a safety decision. A semantic cache timeout becomes a miss. An unverifiable enforcing spend cap is refused under the default budget policy. These deliberately have different availability tradeoffs.
Keep the decision local when possible. Cached identities, configuration snapshots and local routing parameters reduce request latency. Their update/invalidation rules must be understood; cached state is not instantly consistent across arbitrary replicas.
Make outcomes attributable. The model requested, model actually served, cached response, interrupted stream, incurred spend and learning observation are separate facts. Reusing one as proof of another creates incorrect accounting or learning.
Supported deployment scope#
The accepted release scope is the documented single-instance v1 Compose profile and the supported API subset. Weighted-random remains the default. Adaptive routing, quality policies, judges, cascades and Jev are experimental and opt-in. This guide does not establish multi-instance availability, zero-downtime upgrades or quality-preserving savings.
Read release status, deployment, operating notes and known issues before changing the deployment boundary.