Jev task routing (experimental, opt-in)

Jev can classify what a conversation asks for. Pointsman uses that label to apply an operator-supplied model ranking within the candidates already permitted by routing policy. This is task-aware selection, not evidence that the selected model is optimal. It does not learn model quality, alter budget enforcement, or replace the existing bandit strategies.

Request flow#

flowchart TD
    A[Resolve model group and existing candidate filters] --> B{Family preference or session affinity selects?}
    B -->|Yes| P[Existing provider path]
    B -->|No: group uses jev| C{Tenant explicitly allowed and multiple models eligible?}
    C -->|No| F[First eligible candidate]
    C -->|Yes| H{Validated task hint?}
    H -->|Yes| M[Apply configured task model ranking]
    H -->|No| I[Check complete text context, characters, messages and serialized bytes]
    I --> J[Immediate capacity, RPM, lifetime and circuit checks]
    J --> K[One bounded Jev call per request]
    K --> L[Validate version, schema, probabilities and confidence]
    L --> M
    I -->|Unsupported or too large| F
    J -->|Refused| F
    K -->|Error or deadline| F
    L -->|Unknown or low confidence| F
    M -->|Eligible match| P
    M -->|No eligible match| F
    F --> P

The normal provider admission and settlement path still runs. Jev cannot authorize models, raise budgets or reopen unavailable deployments. Candidate ordering within the same concrete model remains the existing order. Model mappings select concrete model IDs; they do not introduce new candidates or choose credentials. Family preference and session affinity keep their existing precedence.

The question vocabulary is code, summarize, extract, reasoning, knowledge, chat and unknown. Question version is task-v1, with model version pinned initially to jev-1.13.0. A validated X-Task-Type hint avoids the external call.

Accepted classification includes all supplied text messages, including system and developer messages. Eligibility is decided before strategy execution and retained on the request memo, so later tiers do not ask again.

Request deadline#

TIMEOUT_MS bounds one exchange, including connection, headers and body, on the request thread. A request makes at most one exchange even when several routing tiers are considered. This is a per-exchange bound; the gateway does not supply a global end-to-end request deadline for it to inherit.

Configure deliberately#

Enabling exports accepted conversation text to the configured TypeSafe endpoint. Obtain the necessary tenant permission and review the provider's current data handling terms before enabling. No prompt, credential or remote error body is logged by this integration. Requests are not retried or redirected.

1. Set POINTSMAN_JEV_API_KEY through your deployment's secret mechanism.
2. Set POINTSMAN_JEV_ALLOWED_TENANTS to explicit tenant IDs (comma separated). There is no wildcard; _anon is an explicit opt-in if authentication is disabled.
3. Provide evaluated task rankings and set strategy: jev on an existing group in your operator routing configuration. For example, merge this into that configuration, replacing the model IDs and group with your actual catalog:

pointsman:
  jev:
    task-models:
      code: [your-code-model, your-general-model]
      summarize: [your-economy-model, your-general-model]
      reasoning: [your-analysis-model, your-general-model]
  routing:
    groups:
      your-existing-group:
        strategy: jev
        # Keep this group's existing candidate definitions here.

Check the existing routing.yml group schema when editing. Use an additional Spring YAML configuration file or the existing configuration-management path; this example is a fragment, not a complete runnable deployment.

4. Set POINTSMAN_JEV_ENABLED=true and restart. Missing credentials, tenant allowlist or mappings refuse startup. Properties are bound at startup; restart after changing Jev limits, key, allowlist or mappings.

With Jev enabled, abstention uses the first eligible candidate. With Jev disabled, a group still naming jev uses the registry's existing default-strategy fallback and warning, just like other disabled strategies. For deterministic rollback, change the group's strategy to priority as well as disabling Jev.

Explicit bounds and accounting#

Every setting is listed in docker/.env.example, with placeholders in gateway/src/main/resources/routing.yml. Defaults are initial operating choices, not latency guarantees or calibrated quality thresholds.

Attempt/rate/concurrency limits and the circuit breaker are shared across tenants inside one gateway process, reset on restart, and multiply with replicas. They are not an account-wide budget. Allocate the account's current limits across replicas and leave headroom for other callers. The finite lifetime allowance intentionally falls back after exhaustion; increase it explicitly only after reviewing usage.

Jev calls are operator overhead and are NOT charged to the existing tenant model usage ledger. Provider-reported input tokens are exported; timeouts or invalid usage increment an unverified-usage counter. These counters are operational telemetry, not durable billing. An uncertain call may still incur provider charges. Use provider billing/account limits for financial reconciliation. Do not describe this feature as enforcing an all-in tenant budget that includes classification.

The request-local memo retains both successful assessments and abstentions across tier decisions. Separate requests are never memoized together. There is no shared classification cache, distributed rate limit, shadow mode or model-quality learner in this first slice.

Metrics, reasons and what they count#

Two populations, deliberately not one:

• pointsman.jev.invocations{outcome,reason} - one per entry to the classifier, whether or not it reached the wire. outcome is one of classified, abstained (a valid answer the policy declined: unknown_task, low_confidence), skipped (no call by shape or policy: no_context, unsupported_context, tools_declared, input_limit, serialization), refused (this process's bounds: capacity, rate_limit, call_limit, circuit_open) or failed (the dependency: timeout, connect_failed, interrupted, upstream_rate_limit, upstream_overloaded, upstream_auth, upstream_invalid_request, upstream_error, response_limit, invalid_usage, bad_protocol, transport_failed).
• pointsman.jev.decisions{reason} - one per routing decision, so it can exceed invocations: a request's memoised assessment serves every tier it is decided for, and the strategy-only reasons (disabled, tenant_not_allowed, single_model, invalid_hint, no_task_match, hint_no_task_match, selection_error, hint_<task>, classified_<task>) appear only here. no_task_match follows a classification (an invocation happened); hint_no_task_match follows a hint (none did) - the telemetry-consistency alert relies on that distinction.
• pointsman.jev.selection{effect} - did the decision move? ranked when a task ranking chose a candidate priority would not have; first for every fallback and for a ranking whose top eligible model was already first. Read this to know whether the feature is doing anything.
• pointsman.jev.calls (admitted attempts), .duration (the whole exchange), .input.tokens (provider-reported), .usage.unverified (attempts whose usage was not reported validly - each may still have been charged), .confidence (every valid answer's, including those below the threshold), and the gauge pointsman.jev.enabled (1 while the strategy bean is wired, so an alert can tell enabled-and-silent from off).

Every reason is a member of one closed vocabulary (JevReason); no tenant, prompt, request id or remote error string becomes a tag. Decision provenance carries the question version and the reason (jev_task-v1_<reason>); the structured decision event carries the pinned model version, the question version and the ranking digest (pointsman.jev.ranking-version on the admin effective-config read) so a decision can be read back to the policy it was made under. Classification confidence is a summary of a distribution, not a probability of being right, and never a bandit reward.

Migration from the first slice (2026-09-23). Existing metric names are unchanged. Three labels changed meaning: invalid_or_failed_response split into bad_protocol (the body violated the pinned contract) and transport_failed (an unclassified transport error); timeout now also covers the HTTP client's own deadline, which previously surfaced as invalid_or_failed_response because it arrived wrapped; and a request that declares tools now reports tools_declared instead of no_context. Dashboards keyed on the old labels must follow. The Grafana board Pointsman - Jev task routing is provisioned from docker/grafana/.

Configure alerts#

The jev-classifier group in docker/grafana/alert-rules.yml is RENDERED from docker/grafana/alert-policy.env by scripts/render-alert-rules.sh - the window, the minimum event volume, the two ratios and the for durations live in that one file, edited at deployment time and never in Java; JevAlertRulesRenderedTest refuses a committed block that was not rendered from the committed policy. Every rule is evaluated per instance, so one healthy replica cannot dilute another replica's total failure and one replica's decision series cannot hide missing telemetry elsewhere. JevAdmissionRefusing divides refusals by the invocations that reached admission (every outcome but skipped), so an image-heavy workload cannot hide an exhausted lifetime cap; JevTargetDown speaks only for instances that had Jev enabled within the last hour - every gateway target, Jev or not, is covered by GatewayTargetDown and GatewayTargetAbsent in the gateway-availability group, so on a Jev instance both GatewayTargetDown (critical) and JevTargetDown (warning) fire; see Gateway target alerts for the relationship and an Alertmanager inhibit rule. The four rules keep four situations apart: JevDependencyFailing (this instance's admitted attempts mostly failing), JevAdmissionRefusing (this instance's own bounds refusing most of what asked), JevTargetDown (the gateway target is not being scraped, so every series from it is unobservable - not quiet), and JevInstrumentationInconsistent (decisions whose reason can only come from a classifier call, with no invocation counted). An enabled instance with no jev traffic is idle, a dashboard state, not an alert. An intentional skip, a valid abstention or a hint is never in an alert's numerator. What the expressions actually do - label matching, for, counter resets, staleness, a healthy replica beside a broken one - is proven by promtool unit tests in docker/grafana/alert-rules.test.yml (scripts/alert-rules-test.sh, which runs promtool from the pinned Prometheus image).

Offline evaluation assets (fake transport only)#

load-test/jev-eval.py with load-test/jev-eval-spec.v1.json is the versioned harness for the comparison above: current routing versus explicit hints versus Jev, on a labeled dev/held-out corpus with task and risk strata, reporting confusion, coverage and abstention, per-arm task success with Wilson intervals, cost and latency, and unknown-usage exposure. In this batch it runs only with a fake, deterministic classifier: every output is marked synthetic and evidence_class: synthetic-fake-transport, and nothing in it is evidence of model quality, cost savings or live qualification. Its limits are stated in the output as well: it does not call the gateway's Java routing implementation (it re-implements the ranking rule in Python), its baseline arm is first-candidate priority and not every deployed strategy, and downstream answer quality is read from the case file's synthetic outcomes, not measured. A live mode is specified in the spec (live_mode.required_controls: an operator-supplied credential reference, a maximum number of calls, a maximum input-token budget, a maximum spend, stop on exhaustion, and an explicit unknown-charge exposure report) and refuses to run until implemented and separately approved. The bundled jev-eval-cases.example.jsonl is a synthetic shape example, not a corpus; no private prompts are built or exported by the runner. The active sustained run and its evidence are unrelated to this feature and qualify nothing about it.

Research basis#

Reviewed 2026-09-23. TypeSafe's API documents typed choices, complete distributions and usage. Confidence is derived from the distribution, not a calibrated probability of task correctness. Model specifications provide current limits and pricing; verify your account before enabling. Known limitations make adversarial and mixed-task evaluation necessary. Vendor latency statements are not Pointsman measurements.