# Jev task routing (experimental, opt-in)

Jev can classify what a conversation asks for. Pointsman uses that label to apply
an operator-supplied model ranking within the candidates already permitted by
routing policy. This is task-aware selection, not evidence that the selected
model is optimal. It does not learn model quality, alter budget enforcement, or
replace the existing bandit strategies.

| Area | Status |
|---|---|
| Enablement | Experimental; disabled by default; no default group uses Jev |
| Disabled behavior | No Jev client or strategy bean is created |
| Implementation checks | Mocked protocol, transport, router, configuration, dashboard and alert checks |
| Live Jev inference, quality, cost and latency | Not measured or qualified by these checks |
| Production savings claim | Not established |

## Request flow

```mermaid
flowchart TD
    A[Resolve model group and existing candidate filters] --> B{Family preference or session affinity selects?}
    B -->|Yes| P[Existing provider path]
    B -->|No: group uses jev| C{Tenant explicitly allowed and multiple models eligible?}
    C -->|No| F[First eligible candidate]
    C -->|Yes| H{Validated task hint?}
    H -->|Yes| M[Apply configured task model ranking]
    H -->|No| I[Check complete text context, characters, messages and serialized bytes]
    I --> J[Immediate capacity, RPM, lifetime and circuit checks]
    J --> K[One bounded Jev call per request]
    K --> L[Validate version, schema, probabilities and confidence]
    L --> M
    I -->|Unsupported or too large| F
    J -->|Refused| F
    K -->|Error or deadline| F
    L -->|Unknown or low confidence| F
    M -->|Eligible match| P
    M -->|No eligible match| F
    F --> P
```

The normal provider admission and settlement path still runs. Jev cannot authorize
models, raise budgets or reopen unavailable deployments. Candidate ordering
within the same concrete model remains the existing order. Model mappings select
concrete model IDs; they do not introduce new candidates or choose credentials.
Family preference and session affinity keep their existing precedence.

The question vocabulary is `code`, `summarize`, `extract`, `reasoning`, `knowledge`,
`chat` and `unknown`. Question version is `task-v1`, with model version pinned
initially to `jev-1.13.0`. A validated `X-Task-Type` hint avoids the external call.

Accepted classification includes all supplied text messages, including system and developer messages. Eligibility is decided before strategy execution and retained on the request memo, so later tiers do not ask again.

| Conversation or request | Classifier behavior |
|---|---|
| Complete supported text within bounds | Eligible for classification |
| Oversized input | Skip with `input_limit`; never truncate |
| Images, tool calls/results or reasoning content | Skip with `unsupported_context` |
| No user turn | Skip with `no_context` |
| Declared tools, including `tool_choice: "none"` | Skip with `tools_declared`; task-v1 cannot represent the complete declared tool context |
| Valid explicit `X-Task-Type` | Apply the ranking without a remote call, including tool-bearing requests |

### Request deadline

`TIMEOUT_MS` bounds one exchange, including connection, headers and body, on the request thread. A request makes at most one exchange even when several routing tiers are considered. This is a per-exchange bound; the gateway does not supply a global end-to-end request deadline for it to inherit.

## Configure deliberately

Enabling exports accepted conversation text to the configured TypeSafe endpoint.
Obtain the necessary tenant permission and review the provider's current data
handling terms before enabling. No prompt, credential or remote error body is
logged by this integration. Requests are not retried or redirected.

1. Set `POINTSMAN_JEV_API_KEY` through your deployment's secret mechanism.
2. Set `POINTSMAN_JEV_ALLOWED_TENANTS` to explicit tenant IDs (comma separated).
   There is no wildcard; `_anon` is an explicit opt-in if authentication is disabled.
3. Provide evaluated task rankings and set `strategy: jev` on an existing group
   in your operator routing configuration. For example, merge this into that
   configuration, replacing the model IDs and group with your actual catalog:

```yaml
pointsman:
  jev:
    task-models:
      code: [your-code-model, your-general-model]
      summarize: [your-economy-model, your-general-model]
      reasoning: [your-analysis-model, your-general-model]
  routing:
    groups:
      your-existing-group:
        strategy: jev
        # Keep this group's existing candidate definitions here.
```

Check the existing `routing.yml` group schema when editing. Use an additional
Spring YAML configuration file or the existing configuration-management path;
this example is a fragment, not a complete runnable deployment.

4. Set `POINTSMAN_JEV_ENABLED=true` and restart. Missing credentials, tenant
   allowlist or mappings refuse startup. Properties are bound at startup; restart
   after changing Jev limits, key, allowlist or mappings.

With Jev enabled, abstention uses the first eligible candidate. With Jev disabled,
a group still naming `jev` uses the registry's existing default-strategy fallback
and warning, just like other disabled strategies. For deterministic rollback,
change the group's strategy to `priority` as well as disabling Jev.

## Explicit bounds and accounting

Every setting is listed in `docker/.env.example`, with placeholders in
`gateway/src/main/resources/routing.yml`. Defaults are initial operating choices,
not latency guarantees or calibrated quality thresholds.

| Setting (`POINTSMAN_JEV_` prefix) | Default | Scope |
| --- | --- | --- |
| ENABLED | false | External calls disabled |
| ENDPOINT | https://api.typesafe.ai/v1/systemone | HTTPS, no redirects |
| MODEL | jev-1.13.0 | Pinned model version |
| API_KEY / ALLOWED_TENANTS | empty | Explicit export opt-in |
| TIMEOUT_MS / CONNECT_TIMEOUT_MS | 750 / 250 | Whole HTTP exchange / connection |
| MAX_CONCURRENT | 4 | Immediate refusal, no admission queue |
| REQUESTS_PER_MINUTE | 600 | Rolling minute of admitted attempts |
| MAX_CALLS_PER_PROCESS | 10000 | Lifetime admitted attempts |
| BREAKER_FAILURES / BREAKER_OPEN_MS | 5 / 10000 | Consecutive failures / probe cooldown |
| MAX_INPUT_CHARACTERS / MAX_MESSAGES | 8000 / 32 | UTF-16 characters / complete messages |
| MAX_REQUEST_BYTES / MAX_RESPONSE_BYTES | 16384 / 16384 | Serialized request / streamed body |
| MINIMUM_CONFIDENCE | 0.8 | Provisional abstention threshold |
| PROBABILITY_TOLERANCE | 0.000001 | Distribution validation rounding |

Attempt/rate/concurrency limits and the circuit breaker are shared across tenants
inside one gateway process, reset on restart, and multiply with replicas. They are
not an account-wide budget. Allocate the account's current limits across replicas
and leave headroom for other callers. The finite lifetime allowance intentionally
falls back after exhaustion; increase it explicitly only after reviewing usage.

Jev calls are operator overhead and are NOT charged to the existing tenant model
usage ledger. Provider-reported input tokens are exported; timeouts or invalid
usage increment an unverified-usage counter. These counters are operational
telemetry, not durable billing. An uncertain call may still incur provider charges.
Use provider billing/account limits for financial reconciliation. Do not describe
this feature as enforcing an all-in tenant budget that includes classification.

The request-local memo retains both successful assessments and abstentions across
tier decisions. Separate requests are never memoized together. There is no shared
classification cache, distributed rate limit, shadow mode or model-quality learner
in this first slice.

## Metrics, reasons and what they count

Two populations, deliberately not one:

- `pointsman.jev.invocations{outcome,reason}` - one per entry to the classifier, whether
  or not it reached the wire. `outcome` is one of `classified`, `abstained` (a valid answer
  the policy declined: `unknown_task`, `low_confidence`), `skipped` (no call by shape or
  policy: `no_context`, `unsupported_context`, `tools_declared`, `input_limit`,
  `serialization`), `refused` (this process's bounds: `capacity`, `rate_limit`,
  `call_limit`, `circuit_open`) or `failed` (the dependency: `timeout`, `connect_failed`,
  `interrupted`, `upstream_rate_limit`, `upstream_overloaded`, `upstream_auth`,
  `upstream_invalid_request`, `upstream_error`, `response_limit`, `invalid_usage`,
  `bad_protocol`, `transport_failed`).
- `pointsman.jev.decisions{reason}` - one per routing decision, so it can exceed
  invocations: a request's memoised assessment serves every tier it is decided for, and the
  strategy-only reasons (`disabled`, `tenant_not_allowed`, `single_model`, `invalid_hint`,
  `no_task_match`, `hint_no_task_match`, `selection_error`, `hint_<task>`, `classified_<task>`)
  appear only here. `no_task_match` follows a classification (an invocation happened);
  `hint_no_task_match` follows a hint (none did) - the telemetry-consistency alert relies on
  that distinction.
- `pointsman.jev.selection{effect}` - did the decision move? `ranked` when a task ranking
  chose a candidate `priority` would not have; `first` for every fallback and for a ranking
  whose top eligible model was already first. Read this to know whether the feature is doing
  anything.
- `pointsman.jev.calls` (admitted attempts), `.duration` (the whole exchange), `.input.tokens`
  (provider-reported), `.usage.unverified` (attempts whose usage was not reported validly -
  each may still have been charged), `.confidence` (every valid answer's, including those
  below the threshold), and the gauge `pointsman.jev.enabled` (1 while the strategy bean is
  wired, so an alert can tell enabled-and-silent from off).

Every reason is a member of one closed vocabulary (`JevReason`); no tenant, prompt, request id
or remote error string becomes a tag. Decision provenance carries the question version and
the reason (`jev_task-v1_<reason>`); the structured decision event carries the pinned model
version, the question version and the ranking digest (`pointsman.jev.ranking-version` on the
admin effective-config read) so a decision can be read back to the policy it was made under.
Classification confidence is a summary of a distribution, not a probability of being right,
and never a bandit reward.

**Migration from the first slice (2026-09-23).** Existing metric names are unchanged. Three
labels changed meaning: `invalid_or_failed_response` split into `bad_protocol` (the body
violated the pinned contract) and `transport_failed` (an unclassified transport error);
`timeout` now also covers the HTTP client's own deadline, which previously surfaced as
`invalid_or_failed_response` because it arrived wrapped; and a request that declares tools now
reports `tools_declared` instead of `no_context`. Dashboards keyed on the old labels must
follow. The Grafana board `Pointsman - Jev task routing` is provisioned from `docker/grafana/`.

### Configure alerts

 The `jev-classifier` group in `docker/grafana/alert-rules.yml` is RENDERED from
`docker/grafana/alert-policy.env` by `scripts/render-alert-rules.sh` - the window, the minimum
event volume, the two ratios and the `for` durations live in that one file, edited at deployment
time and never in Java; `JevAlertRulesRenderedTest` refuses a committed block that was not
rendered from the committed policy. Every rule is evaluated **per instance**, so one healthy
replica cannot dilute another replica's total failure and one replica's decision series cannot
hide missing telemetry elsewhere. `JevAdmissionRefusing` divides refusals by the invocations
that reached admission (every outcome but `skipped`), so an image-heavy workload cannot hide an
exhausted lifetime cap; `JevTargetDown` speaks only for instances that had Jev enabled within
the last hour - every gateway target, Jev or not, is covered by `GatewayTargetDown` and
`GatewayTargetAbsent` in the `gateway-availability` group, so on a Jev instance both
`GatewayTargetDown` (critical) and `JevTargetDown` (warning) fire; see
[Gateway target alerts](../operating-notes.md#gateway-target-alerts) for the relationship and an
Alertmanager inhibit rule. The four rules keep four situations apart: `JevDependencyFailing`
(this instance's admitted attempts mostly failing), `JevAdmissionRefusing` (this instance's own
bounds refusing most of what asked), `JevTargetDown` (the gateway target is not being scraped, so
every series from it is unobservable - not quiet), and `JevInstrumentationInconsistent` (decisions
whose reason can only come from a classifier call, with no invocation counted). An enabled instance
with no jev traffic is *idle*, a dashboard state, not an alert. An intentional skip, a valid
abstention or a hint is never in an alert's numerator. What the expressions actually do - label
matching, `for`, counter resets, staleness, a healthy replica beside a broken one - is proven by
promtool unit tests in `docker/grafana/alert-rules.test.yml` (`scripts/alert-rules-test.sh`, which
runs promtool from the pinned Prometheus image).

## Offline evaluation assets (fake transport only)

`load-test/jev-eval.py` with `load-test/jev-eval-spec.v1.json` is the versioned harness for
the comparison above: current routing versus explicit hints versus Jev, on a labeled
dev/held-out corpus with task and risk strata, reporting confusion, coverage and abstention,
per-arm task success with Wilson intervals, cost and latency, and unknown-usage exposure. In
this batch it runs **only** with a fake, deterministic classifier: every output is marked
`synthetic` and `evidence_class: synthetic-fake-transport`, and nothing in it is evidence of
model quality, cost savings or live qualification. Its limits are stated in the output as well:
it does not call the gateway's Java routing implementation (it re-implements the ranking rule in
Python), its baseline arm is first-candidate `priority` and not every deployed strategy, and
downstream answer quality is read from the case file's synthetic outcomes, not measured. A live mode is specified in the spec
(`live_mode.required_controls`: an operator-supplied credential reference, a maximum number
of calls, a maximum input-token budget, a maximum spend, stop on exhaustion, and an explicit
unknown-charge exposure report) and **refuses to run** until implemented and separately
approved. The bundled `jev-eval-cases.example.jsonl` is a synthetic shape example, not a
corpus; no private prompts are built or exported by the runner. The active sustained run and
its evidence are unrelated to this feature and qualify nothing about it.

## Research basis

Reviewed 2026-09-23. [TypeSafe's API](https://docs.typesafe.ai/api) documents typed
choices, complete distributions and usage. [Confidence](https://docs.typesafe.ai/confidence)
is derived from the distribution, not a calibrated probability of task correctness.
[Model specifications](https://docs.typesafe.ai/models) provide current limits and
pricing; verify your account before enabling. [Known limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13)
make adversarial and mixed-task evaluation necessary. Vendor latency statements
are not Pointsman measurements.