# Ask-Jev experiment notes

## Method

Run September 23, 2026, from my laptop through Vercel AI Gateway. Each model receives 24 synthetic messages twice, with the same questions and rules. The questions concern department, explicit refund request, and explicit urgency. Labels were fixed before running.

Calls run sequentially with alternating model order. Timings include network time and end at a complete, validated response. There are no retries; each call has a 20-second timeout. The median averages the middle pair when needed. The p95 uses the nearest rank. Failed calls count against correctness; latency statistics cover successful calls.

Jev uses the native evaluation endpoint. Gemini 2.5 Flash-Lite uses AI SDK 7.0.107's structured-output evaluation adapter over the Gateway language endpoint. The adapter requests a department and two probability estimates. It requests no reasoning by default. Boolean estimates of 0.5 or higher count as yes.

An earlier setup run sent Gemini to the native evaluation endpoint and received HTTP 400. That run is excluded from this comparison and preserved separately. Gateway aliases can change; no resolved Jev version was returned.

There are 24 distinct examples, not 48 independent examples. This small synthetic set does not establish calibration or reliability on real support messages.

- [Questions and labels](triage-fixtures.json)
- [Every response and failure](triage.jsonl)
- [Summary](summary.json)
- [Earlier setup run](../jev-study/triage.jsonl)

Reported Gateway market cost: $0.001071 for Jev's 47 completed calls, plus unknown cost for its failed call; $0.002718 for Gemini's 48 calls. Market cost can differ from the amount charged after promotions.

## Probability training

Training can reward useful probabilities by comparing predictions with known answers. One standard rule squares the difference between the estimate and the outcome: 1 for yes, 0 for no.

Across ten cases with eight yeses:

- Predicting 80% each time gives an average penalty of 0.16.
- Predicting 99% gives 0.1961. The two wrong answers make the extra certainty expensive.

This is a [proper scoring rule](https://sites.stat.washington.edu/people/raftery/Research/PDF/Gneiting2007jasa.pdf). TypeSafe calls its method [Reinforcement Learning for Calibrated Decisions](https://typesafe.ai/blog/introducing-system-one-models-and-jev), but does not disclose its exact reward or training data. The example explains a general principle, not Jev's implementation.

Gemini's generated probability estimates can also be calibrated on a task. Neither the interface nor a training objective establishes calibration on new data. Measure it against representative labeled cases.
