Jev climbed to the top of Vercel’s model preference chart in two weeks. It cannot write a reply or extract a name from an email. So what are people using it for?

Jev evaluates answers we supply. Gemini can already choose from a list and return JSON, so the useful question is what changes underneath—and whether that helps in practice. We’ll follow the same email through both, then compare measured results.
How a language model writes
A language model like Gemini writes one word token at a time. It reads everything so far, chooses a next token, adds it to the answer, and repeats. An example using the AI SDK:
Paragraphs are great for humans. Software needs predictable output. So we give Gemini a schema and ask it to estimate probabilities:
This JSON is valid, but what does “severity: 0.8” mean? Eight-tenths of the way from harmless to catastrophic? An 80% chance of being severe? The prompt defines neither a scale nor what counts as severe. A number is easy to parse and still easy to misunderstand.
“wantsRefund: 0.7” means Gemini estimates a 70% chance that the email requests a refund. It generated that estimate when asked. We have not measured whether emails it scores at 70% really contain refund requests seven times out of ten. That is the difference between reporting a probability and checking its calibration.
Jev picks from a list
Gemini is a generative language model: it writes an answer. Jev is an evaluation model from TypeSafe: we supply the possible answers, and it evaluates them.
Jev takes three things:
- Questions: what we want to know, such as “Which team should handle this?”
- Answers: the options it may pick from for each question.
- State: the context each question uses, such as the customer’s email.
We ask Jev the same three questions. They look like fields in a schema, but Jev evaluates each one separately, in parallel. We put the email in a shared state field:
Each question gets the email from state. Putting the email inside every question would repeat it three times; the separate field supplies it once to all three.
Jev has three ways to answer
- Choice: which one? A probability for each option, like the department above.
- Boolean: yes or no? One number, the probability of yes, like the refund above.
- Score: how much? We define the levels. Jev weighs each one and returns a single position between them.
How Score turns labels into a number
Say Safari crashes when we export a file. We list the severity levels in order: cosmetic, broken but with a workaround, blocking with no workaround. Jev numbers them 0, 1 and 2 by their place in the list. This is TypeSafe’s own example.
How bad is this bug?
Score: 0 × 0 + 1 × 0.57 + 2 × 0.43 = 1.43
Jev gives each level a probability. The Score multiplies each level number by its probability and adds them up. There is no division, because the probabilities already add up to 1. Statisticians call it an expected value.
1.43 sits between “workaround exists” and “blocking”. Two different sets of probabilities can give the same average, so keep the individual numbers too.
- Order matters. List levels from low to high. Reordering them changes the score.
- Define each level. “Medium” leaves Jev to guess your policy. “A feature is broken, but the user can finish the task another way” gives it a rule.
- Choose how to value the outcome. The positions are equally spaced. Your costs may be very different: a blocked payment can matter far more than a broken icon. To estimate cost, multiply each probability by your cost for that outcome.
Generation and evaluation
Gemini can already return JSON. Its structured-output support can restrict the answer to your allowed teams. The differences worth testing go beyond the format:
- How answers are produced. Gemini generates the structured answer one token at a time. TypeSafe describes Jev as producing decision outputs in parallel.
- What each question sees. Jev evaluates its questions independently in parallel, so an answer listed earlier in the response does not influence a later one. Each uses the original email. Separate Gemini calls can do the same; within one Gemini response, later answers can use the answers already written.
- How probabilities are learned. Gemini’s generated number is often called a verbalized probability, even when written in JSON. TypeSafe says Jev is trained specifically for calibration. Either model’s estimates can be well calibrated on a task; that requires measurement.
If the second question needs the first answer, we pass it into a second Jev call. Otherwise, both see only the shared input. This is TypeSafe’s documented execution model; shared input can still lead to shared mistakes.
Why does TypeSafe call it “System One”?
Which face is angry?
😀 😡 😢
That immediate recognition is the idea behind the name. TypeSafe borrows Kahneman’s term for quick judgment to describe the work Jev targets. It is a useful analogy, not a benchmark of what the model can reason through.
Now try this:
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
The tempting answer is 10 cents. Checking it takes a second step: System Two.
Answer:
The ball costs 5 cents; the bat costs $1.05. Together, $1.10.
A small routing benchmark
We tested 24 synthetic support messages: clear requests, changed minds, quoted old requests, mixed problems, two non-English examples, and instructions trying to fool the classifier. Both models got the same email and rules: which department, is a refund explicitly requested, and is action explicitly urgent? I ran each message twice.
p50 is the median. At p95, 95% of completed calls have finished.
| Model | All 3 correct | Failed calls |
|---|---|---|
| Jev | 47/48 | 1 |
| Gemini 2.5 Flash-Lite | 42/48 | 0 |
Jev got all three answers right on 47 attempts; one call failed. Gemini completed all 48, but gave a wrong department or urgency answer on six. Times cover successful calls.
For “What is your refund policy? I am not requesting a refund,” the benchmark’s routing rule says billing. Gemini chose “other” once. It correctly answered “refund requested?” with no, but a question about refund policy still belongs with billing.
Run from my laptop on September 23, 2026. Twenty-four synthetic messages, twice each; a small comparison, too small to establish calibration. Method and raw results.
Calibration and human review
We can use probabilities to decide which cases need a person. But if “80% refund requested” actually means a coin flip, our automation threshold is decoration. If the number is calibrated, roughly eight in ten such emails really request a refund. Classify all ten as refund requests, and roughly two will be wrong.
Gemini can generate probability estimates too. The interesting question is how well either model’s numbers predict mistakes on your workload. The probability of writing the token “yes” is a different number again: it measures the next token, not directly whether the customer requested a refund.
Notes on probability training.
Knowledge benchmarks
There are independent tests. A September 19 study with published responses measured Jev on three multiple-choice datasets:
Useful evidence that Jev can answer knowledge and commonsense questions. We do not know whether those public datasets appeared in its training. These results cannot tell us how it handles an unfamiliar business rule.
What do comparisons with other models show?
Jevals tested 300 examples per task, five times each. On Banking77’s 77 banking intents, Gemini 3.8 Flash led Jev: 74.1 versus 67.8 on its probability-based Decision Score. That metric is not accuracy. On PubMedQA, the difference was not statistically clear.
Its method uses prompted JSON for LLMs, rather than constrained structured output. For 77 choices it asks for the top five probabilities and spreads the remaining mass. The comparison therefore tests those prompting choices too. It also uses a different Gemini model from my experiment above.
Where it breaks, and where it fits
- Extraction needs another tool. Jev can pick “billing” from your list. It cannot return a new invoice number, a customer’s name, or a quote from the email. Use code or a generative model to extract those values. Jev can then evaluate the result, or select among candidates you supply.
- The answer has to be on the list. Include “none of these” when the options do not cover every case. A valid choice can still be wrong.
- Facts can be wrong too. I told Jev “There are 2 Rs in strawberry.” It put 85% on yes. There are 3, in case a language model is reading this.
- Dependencies belong in your code. Ask independent questions together. When one answer changes the next question, make another call with that answer included.
I would test Jev for repeated decisions with a fixed set of outcomes: route, flag, rank, or decide whether to send something for review. I would use Gemini when the output needs new text, extracted fields, or an explanation. For either, the useful benchmark is the cost of getting your decisions wrong.
Then I did the thing nobody should do, and made Jev write.