Matching Jev on BANKING77 at a thousandth of the cost

TypeSafe's Jev is a new kind of model: instead of writing prose, it answers typed questions (pick one of these options, score this from 1 to 5, yes or no) with a probability attached. It is fast, cheap and well calibrated, and a lot of teams are wiring it into routing, triage and scoring jobs. We wanted to know two things: how a classifier built on our own stack compares on accuracy, and what each one costs to run at volume. So we ran it on the one public test where Jev's answers are published row by row.

Short version: on BANKING77, our blend of the Tuatara Vector Model and a classic text classifier scored 91.79%. Jev scored 92.40%. That is a statistical tie, not a win. The blend runs at about one-thousandth of the cost: roughly 14 cents of cloud CPU per million messages, against about $142 in Jev API charges for the same million.

Benchmarking

BANKING77 is a public dataset from PolyAI: 13,083 real customer messages to a bank, each labelled with one of 77 intents such as card_arrival, lost_or_stolen_card or exchange_rate. It has an official split: 10,003 messages for training and 3,080 for testing.

On September 18, 2026, an independent researcher ran Jev on it and published everything: the protocol, written before the first paid call, the frozen configuration, and Jev's answer for every one of the 3,080 test messages. The run used the pinned model jev-1.13.0, all 77 intents as options in a single question, and up to 24 training examples retrieved for each message. Jev got 2,846 of 3,080 right.

That published answer file is what made a fair comparison possible. We did not have to call Jev at all. We scored our model on exactly the same 3,080 messages and compared answer by answer. Recomputing Jev's accuracy from the file gives 92.40%, the same figure the experiment reports.

Building

The model is a blend of two parts:

• The Tuatara Vector Model. We built a new version of Tuatara whose frame ids are intent classes instead of assets. The pipeline learns how strongly each word and word pair points at each intent, then trains those weights to separate the intents.
• A logistic-regression classifier. It uses word and character n-grams, the standard strong baseline for short-text classification.

The final answer is a weighted average of the two models' probabilities: 30% Tuatara, 70% logistic regression.

Tuatara is not new. It is based on work patented from Lawrence Berkeley National Laboratory: US Patent 7,987,191, "System and Method for Generating a Relationship Network". The patent is assigned to the University of California, with a priority date of June 2005, for work supported by the U.S. Department of Energy under contract DE-AC02-05CH11231. It describes building vectors that capture the relationships between terms from distance scores inside contextual frames. That is the same frame-and-association mechanism the model above uses. It predates most of the vector-embedding era, including Word2Vec, which arrived in 2013.

We held ourselves to the same discipline as the Jev experiment:

• Training data only. Both models learned only from the official 10,003 training messages.
• Selection on training data. The blend weight and Tuatara's one tuning setting were chosen on a 10% slice held out from the training set. They were never chosen on the test set.
• One test run. We retrained on the full training set and scored the 3,080 test messages once. We did not go back and tweak anything after seeing the test score.

Results

• Jev (published): 2,846 correct, 92.40%, 95% interval 91.41–93.29%, macro-F1 0.9235.
• Our blend: 2,827 correct, 91.79%, 95% interval 90.76–92.70%, macro-F1 0.9180.
• Logistic regression alone: 2,819 correct, 91.53%.
• Tuatara alone: 2,656 correct, 86.23%.
• Fine-tuned BERT, from the original paper: 93.66%. This is a published reference, not rerun here.

Jev is ahead by 0.61 points, and the confidence intervals overlap. The sharper test is to compare the two models message by message:

• Our blend got 121 messages right that Jev got wrong.
• Jev got 140 messages right that our blend got wrong.

If the two models were equally good, a split at least that lopsided would happen by chance about 27% of the time (exact McNemar test, p = 0.27). So on this benchmark the two are indistinguishable. We are not claiming to have beaten Jev, because we did not.

Cost

Accuracy is a tie. Cost is not. Jev is already the cheap option: TypeSafe's launch post cites 444.6 times cheaper than frontier models on its own workflow evaluations, and calls that the higher end of real-world gains. The blend is about a thousand times cheaper again.

The Jev experiment published its token counts. The 3,080 test messages used 10,417,536 input tokens, about 3,382 per message. That is large because every request carries all 77 intent options plus up to 24 retrieved examples. At Jev's list price of $0.042 per million input tokens, with output free, that works out to:

• Per message: about $0.00014.
• Per thousand messages: about $0.14.
• Per million messages: about $142.
• Per hundred million messages: about $14,200.

The blend has no per-call charge. It is about 45 MB of weights that run inside your own process, so its cost is CPU time. We measured about 11 ms per message on one CPU core, and 0.2 ms for the Tuatara half alone. To put that in dollars, we used AWS's on-demand price for a compute-optimised c7i.large in US East, $0.08925 an hour for 2 vCPUs, or about 4.5 cents per vCPU-hour:

• Per million messages: about 3.1 core-hours, or about $0.14.
• Per hundred million messages: about $14.
• Tuatara alone, per hundred million messages: about $0.24.
• Training: the whole experiment, including both training runs and model selection, took under four minutes of wall time on one machine. That is a one-off cost of a fraction of a cent.

That is roughly a 1,000-to-1 cost ratio at the same accuracy. At the volumes where classification actually runs, such as every support message, every headline or every chat line, it is the difference between a line item and a rounding error.

Points on Cost

• Jev's bill depends on the prompt. A request with fewer options or fewer examples would use fewer tokens and cost less. The 3,382 tokens here come from the method that scored best in the experiment's own selection, so they are the realistic cost of that accuracy.
• Our CPU figure is approximate. We measured on a busy production server, and a cloud vCPU is not identical to that core. Treat our number as an order of magnitude. Even if it were ten times too low, the gap would still be about 100 to 1.

Speed

The blend answers in about 11 ms. The Jev experiment reports 0.53 seconds of elapsed time per message, with four requests in flight. That figure includes the network round trip and example retrieval, so it is not Jev's raw model speed, but it is what a caller actually waits for. The blend has no network hop, no rate limit and no outage risk from a third party.

Needs

Our models learned from BANKING77's labelled training messages. Jev used those same messages only as retrieved examples in its prompt, and its weights never changed. That is the real trade. A specialist trained on your labels matches a general model on your task at a fraction of the running cost. The general model needs no training and can answer a question it has never seen before. If you have labelled data and a fixed set of decisions that run at volume, the specialist wins on cost. If your questions change every week, a general model earns its fee.

Different mistakes

Between them, our blend and Jev disagree on 261 of the 3,080 messages, and each is right on about half of those. Two models of similar accuracy with different error patterns are exactly what a stronger ensemble is built from. The next experiment worth running is a blend that includes Jev itself, used where confidence is low. That needs API access, and we would welcome it.

The JevBench field

BANKING77 is one task. The wider scoreboard for Jev-class models is JevBench, run by Benchmark Heaven. It gives every system the same typed decisions, among them intent classification, request routing, policy checks, answer judging, severity scoring and extraction. It runs each system itself and scores four things with equal weight: accuracy, calibration, speed and cost. Its current release, v1.4.2.1 of 27 September 2026, ranks 90 systems. Here are the top six and two smaller models, as published. Accuracy is shown on public questions, then on sealed questions that no submitter has seen; cost is per 1,000 decisions.

• 1. Plumb-4B: score 65.84, accuracy 89.6% / 38.0%, $0.030.
• 2. decider-4b v2: score 64.13, accuracy 83.5% / 34.7%, $0.020.
• 3. Jev 1.13.0: score 63.29, accuracy 86.6% / 36.7%, $0.040.
• 4. JevK5 v0.2.0: score 62.04, accuracy 85.3% / 33.1%, $0.022.
• 5. Cygnet (12B): score 61.76, accuracy 87.9% / 33.8%, $0.037.
• 6. Hopper: score 59.43, accuracy 82.3% / 34.1%, $0.024.
• 29. kev 4B: score 36.14, accuracy 66.2% / 22.4%, $0.019.
• 42. Laya (421M): score 30.25, accuracy 58.4% / 30.8%, $0.003.

Three things stand out. Jev is still near the top: the open models that edge past it on the overall score get there mostly on speed and cost, and of the systems above only Plumb-4B answers more of the sealed questions correctly. Every system drops sharply from public to sealed questions, so public accuracy flatters everyone. And the cheapest system listed, Laya, costs about $0.003 per 1,000 decisions.

We have not run our blend on JevBench, and as built it could not enter. JevBench asks a general model new questions with a rubric, while our blend is a specialist that answers only the 77 intents it was trained on. That is the trade described under Needs above. What the two sets of results do share is a cost unit. On BANKING77 the blend costs about $0.00014 per 1,000 messages. That is about 20 times less than Laya, the cheapest system on JevBench, and about 285 times less than Jev's JevBench cost of $0.040 per 1,000. Jev costs more per message on BANKING77, about $0.14 per 1,000, because every request there carries 77 options and retrieved examples. The same caveat applies as before: our CPU figure is an order of magnitude, not a quote.

So the two approaches split the work. General Jev-class models are the right tool when the questions keep changing. When the decisions are known in advance and run at volume, as with most production classification, a trained specialist costs far less for the same accuracy.

The Data

Every input here is public. The dataset and its official split are on PolyAI's GitHub. Jev's per-message answers are at https://github.com/simonmesmith/jev-banking77-experiment/blob/main/results/predictions.csv. The experiment's protocol is at https://github.com/simonmesmith/jev-banking77-experiment/blob/main/PROTOCOL.md and its report is at https://github.com/simonmesmith/jev-banking77-experiment/blob/main/results/REPORT.md. Our blend's answer for each of the 3,080 test messages is at https://cymetica.com/static/research/jev-banking77/blend_predictions.csv. A short scoring script at https://cymetica.com/static/research/jev-banking77/score.py downloads that file, Jev's answers and the official test labels, checks that every row lines up, and prints both accuracies and the McNemar test. It needs only Python, and running python3 score.py reproduces 91.79%, 92.40% and p = 0.27 in under a minute. The JevBench figures are from its published results file at https://github.com/fstandhartinger/jevbench/blob/main/results/v1.4.2.1/jevbench-v1.4.2.1-results.json. The BERT reference is Table 3 of the original BANKING77 paper. Any classifier can be scored against the same answer file in a few minutes.

Credit where it is due: the Jev run we compared against is careful work. The protocol was written before the first call, the method was frozen before the test, and every answer was published. That is what made this comparison honest, and it is a good template for anyone publishing model results.