Tuatara Decide vs Jev on BANKING77: a tie at a thousandth of the cost, then a win

TypeSafe's Jev is a new kind of model: instead of writing prose, it answers typed questions (pick one of these options, score this from 1 to 5, yes or no) with a probability attached. It is fast, cheap and well calibrated, and a lot of teams are wiring it into routing, triage and scoring jobs. We wanted to know two things: how a classifier built on our own stack compares on accuracy, and what each one costs to run at volume. So we ran it on the one public test where Jev's answers are published row by row.

Short version: on BANKING77, our blend of the Tuatara Vector Model and a classic text classifier scored 91.79%. Jev scored 92.40%. That is a statistical tie, not a win. The blend runs at about one-thousandth of the cost: roughly 14 cents of cloud CPU per million messages, against about $142 in Jev API charges for the same million.

Update, September 28, 2026: Tuatara Decide now beats Jev on the same test. We added an embedding head: an open sentence-embedding model running on our own hardware, with a classifier trained on the same labelled messages. It scored 94.09% against Jev's 92.40%, and message by message the gap is very unlikely to be chance (p = 0.00014). It costs more CPU than the blend, about $5.70 per million messages instead of 14 cents, which is still about 25 times less than Jev. Jev still leads when there are no training examples at all. The details are in the section on the embedding head below.

Benchmarking

BANKING77 is a public dataset from PolyAI: 13,083 real customer messages to a bank, each labelled with one of 77 intents such as card_arrival, lost_or_stolen_card or exchange_rate. It has an official split: 10,003 messages for training and 3,080 for testing.

On September 18, 2026, an independent researcher ran Jev on it and published everything: the protocol, written before the first paid call, the frozen configuration, and Jev's answer for every one of the 3,080 test messages. The run used the pinned model jev-1.13.0, all 77 intents as options in a single question, and up to 24 training examples retrieved for each message. Jev got 2,846 of 3,080 right.

That published answer file is what made a fair comparison possible. We did not have to call Jev at all. We scored our model on exactly the same 3,080 messages and compared answer by answer. Recomputing Jev's accuracy from the file gives 92.40%, the same figure the experiment reports.

Building

The model is a blend of two parts:

• The Tuatara Vector Model. We built a new version of Tuatara whose frame ids are intent classes instead of assets. The pipeline learns how strongly each word and word pair points at each intent, then trains those weights to separate the intents.
• A logistic-regression classifier. It uses word and character n-grams, the standard strong baseline for short-text classification.

The final answer is a weighted average of the two models' probabilities: 30% Tuatara, 70% logistic regression.

Tuatara is not new. It is based on work patented from Lawrence Berkeley National Laboratory: US Patent 7,987,191, "System and Method for Generating a Relationship Network". The patent is assigned to the University of California, with a priority date of June 2005, for work supported by the U.S. Department of Energy under contract DE-AC02-05CH11231. It describes building vectors that capture the relationships between terms from distance scores inside contextual frames. That is the same frame-and-association mechanism the model above uses. It predates most of the vector-embedding era, including Word2Vec, which arrived in 2013.

We held ourselves to the same discipline as the Jev experiment:

• Training data only. Both models learned only from the official 10,003 training messages.
• Selection on training data. The blend weight and Tuatara's one tuning setting were chosen on a 10% slice held out from the training set. They were never chosen on the test set.
• One test run. We retrained on the full training set and scored the 3,080 test messages once. We did not go back and tweak anything after seeing the test score.

Results

* Published reference, not rerun here.

Jev is ahead by 0.61 points, and the confidence intervals overlap. The sharper test is to compare the two models message by message:

• Our blend got 121 messages right that Jev got wrong.
• Jev got 140 messages right that our blend got wrong.

If the two models were equally good, a split at least that lopsided would happen by chance about 27% of the time (exact McNemar test, p = 0.27). So on this benchmark the two are indistinguishable. For the blend, we are not claiming to have beaten Jev, because it did not. The embedding head described further down did.

Cost

Accuracy is a tie. Cost is not. Jev is already the cheap option: TypeSafe's launch post cites 444.6 times cheaper than frontier models on its own workflow evaluations, and calls that the higher end of real-world gains. The blend is about a thousand times cheaper again.

The Jev experiment published its token counts. The 3,080 test messages used 10,417,536 input tokens, about 3,382 per message. That is large because every request carries all 77 intent options plus up to 24 retrieved examples.

The blend has no per-call charge. It is about 45 MB of weights that run inside your own process, so its cost is CPU time. We measured about 11 ms per message on one CPU core, and 0.2 ms for the Tuatara half alone. Costs below use Jev's list price of $0.042 per million input tokens (output free) and AWS's on-demand price for a compute-optimised c7i.large in US East, $0.08925 an hour for 2 vCPUs, about 4.5 cents per vCPU-hour.

Per message, Jev costs about $0.00014. Training the whole experiment, including both training runs and model selection, took under four minutes of wall time on one machine — a one-off cost of a fraction of a cent. That is roughly a 1,000-to-1 cost ratio at the same accuracy. At the volumes where classification actually runs, such as every support message, every headline or every chat line, it is the difference between a line item and a rounding error.

Points on Cost

• Jev's bill depends on the prompt. A request with fewer options or fewer examples would use fewer tokens and cost less. The 3,382 tokens here come from the method that scored best in the experiment's own selection, so they are the realistic cost of that accuracy.
• Our CPU figure is approximate. We measured on a busy production server, and a cloud vCPU is not identical to that core. Treat our number as an order of magnitude. Even if it were ten times too low, the gap would still be about 100 to 1.

Speed

The blend answers in about 11 ms. The Jev experiment reports 0.53 seconds of elapsed time per message, with four requests in flight. That figure includes the network round trip and example retrieval, so it is not Jev's raw model speed, but it is what a caller actually waits for. The blend has no network hop, no rate limit and no outage risk from a third party.

Needs

Our models learned from BANKING77's labelled training messages. Jev used those same messages only as retrieved examples in its prompt, and its weights never changed. That is the real trade. A specialist trained on your labels matches or beats a general model on your task at a fraction of the running cost. The general model needs no training and can answer a question it has never seen before. If you have labelled data and a fixed set of decisions that run at volume, the specialist wins on cost. If your questions change every week, a general model earns its fee.

Different mistakes

Between them, our blend and Jev disagree on 261 of the 3,080 messages, and each is right on about half of those. Two models of similar accuracy with different error patterns are exactly what a stronger ensemble is built from. The next experiment worth running is a blend that includes Jev itself, used where confidence is low. That needs API access, and we would welcome it.

A second opinion where the blend is unsure

We have since run that experiment with an open model instead of Jev. We used Laya, the 421-million-parameter model listed 42nd on JevBench below and the cheapest system there, running on our own CPUs through Ollaya. The rule is simple. The specialist answers every message. When its top probability is under a threshold, Laya is shown the message and the specialist's five most likely intents, and gives its own probabilities for those five. The two sets of probabilities are mixed, and the highest combined score wins. Laya never adds an intent outside the specialist's top five, and it never sees a message the specialist is sure about.

We tested this on BANKING77 and on two more public intent benchmarks, HWU64 (64 intents) and CLINC150 (150 intents). The threshold and the mixing weight were chosen on a held-out slice of each dataset's training data, and each test set was scored once. The specialist was retrained for this experiment, so its baselines differ slightly from the figures above and from our benchmark page.

The gain is real but uneven. On BANKING77 and HWU64 alone it could be chance, and most of the evidence comes from CLINC150. On the messages the specialist was unsure about, Laya alone was less accurate than the specialist on BANKING77 and HWU64 and more accurate on CLINC150. We do not know what Laya was trained on. If public intent datasets were part of it, the CLINC150 result is flattered.

Tuatara Decide now does this by default. One setting has to serve every customer, so we chose a threshold of 0.7 and a weight of 0.3 after these runs. It improved the held-out slice of all three datasets. Because we chose it after seeing the test results, its test scores are optimistic: 91.82%, 88.66% and 94.33%, with 143 answers fixed and 62 broken. About one message in five gets a second opinion, which adds about half a second to that message. Every other message is answered in milliseconds, as before. Each answer says whether it had a second opinion, and a caller can switch it off per request.

The second opinion costs more CPU. Even if each Laya call used four vCPUs for its full half second, it would come to about $5 per million messages at the AWS price above. That is well above the specialist's 14 cents, and still about 28 times below Jev's $142.

An embedding head beats Jev

The blend reads words. It learns that "card" and "arrive" point at card_arrival because it saw them together in training. A message that says the same thing in different words can slip past it. So we added a third part that reads meaning. An open sentence-embedding model, BAAI's bge-large-en-v1.5, turns each message into a list of 1,024 numbers, and a logistic regression trained on the labelled examples picks the intent from those numbers. The embedding model runs on our own hardware, so no message goes to an outside API.

The final answer mixes the blend and the new head. The mixing weight is chosen the same way as before, on the 10% slice held out from the training set. On BANKING77 the best weight on that slice was 0.5, which scored 93.72%. The head alone scored 93.32%. That difference is within the noise of a slice of that size. The rule we ship takes the weight that leans most on the head among those within one standard error of the best, and here that is the head alone. So the BANKING77 answers below come from the embedding head. The Tuatara blend is still part of every classifier: it answers when the embedding server is down or slow, and it gets weight wherever it helps on a customer's own held-out examples.

Message by message, the head got 117 messages right that Jev got wrong, and Jev got 65 right that the head got wrong. If the two were equally good, a split this lopsided would happen by chance about once in 7,000 tries (exact McNemar test, p = 0.00014). The same change, measured the same way, raised HWU64 from 87.73% to 92.01% and CLINC150 from 93.00% to 96.76%.

One caveat about how clean this was. We tried two rules for choosing the weight, and both were scored on the test set. The first simply took the best weight on the held-out slice, 0.5, and scored 94.68%. We ship the second rule because on classifiers with only a few examples per intent the held-out slice is too small to trust a single best score. We report the result of the rule that ships, which is the lower of the two.

We then checked it through the public API, the way a new customer would use it. We opened a new account, trained a classifier on 20 BANKING77 intents with 10 training examples each, and sent it 800 test messages it had never seen. With the embedding head it got 93.50% right. The same setup without the head got 84.25%. Both runs include the second opinion described above. The 94.09% on the full test set is the classifier alone; we have not yet measured the second opinion on top of the head.

Cost of the head

The head costs more to run than the blend, because the embedding model is much larger. We measured 0.46 CPU-seconds per message on our embedding server, an AMD EPYC 7713 shared with other work, running the model at full precision. At the same AWS price as above, about 4.5 cents per vCPU-hour, that is:

• Per million messages: about 128 vCPU-hours, or about $5.70.
• Compared with the blend: about 40 times its 14 cents.
• Compared with Jev: about 25 times less than its $142.

One message takes about 0.2 seconds to embed. Through the public API, a batch of 50 took about a quarter of a second per message. That is much slower than the blend's 11 ms, and still faster than the 0.53 seconds per message the Jev experiment reports. As with the blend, treat the CPU figure as an order of magnitude, not a quote.

Where Jev still leads

Jev is ahead when there are no training examples at all. Tuatara Decide has a mode for that, Decide Zero, which picks among options from their names and descriptions alone, using the same embedding model. The Jev experiment published its answers for 154 screening messages given only the intent definitions, in https://github.com/simonmesmith/jev-banking77-experiment/tree/main/runs/screen-definitions. Given the same definitions, Jev got 122 right (79.22%) and Decide Zero 118 (76.62%). Decide Zero was right on 9 messages where Jev was wrong, and wrong on 13 where Jev was right (p = 0.52). That is too few messages to call, but Jev is ahead.

With only a few labelled examples, the head scores 76.9% at 3 examples per intent, 84.3% at 5 and 88.8% at 10. Jev's 92.40% used retrieval over all 10,003 training messages, so this is not a like-for-like comparison. Still, a team with a handful of examples should not expect to beat Jev yet. The win above comes from training on a full labelled set.

The JevBench field

BANKING77 is one task. The wider scoreboard for Jev-class models is JevBench, run by Benchmark Heaven. It gives every system the same typed decisions, among them intent classification, request routing, policy checks, answer judging, severity scoring and extraction. It runs each system itself and scores four things with equal weight: accuracy, calibration, speed and cost. Its current release, v1.4.2.1 of 27 September 2026, ranks 90 systems.

Three things stand out. Jev is still near the top: the open models that edge past it on the overall score get there mostly on speed and cost, and of the systems above only Plumb-4B answers more of the sealed questions correctly. Every system drops sharply from public to sealed questions, so public accuracy flatters everyone. And the cheapest system listed, Laya, costs about $0.003 per 1,000 decisions.

We have not run our blend on JevBench, and as built it could not enter. JevBench asks a general model new questions with a rubric, while our blend is a specialist that answers only the 77 intents it was trained on. That is the trade described under Needs above. What the two sets of results do share is a cost unit. On BANKING77 the blend costs about $0.00014 per 1,000 messages. That is about 20 times less than Laya, the cheapest system on JevBench, and about 285 times less than Jev's JevBench cost of $0.040 per 1,000. Jev costs more per message on BANKING77, about $0.14 per 1,000, because every request there carries 77 options and retrieved examples. The same caveat applies as before: our CPU figure is an order of magnitude, not a quote.

So the two approaches split the work. General Jev-class models are the right tool when the questions keep changing. When the decisions are known in advance and run at volume, as with most production classification, a trained specialist costs far less for the same accuracy.

The Data

Every input here is public. The dataset and its official split are on PolyAI's GitHub. Jev's per-message answers are at https://github.com/simonmesmith/jev-banking77-experiment/blob/main/results/predictions.csv. The experiment's protocol is at https://github.com/simonmesmith/jev-banking77-experiment/blob/main/PROTOCOL.md and its report is at https://github.com/simonmesmith/jev-banking77-experiment/blob/main/results/REPORT.md. Our blend's answer for each of the 3,080 test messages is at https://cymetica.com/static/research/jev-banking77/blend_predictions.csv. A short scoring script at https://cymetica.com/static/research/jev-banking77/score.py downloads that file, Jev's answers and the official test labels, checks that every row lines up, and prints both accuracies and the McNemar test. The embedding head's answer for each test message is at https://cymetica.com/static/research/jev-banking77/head_predictions.csv, and the same script scores it too. It needs only Python, and running python3 score.py reproduces 91.79%, 92.40% and p = 0.27 for the blend, and 94.09% and p = 0.00014 for the head, in under a minute. The JevBench figures are from its published results file at https://github.com/fstandhartinger/jevbench/blob/main/results/v1.4.2.1/jevbench-v1.4.2.1-results.json. The second-opinion runs, with every test message, the answers under both rules and the scripts, are at https://cymetica.com/static/research/decide-second-opinion/summary.json. The BERT reference is Table 3 of the original BANKING77 paper. Any classifier can be scored against the same answer file in a few minutes.

Credit where it is due: the Jev run we compared against is careful work. The protocol was written before the first call, the method was frozen before the test, and every answer was published. That is what made this comparison honest, and it is a good template for anyone publishing model results.