Three decision models walked into a Berkeley exam.

It started with one exam.

In my first experiment, I turned a Berkeley CS 70 final into a closed-choice test for Jev. It got 32 of 35 counting and probability items right. Then I expanded the experiment to fourteen exams and 996 adapted items.

That left an obvious question. Was I seeing something specific to Jev, or would other decision models do about as well?

I ran Laya and SemIf against the frozen question set. I kept the original Jev results. I did not rewrite difficult questions, move the correct answers, or teach either new system using the answer key.

The corpus comes from Berkeley’s public CS 70 exam archive. It covers discrete mathematics as well as probability: logic, graphs, modular arithmetic, proofs, counting, random variables and more. These scores are for my adaptation, not Berkeley’s exam grading scheme.

The same test, three different outcomes.

The counting and probability slice uses the same 380-item inclusion list as the earlier post. I did not select a new subset after seeing the comparison.

SemIf also got 42 questions right that Jev missed. Jev got 262 right that SemIf missed. That is more useful than treating one score as a statement that a model is better at everything. The paired answers show both the size of the lead and where it breaks down.

Laya’s default result is close to the roughly 30.1% expected from uniformly random choices under this grading scheme. That describes its performance here; it does not establish that its predictions are random, or that it cannot work well on other classification tasks.

A different question from JevBench.

JevBench evaluates decision models across a broader set of tasks. Its headline score combines intelligence, calibration, speed and cost, with each axis contributing 25% through a geometric mean. That is a useful answer to a deployment question: how attractive is the overall package?

This experiment asks something narrower: how often does the model choose the correct mathematical answer? I report accuracy directly. A cheaper or faster wrong answer does not increase that number.

I would use this alongside JevBench when evaluating a model, especially for decisions involving quantities, constraints or several linked conditions. It is not evidence that mathematics predicts every production workflow. A model that struggles here might still be excellent at routing support tickets.

The alternative is the lens: recognizable academic problems, fixed alternatives, and an answer-by-answer comparison. I would not call this a comprehensive replacement benchmark.

Recognizing a structure is easier than calculating with it.

The earlier Jev post suggested this pattern. All three systems scored higher on the recognition group than on the calculation group in the follow-up exams.

These are labels for the adapted tasks, assigned before the original inference run. They are not a view inside the models. The groups differ in subject matter and answer formats, and multi-step questions include supplied proofs. I would not infer a clean hierarchy of reasoning ability from the three rows.

For product work, the implication is practical: test the actual decisions you need. A familiar-looking interface and a strong aggregate benchmark do not tell you whether a model can reliably resolve the calculations in your workflow.

What I held fixed.

All systems received the same frozen mathematical text, shared context, alternatives and option order. Their native interfaces serialize that material differently. Each run used a single first pass, with no tools, retrieval, answer feedback or exam fine-tuning. Scoring happened separately from inference.

The 996 scored items required 1,000 independent judgments because some items combine checkbox decisions. Select-all questions require the exact set of answers; partial matches do not earn credit. The original final remains a discovery sample, separate from the thirteen-exam follow-up. Its context was organized differently, and every adapter preserves that distinction.

Jev: the preserved Jev 1.13.0 hosted runs. SemIf: its official MLX direct scorer using Qwen3.5-4B at original checkpoint precision, without quantization, generated reasoning or input truncation. Laya: the English checkpoint, convaiinnovations/laya, using SDK 0.3.4 defaults. These results describe those configurations, not every variant of each project.

Laya’s default input limits affected 66 scored items. I also ran a separately declared full-input condition preserving the text beyond those limits. It scored 308/996, or 30.9%, versus the default’s 311/996. Truncation did not explain the overall gap in this experiment. The full-input adapter is a separate condition, not the stock SDK behavior.

I am not ranking speed here. Jev ran through hosted section batches; SemIf and Laya ran locally with different runtimes. SemIf’s measured inference run took about 9 minutes 45 seconds, excluding model loading and an unrelated warmup. Putting that beside Jev’s batch HTTP timings as if they measured the same thing would obscure more than it explains.

Likewise, a native option probability is not automatically a calibrated probability of being correct. This comparison does not establish a cross-model confidence ranking.

The test can make models look smarter than they are.

The biggest qualification is that I had the official solutions when authoring the alternatives. A free-response question became a choice among plausible answers. That reduces the task, and it creates opportunities for cues in wording, answer length and distractor construction.

One result makes this concrete: Jev answered all 113 supplied-proof items correctly in the earlier analysis, but simply picking the longest option would have answered 90 of 113 correctly. Selecting a supplied proof is not the same as producing one. The raw totals retain these questions so the comparison stays fixed; they should not be read as evidence of perfect proof ability.

The exams are public, training exposure is unknown, and recurring concepts mean the 996 items are not independent samples of mathematical ability. The probability subset was defined descriptively after the original Jev run and then held fixed for the comparisons. There is no human comparison under the same adaptation and grading rules.

So I am comfortable saying Jev led these two tested configurations on this corpus. I am not comfortable saying it beat Berkeley students, solved unseen exams, or proved that one model architecture is fundamentally better.

The next version should try to break this result.

A stronger benchmark would add newly authored held-out problems, review distractors for length and wording cues, and measure sensitivity to option order. It would report those results separately from this frozen archive, rather than quietly improving the test after seeing model outputs.

For now, this is a useful second opinion. Jev’s lead survived two additional systems and thirteen follow-up exams. SemIf showed meaningful overlap and some distinct successes. Laya’s tested English configuration struggled. Those differences are exactly why I wanted to look beyond a single leaderboard.