I gave Jev a Berkeley freshman probability exam.

Recognizing the structure is the strong part.

Jev is remarkably good at picking out familiar mathematical structure. Its answers become less reliable when the problem requires arithmetic, tracking several constraints, or carrying an intermediate result into the next step. That was my impression after giving it one CS 70 final. I wanted to know whether the pattern survived a much larger test.

So I took every exam linked from Berkeley’s CS 70 previous-exams page: seven midterms and seven finals, spanning Spring 2023 through Fall 2025. This is discrete mathematics and probability: logic, graphs, stable matching, modular arithmetic, polynomials, counting, random variables and Markov chains.

The original Fall 2025 final stays the discovery sample: 52/58 correct. The other thirteen exams became a follow-up of 938 adapted questions. Jev answered 782 correctly, or 83.4%, in a measured 71.8 seconds. Across the two runs, that is 834/996. I keep the runs separate in the analysis because the first result motivated the hypothesis and the input organization changed.

The follow-up supports the pattern, with a substantial warning about proof questions.

“Recognition” means selecting a definition, theorem, property or symbolic form without calculating an intermediate result. “Calculation” means evaluating a quantity with one formula or transformation. “Multi-step” means linked intermediate results, algorithm execution, conditional analysis or a proof argument. I assigned these labels before inference. They describe the task as I adapted it, not what happened inside the model.

Some stronger topic slices were distributions (32/33), countability (22/23) and logic (65/71). Some weaker slices were Markov chains (16/25), RSA (9/14) and Bayes problems (7/12). These are small, uneven groups with recurring concepts, so I would not turn them into a universal ranking of Jev’s abilities.

What if we only count counting and probability?

Across both runs, Jev got 322/380 correct: 84.7% on counting and probability-related questions. The original final contributed 32/35 (91.4%); the thirteen-exam follow-up contributed 290/345 (84.1%).

This includes combinatorics, random variables, expectation and variance, probability bounds, Markov chains, and their associated proof questions and counting recurrence. It excludes countability and unrelated discrete mathematics. I calculated this descriptive subset after the run, using the existing subject tags and explicitly adding the related proof items that had been tagged by response type. No questions were selected based on whether Jev got them right. The exact inclusion list is available.

The narrower subject scope changes the score only modestly. The broader distinction between recognizing a familiar form and working through a calculation remains more useful than calling this a pure probability test.

It started with one final.

The first experiment used the Fall 2025 final: 52/58 adapted items correct (89.7%), in 6.179 seconds across sixteen sequential requests. The median section batch took 393 ms. It had no request failures or retries.

The six misses were an RSA decryption exponent, a modular-power calculation, a finite-field polynomial, a conditional probability, a recursive expectation and an expected-distance calculation. Those mistakes prompted the recognition-versus-calculation hypothesis tested in the larger follow-up.

Thirty of the original Choice answers had API confidence at least 0.9; all thirty were correct. The larger run is more informative: it includes three high-confidence errors. The original exam also has two native select-all items, scored using a fixed Noul threshold of 0.5 and exact set matching; those do not enter Choice confidence bins.

The original run’s shared context included sibling questions. The follow-up isolates each question, as explained below. I preserved the original results instead of rerunning them under the new protocol. Its secondary adapted point score was 129/154, using an explicitly invented equal allocation for one section; this article uses item accuracy throughout because it is clearer and does not imply an original student grade.

Original side-by-side PDF · Original evidence. All 58 answers also appear in the combined explorer below.

The proof result needs a warning label.

Jev got 113/113 supplied-proof questions correct. That sounds like the most impressive result here. It is also the result I trust least as a measure of mathematical reasoning.

A multiple-choice proof question asks the model to recognize a completed argument. It does not require constructing that argument. And my alternatives contain an obvious authoring cue: the correct proof is often more detailed than the incorrect ones. A rule that simply selects the longest option, breaking ties by option letter, scores 90/113 on those same questions. Across all 711 adapted written mathematical responses, that rule scores 271/711.

This cue does not explain everything, but it makes “Jev solved every proof” an unjustified headline. The defensible claim is that it selected every correct proof from this set of alternatives. Excluding the 108 proof-selection items tagged multi-step, the remaining multi-step questions score 139/203, or 68.5%. Five other proof items were tagged recognition. Both the full score and this diagnostic subset remain visible.

A stronger future test would use independently written, length-matched proof alternatives and fresh problems. I did not revise the options after seeing these results or run again to improve the headline.

Confidence helps. It does not make errors disappear.

For the 936 mathematical follow-up items, the API’s Choice confidence separated reliable answers from shaky ones surprisingly well:

At confidence 1.00, Jev correctly identified the distribution of a sum of independent binomials, the exponential density, and the Poisson probability of at least one event. But the three high-confidence errors are worth looking at individually:

The first answer resembles the distance instead of the number of shortest paths. The second resembles applying linear rather than quadratic variance scaling. The third omits the seven singleton subsets. Those are interpretations of the selected answers, not access to the model’s reasoning.

For an extension, this makes confidence interesting as a routing signal: accept some classifications immediately and send uncertain ones to a slower path. It does not establish a production threshold. This is one public, adapted corpus; the confidence field is also distinct from the API’s per-option probabilities. The 99.4% figure is observed accuracy in a bin, not a general calibration guarantee.

How I made the exam solvable by Jev.

TypeSafe’s Choice interface selects among supplied candidates. It is not an unrestricted essay-writing interface. To use Jev in its intended form, I preserved native choice alternatives and converted written answers into closed choices with plausible mathematical mistakes as distractors.

Numeric questions got numeric or symbolic alternatives. A request to write an equation kept equation alternatives. A proof became a choice among complete arguments. A drawing or graph was represented by its mathematical content: an edge list, transition table, support region or candidate construction. I transcribed the diagrams instead of quietly dropping them. Some related answer boxes were grouped into one tuple, scored correct only if the entire tuple matched.

I read the official solutions while adapting the problems. Jev did not receive those solutions, answer labels, task tags, reference explanations or correctness feedback. This is evaluator-authored multiple choice, not independently validated test design. Alongside review, I used 30 executable checks for selected risky keys—modular powers, interpolation, stable matchings, a corrected probability bound, geometric residuals and small exhaustive counting cases. That is not a proof that every adaptation is flawless.

All question requests and private reference labels were hashed before the new run. The model was pinned to jev-1.13.0. Answer positions were shuffled with a fixed seed per exam and item. Each item received one first-pass judgment, without a calculator, code solver, retrieval, an external reasoning model or answer-driven retries.

Questions were batched by source section. In the follow-up, shared state contains only generic exam instructions. Each judgment’s own instructions contain its complete question and necessary context; it does not see sibling questions or their alternatives. That matters when a later prompt says “prove why X is true,” which could otherwise disclose an earlier true/false answer. The discovery run used shared sibling context, so the expanded benchmark is a disclosed protocol change, not a perfectly controlled replication.

The primary score is exact correctness per adapted item, unweighted. It is not the original partial-credit grading rubric. The follow-up includes two answerable course-context warmups—one birthday and one TA-count question—with their original cross-exam clues restored. Jev got one right. They stay in the 782/938 total, while mathematical slices use 781/936.

Coverage means accounting for omissions, too.

Every linked exam is represented. Administrative pledges, one ungraded drawing and one zero-point joke are excluded. Seven potentially substantive parts are explicitly withheld: four warmups depend on unavailable classroom responses; three Fall 2024 final questions treat arbitrary infinite coefficient sequences as functions without the convergence or evaluation rules needed to define them.

I made source repairs before inference. For example, the Fall 2024 final’s geometric residual uses X ≥ x, which permits zero remaining trials; under the stated trial-count convention, the mean residual is (1−p)/p, not 1/p. A Spring 2025 probability lower bound needed overlap forced both inside an event and outside it. A Spring 2023 “CLT upper bound” became an explicitly labeled normal approximation. Indexing, field-size assumptions, orientation conventions and degenerate cases are recorded as well.

Sixty-six mathematical items carry explicit item-level repair notes. They score 55/66; the other 870 score 726/870. Exam-wide convention notes are also in the coverage file. None of these decisions were based on whether Jev answered correctly.

The coverage inventory lists included source-part identifiers, grouped boxes, repair notes and every withheld part. The reproducibility bundle includes the authoring files, frozen requests and keys, source URLs and hashes, runner, raw responses and offline scorer. The all-results PDF and explorer below include every scored item, including every miss.

Every exam, not a highlight reel.

The thirteen-exam macro average is 82.9%; pooling their adapted items gives 83.4%. More questions do not automatically mean independent evidence: these exams repeat concepts, some repeat nearly identical problems, and public-exam exposure in training is unknown. No deduplication was used to improve the result.

The task groups also differ in option counts, subject mix and authored difficulty. Uniformly random selection would average about 30.2% on the mathematical follow-up, but that is a descriptive baseline, not a difficulty adjustment. The analysis is exploratory; it does not isolate the causal effect of “reasoning depth.”

What the timing measures.

The follow-up used 192 sequential API requests, one attempt each, with no request failures or missing choices. Median full-response latency was 357 ms per section batch; the 95th percentile was 480 ms. The measured run took 71.758 seconds, including local checkpoint writes. Summed response time was 71.282 seconds. Reported usage was 188,156 input tokens and 39,800 output tokens.

Those are batch timings, not latency for one question, and not time spent by a student constructing written solutions. Authoring and review took much longer and are excluded from API timing. The original discovery run’s 6.179 seconds is reported separately.

Inspect all 996 answers.

Filter the full corpus by exam or outcome. Open any item for its alternatives, reference answer, Jev output, confidence and batch timing. The PDF is the complete side-by-side companion.

Loading recorded results…