2026-11-11 · JevGym
My baseline Jev Choice prompt scored 85.7%*: a breakdown
What a Wilson interval, Brier and log loss each tell us about the results.
My first JevGym run worked pretty well, but there is room for improvement on this dataset as I predicted. The most basic prompt I have got 857 of 1,000 questions right, for 85.7%.
However, it reported a log loss of 1.545, which looks very high for a model that’s right 86% of the time. Most of that 1.545 comes from just 35 questions. (I’ll explain later.)
This post is about taking a first measurement and about what each of its numbers actually tells you.
What I ran: the barest prompt on 1,000 questions
JevGym is a prompt optimizer I’m building for Jev Choice questions. I’m building it in public with an emphasis on experimentation. I’m not sure what this will look like in the end.
The task is TREC question classification. Each question gets one of six classes for the kind of answer it asks for: abbreviation (ABBR), description (DESC), entity (ENTY), human (HUM), location (LOC) or numeric (NUM). “What is the smallest country in Africa ?” asks for a location.
My ultimate goal for this stage is to run one fully logged evaluation on Dev. The prompt was what I’ll call the Minimal prompt, the barest one in the plan. The instruction is “What kind of answer does question ask for?”, and each option is described only by its class name, like “human” or “location”. 1,000 Jev calls for a total of $0.016.
Here’s one of those calls, for the smallest-country question from above. You can see Jev mislabeled this one according to the dataset.
What kind of answer does `question` ask for?
Confidence: 57%
*85.7% means “somewhere between 83.4% and 87.7%”
1,000 questions is just a sample. Score the same prompt on a different 1,000 TREC questions and you’d get a slightly different number. A Wilson interval defines a range for this: it’s the set of “true” accuracies that fit 857 correct out of 1,000 at 95% confidence. For this run it’s 83.4% to 87.7%, so roughly two points either side. If I ran the Minimal prompt on a fresh 1,000 questions, a score anywhere in that range wouldn’t be surprising.
What I can’t say yet is whether 85.7% is good, because there’s nothing to compare it with. That needs a second prompt to compare against, and a second number, the noise floor, or the variation from run to run. (That’s the next post.)
Almost all the misses land in “entity”
Minimal got 143 questions wrong. In 104 of those (73%), Jev’s answer was ENTY. It answered ENTY 315 times, and Dev only has 232 ENTY questions.
This chart is a confusion matrix. Each row is the correct label, each column is what Jev picked, and the diagonal is where it got things right. Reading along a row gives you recall, the share of each class Jev gets right: ABBR 81% (13 of 16), DESC 90%, ENTY 91%, HUM 77%, LOC 78% and NUM 93%.
ENTY’s 91% sounds great until you read down its column. Of the 315 times Jev said ENTY, 211 were right, which is 67%. That share is called precision, and it’s where the ENTY problem shows up. In particular, we see 21% of HUM questions and 22% of LOC questions get called “entity”.
TREC also has finer labels underneath the six classes, and they show which questions get lost. Of the 35 Dev questions labeled HUM:gr (a group or organization), 30 went to ENTY. One of them is “What record company produced the 1978 movie The Wiz ?”. On the location side, 21 of 92 LOC:other and 8 of 21 LOC:country questions went to ENTY, including that African country question above.
TREC files companies and groups under “human” and named countries under “location”. A bare class name doesn’t tell Jev that. It reads “human” as a person, so it calls a company an entity. So the problem is in the option descriptions. Jev reads the questions fine; it just doesn’t know TREC’s filing rules. That gives the next two prompts a clear target: the Hand-written prompt uses descriptions I wrote, and the AI-written prompt uses descriptions an LLM writes.
How confidently wrong was it?
Accuracy only checks Jev’s top pick. But Jev returns a probability for every option, and we can use other scores to grade those probabilities.
We can use a loss function to measure the model’s right and wrong answers against how “surprised” the model should be by the actual answer. For example, if a right answer was given a 0.9, that question would cost some penalty even though the model was correct. The goal is obviously to minimize this score.
Log loss became an easy way to compute and represent this value. It uses natural logarithm to turn tiny probabilities into larger negative numbers we can better compare, and then averages all the penalties.
If Jev gave the right answer 0.9, that question costs 0.11. At 0.5 it costs 0.69, and at 0.01 it costs 4.6. As heads toward 0, the cost heads toward infinity, which makes log loss very hard on confident misses.
On 35 questions, Jev gave the right answer a probability of exactly 0. Since is infinite, we substitute in a very small at (that’s 0.000000000000001). Each of those 35 questions then costs . The other 965 questions average 0.349. So 1.545 is mostly from several confident misses.
Another note: every probability Jev returned is a multiple of 0.01, so “exactly 0” really means “below half a percent”. It doesn’t mean “impossible”. Zeros are common, too. 62% of all the probabilities Jev returned in this run are exactly 0. If I clip at half a rounding step (0.005) instead of , the same run, with the same answers, has a log loss of 0.522, about a third. I guess it depends on how much I want to penalize giving right answers 0.00 probability.
The other probability score is the Brier score. Write the right answer as 1 for the true option and 0 for the other five. Brier is the squared distance between Jev’s probabilities and that, added up over all six options. 0 is perfect. The worst case is putting 100% on a wrong option, which costs 2 (1 for the wrong option, 1 for the missed right one). Because it can’t go past 2, a handful of confident misses can’t blow it up. Minimal’s Brier is 0.251, and unlike log loss, it isn’t dominated by those 35 questions.
This goes back to the ENTY problem. 28 of the 35 zero-probability questions are true HUM, and 25 of them were predicted ENTY. Most of the confident misses are the “a company is an entity” mistake again.
Some of the misses are TREC’s fault
When I looked through those 35 zero-probability questions, a few of them are just mislabeled. “What role do fruits play in the survival of plants ?” is labeled HUM.
My first reaction, straight from my notes: “label errors? is this dataset really reliable? should we choose another one?” And then: “yes, it is a surprise but suppose it’s real world”.
I’m keeping TREC. Wrong labels make comparisons between prompts fuzzier, but they don’t tilt them toward one prompt or the other. They also cap the best score anyone can get, since a prompt that answers a mislabeled question correctly still gets marked wrong. To find out how much this matters, I added a candidate round to the plan for after the baselines are in. I’ll take the Dev questions where all four baselines confidently disagree with the label, review about 50 of them by hand, and estimate Dev’s label-error rate with (of course) a Wilson interval.
Next: how much does the score wiggle on its own?
The next round will run the same prompt a few more times on Dev to measure the run-to-run variance. My guess is that it’ll be under 1 point, and a handful of questions will have their answers change.
Then we’ll start comparing against the Hand-written and AI-written prompts. Then 85.7% finally has something to be measured against.