I'm building a prompt optimizer to learn statistics

I’m building JevGym💪, an open-source prompt optimizer for Jev.

My prior Industrial & Systems Engineering experience taught me how to measure and control processes through statistical methods. They were first applied to manufacturing assembly lines, where defects were previously considered a random occurrence. Statistical process control was developed to monitor quality and efficiency and predict where optimizations would have the most benefit. This practice has since spread to many industries to basically control “randomness.”

When the current LLM boom started, I naturally viewed models as a black box that takes inputs and gives outputs. It’s an incredibly intelligent and magical black box, but we also have to learn how to talk to it so it doesn’t gaslight us. Quite the connundrum.

After a few years of applying AI to development workflows and isolated use cases, I wanted to dive deeper into mathematical methods used to train and optimize models and how we use them. I’m currently taking the Probability in AI online course from Stanford to dive deep into exactly this, and I wanted to get hands on with a real project.

What Jev is

Note: I’m not associated with Jev or TypeSafe in any way! Just a fan.

Jev is a model from TypeSafe. It doesn’t generate text, but just probabilities. They call is a System One model for “quick decisions”, meant to bridge the gap between LLM intelligence and procedural code. Think easy classification, smart if statements, and more. It’s fast and cheap: up to about 200x faster and 400x cheaper than frontier LLMs. 70-500ms response times and fractions of a penny to run many requests.

You pass it a “state” which can be any text blob you want to ask questions about, including structured data. Then you can ask many questions in parallel, either returning a Choice (a question with multiple choices), a Score (give it a rating based on your criteria), or Noul (basically true/false on a sliding scale between 0-1.) The best thing to learn how it works is to sign up and try the playground examples.

It got me thinking: training LLMs and evaluating text output is a hard task, and cost bloats quickly. But what if the output consists of bounded numerical values? And it’s really cheap and fast to train and eval? That made Jev an ideal target to experiment with automated optimization.

Choices

I’m just going to focus on Choice prompts for now. A Choice prompt has two parts, the Instructions and one Description per option. Here’s the most basic prompt starter in this project, to fit the TREC dataset we’ll be using (more on that below.)

Prompt(
    instructions="What kind of answer does `question` ask for?",
    descriptions={
        "ABBR": "abbreviation",
        "DESC": "description",
        "ENTY": "entity",
        "HUM": "human",
        "LOC": "location",
        "NUM": "numeric",
    },
)

The option IDs are straight from the dataset and they don’t change. The optimizer only gets to touch the instructions and the six descriptions.

The task: sorting questions by answer type

The benchmark is TREC question classification, a set of questions labeled by the kind of answer they ask for. “What is the smallest country in Africa ?” is a location question. “What record company produced the 1978 movie The Wiz ?” is a human question, because TREC files companies and groups under human. (That second one comes up again in the next post.)

The total dataset has 5,870 total rows. This needs to be split up into different sets, basically because we don’t want to evaluate the prompt with the same datapoints we used to train it. By a general “rule” I put about 20-25% of data rows aside for that purpose.

• 4,370 in the Training set to feeding the optimizer examples
• 1,000 in the Dev set, scoring prompts while I explore
• 500 in the Test set, the final evaluation run at the very end

Test is the one score that never had a say in any decision.

Rules before results

Before running anything, I wrote down a few fixed rules for how results get reported. These are where most of the statistics comes in, and each one exists because a specific mistake is easy to make.

Measure the spread of results first. If Jev’s answers vary at all from call to call, the same prompt on the same 1,000 questions will score a little differently each run. The noise floor is that run-to-run difference, and I’ll measure it by running one prompt five times. (It could turn out to be zero. That’s a result too.) Any “improvement” inside that range would not be a considerable change.

Measure accuracy relative to confidence. 1,000 questions is a sample. A different 1,000 would give a different score. A Wilson interval is the range of true accuracies that fit the score I got, at 95% confidence. If one prompt’s improvement lands inside that range, it’s “not yet resolved”.

Compare on the same questions. When I compare two prompts, both have answered the same 1,000 questions, so I can see which questions caused a wrong answer to turn right (and vice versa.) This is more accurate than comparing overall accuracy scores, as they could hide actual changes happening.

Don’t let the optimizer see the Dev set. Anything that writes or critiques prompts only sees Train. Once there’s an optimizer, a leak check will scan every prompt it sends for Dev or Test questions and fail on a match.

Label best-of-N. If I generate five prompts and report the best one, that number is optimistic. Some of it is luck. Any number picked from several tries says “best of 5” wherever it appears.

The last one may be an overoptimization, but I want to label properly instead of conveying overconfidence in a potentially weak result.

How I split the work with an agent

I’m doing this with Claude Code, and the setup is built so the AI doesn’t just do the learning for me. When I picked the first round, I wrote down my reason as:

generating a foundation for this project so it runs with experimentation instead of the AI just optimizing everything for me. i want to learn more about the statistical methods

The work runs in rounds. A round answers one question and fits in an evening or two:

1. The agent proposes two or three next moves, each with a guess at the outcome, the cost and what we’d learn either way.
2. I pick one and record a one-sentence reason, so my intuition gets checked against the result later.
3. The agent builds it and runs it on the Dev set, fully logging every call.
4. A separate audit agent verifies the round was run correctly, builds the numbers and charts from the raw logs, and marks each rule pass or fail.
5. A third agent drafts a lab notebook entry with the result, its interval and a “worth knowing” paragraph on whatever stats concept the result depends on. It asks me a few short questions (why I picked this, whether my guess held up), and my answers go in word for word.
6. Rinse and repeat. Decide whether to continue or pivot until I’m satisfied with the result.

What’s next

Before any optimizing, I’ll set up the core of the project, split the data up, and do one run of the minimal prompt above. Then I’ll test the noise floor to see how much spread I get from running the same prompt several times. I should make some more baseline prompts to test against too that don’t use an LLM at all, namely a hand-written prompt and naive Bayes. And then the training will start.

After that comes the question I actually care about. Does an iterative optimizer beat one good pass from an LLM writer model, and if it does, where does the gain come from? I don’t know yet.