A recent technical write-up by developer Nish Tahir walks through building what it calls a “decision model” — a stripped-down way of using a large language model that answers multiple-choice-style questions in a single forward pass instead of generating a full text response token by token. The project, built around a small open-source model called Qwen3-1.7B and a reference implementation named Jev, matters because it highlights both a practical speed trick for structured AI outputs and a deeper, less obvious problem: these models are often confidently wrong.

Why generating structured output is slower than it looks

When you ask a typical language model to return JSON or any other typed output, techniques like Structured Output (also called constrained decoding) are used to force the model’s output to match a valid schema. The catch is that even though the model processes your input in a single pass, it still has to generate the response one token at a time. Tahir’s example shows that producing a short structured answer required 11 separate passes through the model — one for each output token — even before accounting for optimizations like speculative decoding.

That per-token generation loop is a major reason structured outputs can feel slow, especially for latency-sensitive applications. Every additional token in the response adds another round-trip through the model’s computation.

How a decision model collapses this into one pass

Decision models like Jev take a different approach: they assume the universe of possible answers is small and fixed, such as a multiple-choice question with options A, B, C, D, and E. Instead of letting the model freely generate text, the system masks every token in the model’s vocabulary except the allowed answer choices. The model then only needs to produce one token, and whichever option has the highest probability becomes the answer.

This is a meaningful architectural difference from standard structured-output approaches. Rather than constraining a multi-token JSON response, it constrains the problem down to a single token emitted in a single pass. The tradeoff is scope: this only works when the answer space is genuinely limited to a known set of discrete choices, not open-ended text generation.

Constraining the vocabulary guarantees the model’s output will be one of the allowed options — but it does not guarantee that option is correct. The method restricts the format of the answer, not its accuracy.

Testing accuracy against a real benchmark

To measure how well this actually performs, Tahir ran the Qwen3-1.7B-based decision model against a held-out random sample from CommonsenseQA, a public benchmark dataset for commonsense reasoning questions. The source describes the results as “not bad for a 1.7B model,” and notes that a quick fine-tune on the dataset produced a further, modest accuracy improvement. No exact accuracy percentages for this baseline test are given in the excerpt, but the comparison point — a relatively small 1.7-billion-parameter model performing reasonably on a recognized reasoning benchmark — is presented as a notable result for a lightweight, locally runnable setup.

The calibration problem: confident does not mean correct

The more significant finding in the piece concerns confidence scores. It’s common practice to treat a constrained model’s output token probability as a kind of confidence score — essentially, “how sure is the model that this is the right answer.” Tahir tested this assumption with a deliberately ambiguous question that had no clear correct answer, and found the model still reported extremely high confidence in one option, despite there being no good reason to be sure.

Digging further, the analysis bins the model’s confidence scores from the CommonsenseQA evaluation into ranges and compares each bin’s stated confidence to its actual accuracy. The mismatch is stark: in the 0.9–1.0 confidence bin, the model was only correct about 70% of the time, and in the 0.8–0.9 bin, accuracy dropped to roughly 40%. In other words, when the model claims to be almost certain, it’s frequently wrong — a textbook sign of a poorly calibrated model. The underlying explanation offered is that, without additional training for this purpose, token probabilities reflect the model’s confidence in predicting the next token linguistically, not a genuine estimate of whether its final answer is correct.

Fixing confidence with temperature scaling

To address the mismatch between stated confidence and real-world accuracy, the write-up applies a calibration technique called temperature scaling. This involves adjusting a single temperature parameter that reshapes the model’s output probability distribution, flattening overconfident spikes so the resulting scores better track actual accuracy. By curve-fitting this parameter against the model’s measured performance, Tahir found a temperature value of approximately 3.8 that produced noticeably better-calibrated confidence scores — meaning a stated 90% confidence actually corresponds much more closely to being right 90% of the time.

What this means for developers experimenting with it

The practical takeaway is twofold. First, for applications that genuinely only need to pick from a small, fixed set of answers — classification tasks, multiple-choice evaluation, routing decisions, and similar use cases — a single-pass, vocabulary-masked decision model can be dramatically more efficient than generating and parsing full structured text output. Second, anyone using raw token probabilities as a trust signal should be cautious: without calibration, a model’s apparent confidence can be misleading, which matters in any system where that confidence score is used to decide whether to trust, escalate, or double-check an automated answer.

Tahir has published the accompanying code as a GitHub repository with scripts covering dataset construction, evaluation, fine-tuning, and calibration, and explicitly encourages others to try the approach on larger models beyond the 1.7B-parameter example used in the original testing.

Frequently asked questions

Is a decision model the same as using Structured Output or JSON mode?
No. Structured Output constrains a multi-token response to match a schema like JSON, which still requires one model pass per token. A decision model goes further by narrowing the task to picking one token from a small, predefined set of options, letting it answer in a single pass.

Does constraining the output guarantee a correct answer?
No. Masking the vocabulary only guarantees the model picks from the allowed options — it says nothing about whether that choice is actually right, which is why separate accuracy testing and calibration are still necessary.

Why does model calibration matter if accuracy is already measured?
Accuracy tells you how often a model is right overall, but calibration tells you whether its confidence score for a specific answer can be trusted. An uncalibrated model can be right 70% of the time while claiming 90%+ confidence, which is risky for systems that use that confidence score to make automated decisions.

Source: Build your own decision model