Turning Qwen3-1.7B into a one-pass multiple-choice classifier
A tutorial shows how masking Qwen3-1.7B's output to fixed labels gives one-pass decisions, and why its raw probabilities are overconfident.
A tutorial published October 10 on Nishtahir's blog shows how to turn a small open model into a fixed-choice classifier. Restrict the output to a few allowed tokens and read the answer from a single forward pass, with no token-by-token generation. If you route tickets, triage inputs or tag content against a known label set, this pattern is cheap to try. It was shared on X by @bibryam on October 11.
The author uses Qwen/Qwen3-1.7B with transformers and torch. The script formats a question with options A–E, runs one pass and takes the last-position logits for the option tokens only. A softmax over those logits picks the answer. For comparison, the post counts 11 passes for an equivalent structured JSON output. It gives no wall-clock timing.
The post reports these results:
- Accuracy: 59.4% on a 1,221-item CommonsenseQA holdout, and 62.4% after a quick fine-tune.
- Confidence: the raw probabilities are not calibrated. Predictions scored 0.9–1.0 were right about 70% of the time. Those scored 0.8–0.9 were right about 47% of the time.
- Calibration: fitting a temperature-scaling value of about 3.80 brought confidence much closer to accuracy.
The author says a GitHub repo covers dataset building, evaluation, fine-tuning and calibration, but the post excerpt doesn't link it.
One practical caveat is our reading of the code, not something the post states. The script keeps only the first token of each option, so it assumes every label is a single token. Multi-token labels would need extra handling. The post also doesn't give pinned library versions. The fixed label set also guarantees a valid answer, not a correct one. Before you trust the scores, evaluate and calibrate on your own data.
Sources
- Build your own decision model · Another Dev's Two Cents
- @bibryam: Build your own decision model with Qwen · @bibryam