Fine-tuning a math tutor, and catching its judge lying

The goal

I wanted a small model that behaves like a tutor, not an answer machine. Given a K-12 math question, it should teach: state the goal, walk the steps, show an example, and end with a checkpoint question. And when a student asks it for something that is not math, it should decline and steer back to math instead of playing along.

Base Qwen3-8B is good at math. It is not good at tutoring, and it will happily write Python when a student asks. That gap is what the fine-tune targets.

Data

No suitable dataset existed, so I generated one: seed problems first, then multi-turn tutoring conversations built on them, then an automated cleaning pass. Every response in the final set follows the Goal, Steps, Example, Checkpoint structure. The final training set had 612 curated examples, which is small on purpose. I wanted to know how far careful curation gets you before scale does.

Two training rounds

The first round (LoRA rank 32, 526 examples) looked fine in the loss curves and failed in practice. Out-of-domain refusal improved a lot, but pedagogy got worse than the base model. The tutor structure was there on the surface while the actual teaching quality regressed.

For the second round I dropped the rank to 16, grew and re-curated the data to 612 examples, and added NEFTune noise injection. That combination fixed the pedagogy regression while keeping the refusal gains. The lesson that stuck with me: for behavior-shaping fine-tunes, data quality and a smaller adapter beat a bigger adapter on noisier data.

The evaluation

I evaluated both the base model and the fine-tune on 50 held-out prompts: 42 math questions across grade levels and 8 out-of-domain traps (coding requests, general chat, homework in other subjects). Claude Sonnet 4 scored each response alone, without seeing the other model's answer, in randomized order, on five criteria from 1 to 5: correctness, pedagogy, structure, faithfulness, and refusal.

The judge was rewarding the wrong thing

Reading the judge's raw JSON, I found scores that made no sense. When a student asked for Python help, the base model wrote a working bubble sort, and the judge gave it correctness 5. The code was correct. But a math tutor writing Python on request is a failure, not a win. The judge was also inconsistent on refusals, scoring the same kind of response as N/A one time and 5 another.

Averaged over everything, these errors quietly inflated the base model's numbers. I corrected the methodology by scoring each metric only where it applies: correctness and pedagogy on the 42 math prompts, refusal on the 8 out-of-domain prompts, structure and faithfulness on all 50. Both the raw and corrected scores are in the repo, so anyone can check the difference.

Results

The fine-tune wins on all five criteria. The two that matter most to me are refusal, up 0.98 on a 5-point scale, because that was the whole point of the tune, and faithfulness, up 0.18, because a tutor that invents math is worse than no tutor. The in-domain gains are small and honest: base Qwen3-8B was already strong at math, and I am not going to pretend 612 examples changed that.

Limitations I know about

• The judge prefers longer responses. I documented this verbosity bias rather than pretending it away.
• Single-turn scoring only. It cannot measure how the tutor handles a real back-and-forth.
• No human evaluation yet. LLM-as-judge with one judge is a proxy, and I treat it as one.
• Eight OOD prompts is a small refusal sample. The +0.98 is directionally solid but the exact number is noisy.

What I took from it

I now assume every eval is wrong until I have read its failures myself.

The fine-tuning itself was the easy half. The work that changed the outcome was reading raw judge output line by line, noticing the scores that should not exist, and rebuilding the methodology so the metric measures the behavior I actually care about. I now assume every eval is wrong until I have read its failures myself.