Does on-policy preference optimisation help after targeted fine-tuning?

Question

After fine-tuning Atom on targeted, automatically checked examples, does a further round of on-policy preference optimisation improve it?

Method

In on-policy preference optimisation, the model samples several answers to each training prompt, an automatic checker marks each one, and the model is trained to prefer the answers that passed over those that failed. We used a length-normalised variant of direct preference optimisation (DPO).

We ran it twice, each time starting from a fine-tuned checkpoint:

• Baseline probe3, treatment probe3-dpo
• Baseline probe6, treatment probe6-dpo

We measured all four checkpoints on our internal development set: 36 automatically checked items, run through the live product three times for the probe3 pair and twice for the probe6 pair. The v1.2 release report describes that set. It also guides our training, so treat these results as indicative rather than held-out.

Results

Item by item, an item counting as better if it passed more of its runs:

• probe3-dpo against probe3: better on 4 items, worse on 6. Sign test p ≈ 0.75. No measurable difference.
• probe6-dpo against probe6: better on 1 item, worse on 6. Sign test p ≈ 0.13.

The probe6 comparison points towards harm but does not establish it. The items probe6-dpo lost were two maths word problems, two arithmetic puzzles and two instruction-following prompts.

Conclusion

In the configuration we tested, preference optimisation did not improve Atom, and after probe6 it may have made it worse. We did not ship either DPO checkpoint.

Hypothesis, not yet tested

In the product, Atom answers many maths questions by writing a short program that the Python sandbox runs. The sampler that generated answers for preference training could not run programs. Our hypothesis: answers that took the program route would have reached the checker with no final number, been marked as failures and ended up on the rejected side of the pairs. The training would then have pushed Atom away from the program-aided maths that works in the product.

Four of the six items probe6-dpo lost involve arithmetic, which fits this explanation, but six items cannot separate it from other causes. Two experiments would test it:

1. Leave maths and arithmetic prompts out of the preference data, and check whether the maths items still regress.
2. Execute programs during sampling, so that program-aided answers are judged on their results.

Until one of these is run, the explanation stays a hypothesis.