Highly recommended paper from CMU and others on how foundation models get trained.

It fundamentally asks whether pretraining, midtraining, SFT, and RL really need to be separate stages.

They show that this separation isn't fundamental.

All of these stages can run under a single likelihood-based algorithm, which they call unified training.

The key step treats suitable RL objectives as maximum-likelihood fitting to model-generated targets. RL generations and static data then share one loss and one trainer.

This changes the design problem. You now choose which data to train on, in what proportions and when. Staged training becomes one special case.

They first trained a 124M model from scratch on reasoning puzzles. The best unified runs beat staged training on the RL tasks. They also beat it on larger puzzles the model never saw.

Those runs reused the midtraining corpus up to 4 times. Staged training lost accuracy when midtraining ran past its best budget.

Under a fixed compute budget, unified training had the best accuracy at the smallest and largest budgets. Staged training led at intermediate budgets.

The real-world test used a 0.76B model on math RL, with 0% to 5% midtraining data mixed in. Math accuracy stayed similar while retention improved. Coding accuracy dropped as the mix grew.

Continual learning gave the clearest result. After math training, all unified models beat staged training on final coding accuracy, despite starting lower. The 5% mix did best.

The experiments are small-scale. The authors have not found a mixture or schedule that works best across the board.

https://arxiv.org/abs/2610.40564

Chat with Paper: https://academy.dair.ai/papers/training-foundation-models-with-a-single-algorithm-2610.40564