Hugging Face 的 Thomas Wolf 发布免费开源的《The Ultimate MoE Training Handbook》，基于团队自费 800k GPU 小时完成的实验，围绕训练团队常遇到的问题组织内容。手册约 7 万字，含 55 张交互图表、可运行代码和实测数据，覆盖专家数量与规模、all-to-all 通信开销、router 选择与 fp32 保留、aux loss 与 DeepSeek 的 loss-free bias、dense checkpoint upcycle 以及推理时专家数少于训练时等话题。东锡 NLP 转发并推荐了这份手册。
Hugging Face 基于 800k GPU 小时实验整理出 MoE 训练手册，覆盖专家配置、all-to-all 通信与路由选择等实操问题。
裂墙推荐就完事了！
如何训练 Mixture-of-Experts.
The Ultimate MoE Training Handbook
【引用 @Thom_Wolf】18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.
So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?
⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.
Read it here: https://huggingface.co/spaces/huggingface/moe-blogpost
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: https://huggingface.co/spaces/huggingface/moe-blogpost在 X 查看被引用的帖子
来源：Dongxi 东锡 NLP · x.com