Mira Murati· @miramurati · X·· 9 天前AI 评分58
AI 导读
Thinking Machines 在 Connectionism 上详解了 Inkling 2 Small 的端到端 4-bit 训练方法,在 NVIDIA GB300 上从预训练第一步到最终 RL rollout 全程使用原生 4-bit 前向。Mira Murati 表示这是整套技术栈多项改进之一,将预训练时间从数周缩短到 68 小时。引用内容指出,用更少比特训练速度更快,但量化误差可能干扰学习。
正文 · 原文
Today on Connectionism: how we trained Inkling 2 Small with a native 4-bit forward pass on @nvidia GB300s, from the first pretraining step through the final RL rollout, one of the many improvements across the stack that took pretraining from weeks to 68 hours.
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. https://thinkingmachines.ai/blog/end-to-end-4-bit-training在 X 查看被引用的帖子
来源:Mira Murati · x.com