跳到正文
Sebastian Raschka· @rasbt · X·· 2 天前AI 评分58
AI 导读

Sebastian Raschka 点评了 Marin Mochi(535B-A23B)的检查点发布,称其技术报告说明了各组件为何这样选、如何选,出发点包括训练稳定性和 GPU 吞吐。

正文 · 原文

Whoa, great and even excessive Marin Mochi (535B-A23B) checkpoint drop (in a good way!)

Comes with great tech report btw with insights into why / how components were picked (for training stability and GPU throughput).

Short fact sheet:

1) Latent MoE with an extra RMSNorm on the latent that made latent MoE actually work (better)

2) Classic 3:1 sliding-window/global gated GQA, partial RoPE (local) and NoPE (global)

3) Adding short convolutions on keys, attention output, and MoE output

4) Exclusive self-attention (XSA) -- I guess the end-of-the year architecture roundup this year won't be boring 😅

引用David Hall@dlwh
We’re releasing all 57 intermediate checkpoints from Marin “Mochi” 535B-A23B pretraining, including 10 from long-context extension. 18.2T tokens, about 61 TB of weights. https://huggingface.co/open-athena/Marin-Mochi-535B-A23B Blog: https://openathena.ai/blog/marin-mochi-pretraining-checkpoints/
在 X 查看被引用的帖子

来源:Sebastian Raschka · x.com