Lisan al Gaib· @scaling01 · X·· 21 小时前AI 评分58
AI 导读
Lisan al Gaib 评论 Thinking Machines 发布的 Inkling 2 Mini,认为这个 26B 参数的循环 Transformer 比他本地试过的小型循环模型训练得更充分。
正文 · 原文
As I said. It's much too late. The loopers are already here.
This one is a much bigger and more well trained looping model.
It has pretty much the architecture I expected GPT-6 to have with the middle half of layers looped, block attention residuals and input gates for the previous loop's message.
(Locally I only tried a small looping model with different positions for the recurrent block. I used "Hyperloop" which also adds some residual stream expressiveness with hyperconnections and gates. Well, but the Inkling one is clearly better than whatever I tried lol)
Fable 5.2's architecture visualization is a bit messy.
Some of you thought that Inkling 2 Small was too big at 594B parameters. We heard you: Inkling 2 Mini is a looped transformer with just 26B parameters, small enough to run on your laptop, powerful enough for agentic work with a score of 20.1 on AA Index. https://thinkingmachines.ai/news/inkling-2-mini/在 X 查看被引用的帖子
来源:Lisan al Gaib · x.com