跳到正文
Lisan al Gaib· @scaling01 · X·· 21 小时前AI 评分58
AI 导读

Lisan al Gaib 评论 Thinking Machines 发布的 Inkling 2 Mini,认为这个 26B 参数的循环 Transformer 比他本地试过的小型循环模型训练得更充分。

正文 · 原文

As I said. It's much too late. The loopers are already here.

This one is a much bigger and more well trained looping model.

It has pretty much the architecture I expected GPT-6 to have with the middle half of layers looped, block attention residuals and input gates for the previous loop's message.

(Locally I only tried a small looping model with different positions for the recurrent block. I used "Hyperloop" which also adds some residual stream expressiveness with hyperconnections and gates. Well, but the Inkling one is clearly better than whatever I tried lol)

Fable 5.2's architecture visualization is a bit messy.

引用Thinking Machines@thinkymachines
Some of you thought that Inkling 2 Small was too big at 594B parameters. We heard you: Inkling 2 Mini is a looped transformer with just 26B parameters, small enough to run on your laptop, powerful enough for agentic work with a score of 20.1 on AA Index. https://thinkingmachines.ai/news/inkling-2-mini/
在 X 查看被引用的帖子

来源:Lisan al Gaib · x.com