Looped Transformer 通过共享同一组 Transformer 块进行多次迭代,将隐藏状态反馈回输入以复用权重。该架构需明确重复的模块、传递的状态及执行规则(如固定或自适应步数)。此设计在不增加额外参数的前提下提升了计算深度,并支持 KV cache 优化。
Looped Transformers are low-key fascinating. First of all, you need to understand what travels around the loop and the 3 core things you set for the architecture.
A Looped Transformer runs the same Transformer block (or stack) for multiple passes, feeding each output back in as the next input.
▪️ What travels around this loop is the hidden state:
- Token embeddings form the initial hidden representation.
- The shared stack updates that representation via attention and feed-forward operations.
- Its output goes back to the stack for another pass, using an updated hidden state. Some designs also reuse the input, add time-step embeddings, or run multiple residual streams.
- After the chosen number of passes, the output head converts the selected representation into token scores.
That's all to win additional computation without an additional set of block weights for every pass.
Although the weights stay the same during inference, changing inputs let the same block produce different updates. During decoding it attends via the KV cache, during prefill it updates masked tokens in parallel.
A model can loop its entire stack or repeat only a middle stack between ordinary input and output blocks.
Note that repeating the stack doesn't require generating a new text token between passes.
▪️ So there are 3 basic things you need to identify for the specific Looped Transformer architecture:
- The repeated blocks
- The state they receive
- The rule that controls their execution - a fixed number of passes, a user-selected count, or an adaptive rule
If you want to nerd out on the rest, like adaptive depth, what happens to the KV-cache and hardware, 2026 breakthroughs, and when another pass is worth the cost - we put together a full guide over here: https://www.turingpost.com/p/looped-transformers-when-is-another-pass-worth-the-cost
来源:Turing Post (Ksenia Se) · x.com