OpenAI 公开了基础模型 log-perplexity(每 token 平均交叉熵损失)随版本迭代的下降曲线,从 GPT-3.5 一路延伸到 GPT-5.5、GPT-5.6 Sol 及一个内部模型。Jakub Pachocki 表示最早 2027 年可能出现类似数学领域那样的预训练基础性突破潮,但他认为对齐与监控能力的进展难以跟上预训练加速,并呼吁各 AI 实验室以共同限制来控制节奏。
OpenAI just gave us a glance behind the curtain.
This is how much models have improved in terms of log perplexity or the average cross-entropy loss per token, which measures how well a base model predicts the next token
Jakub Pachocki says: "as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements" (similar to the advances in mathematics)
He also warns that: "While I expect AI will also significantly contribute to progress on alignment and monitoring, I think that progress is unlikely to be fast enough to allow the full potential acceleration in pretraining."
Jakub also calls for pacing with "common limits" for AI labs to make sure we first understand and are able to monitor new, even more capable systems.
We decided to provide more transparency into pretraining progress at OpenAI for the sake of moving toward shared safety bars. In “Perplexity and Intelligence” I argue for common limits based on perplexity and other measurements, with progress beyond them gated on our ability to understand AI behavior. https://openai.com/index/perplexity-and-intelligence在 X 查看被引用的帖子
来源:Lisan al Gaib · x.com