swyx 转述 OpenAI 的观点,OpenAI 提议用困惑度而非 FLOPS 作为行业自我监管与节奏把控的通用指标。引用内容称,新能力主要来自压缩方面的进展,基于困惑度的指标可为通用智能提供共享基准,并预计进入约 0.68-0.7 log-perplexity 区间时会出现更多变革性能力。
OpenAI proposes perplexity as the universal metric for industry self regulation and pacing (as opposed to eg FLOPS)
“Given that the main way new surprising capabilities emerge is by progress on compression, a measure rooted in perplexity could give us a shared benchmark for general intelligence, regardless of the degree of automation of the development process. Extrapolating from recent progress, we should expect further transformative capabilities as we enter the ~0.68-0.7 log-perplexity window.”
this chart is one of the most beautiful ways to tell the story of the last 4 years. it also shows that a lot of the post training issues we debate is kinda the B plot to the overall A plot of large scale pretrain progress.
We decided to provide more transparency into pretraining progress at OpenAI for the sake of moving toward shared safety bars. In “Perplexity and Intelligence” I argue for common limits based on perplexity and other measurements, with progress beyond them gated on our ability to understand AI behavior. https://openai.com/index/perplexity-and-intelligence在 X 查看被引用的帖子
来源:swyx · x.com