清华大学与月之暗面(Moonshot AI)的论文提出 TCSExam 基准,用于评测前沿 LLM 在理论计算机科学(TCS)研究上的能力。该基准含 100 个专家评审任务,分 Lean 定理证明、求解器工程、破纪录构造和 30 个开放猜想四条赛道,由 Lean kernel 或任务检查器校验每次提交。
论文用可机器校验的基准同时测定理证明与开放猜想,读者可据此看到当前模型在真实研究任务上的边界。
New Tsinghua and Moonshot AI paper finds that GPT-6 Astra proves 70.0% of research-level computer science theorems in Lean, yet the same model resolves 0 of 30 open conjectures in theoretical computer science (TCS).
Theoretical computer science is a natural test for AI research, because proofs can be machine-checked and records can be beaten. Existing benchmarks cover only slices, like proofs or code.
TCSExam has 100 expert-reviewed tasks in 4 tracks: Lean theorem proving, solver engineering, record-setting constructions, and 30 open conjectures. The Lean kernel or a task checker verifies every submission.
Without supplied lemmas, GPT-6 Astra fully proved theorems in 70.0% of attempts, Kimi K3.1 Pro in 38.3%, and the other 6 agents in at most 11.7%. The few accepted conjecture submissions adapted counterexamples and proofs OpenAI had already published.
来源:Rohan Paul · x.com