Google 研究团队提出 BuggyMLE,通过让智能体向 222 个可运行的 ML 仓库注入受控 bug,构造更难的仓库级 ML 工程任务。
Banger paper from Google on ML engineering agents.
Pay attention to this one, builders. It's a clever way to scale ML engineering by injecting bugs.
(bookmark it)
Repository-level coding benchmarks are saturating. BuggyMLE makes tasks harder on purpose by having agents inject controlled bugs into 222 working ML repositories.
- A Code Complexity Index built from ten repo features predicts average solve rates across eight coding benchmarks with R^2 = 0.951.
- On the graded MLE-Bench Lite variant, the best of 14 models drops from a 95% solve rate on the easiest tier to 9% on the hardest.
- A Qwen3.5-9B trained on the bug curriculum rivals frontier models there and raises SWE-bench Lite from 24% to 33%.
Patch size and code complexity are the strongest predictors of difficulty, so the generator can raise both to produce harder tasks and training data from the same repos.
This is a great example of how to build harder tasks from your own repos to improve the overall system.
https://arxiv.org/abs/2611.25088
Chat with Paper: https://academy.dair.ai/papers/buggymle-scaling-machine-learning-engineering-difficulty-with-bug-injection-2611.25088
来源:elvis · x.com