跳到正文
elvis· @omarsar0 · X·· 3 小时前AI 评分58
AI 导读

Google 研究团队提出 BuggyMLE,通过让智能体向 222 个可运行的 ML 仓库注入受控 bug,构造更难的仓库级 ML 工程任务。

正文 · 原文

Banger paper from Google on ML engineering agents.

Pay attention to this one, builders. It's a clever way to scale ML engineering by injecting bugs.

(bookmark it)

Repository-level coding benchmarks are saturating. BuggyMLE makes tasks harder on purpose by having agents inject controlled bugs into 222 working ML repositories.

  • A Code Complexity Index built from ten repo features predicts average solve rates across eight coding benchmarks with R^2 = 0.951.
  • On the graded MLE-Bench Lite variant, the best of 14 models drops from a 95% solve rate on the easiest tier to 9% on the hardest.
  • A Qwen3.5-9B trained on the bug curriculum rivals frontier models there and raises SWE-bench Lite from 24% to 33%.

Patch size and code complexity are the strongest predictors of difficulty, so the generator can raise both to produce harder tasks and training data from the same repos.

This is a great example of how to build harder tasks from your own repos to improve the overall system.

https://arxiv.org/abs/2611.25088

Chat with Paper: https://academy.dair.ai/papers/buggymle-scaling-machine-learning-engineering-difficulty-with-bug-injection-2611.25088

来源:elvis · x.com