跳到正文
elvis· @omarsar0 · X·· 3 天前AI 评分60
AI 导读

Google 一篇论文提出自改进 Agent harness 的统一框架,让优化器围绕冻结的 Gemini 3.8 Flash 在 SWE-bench Lite 上重写 harness,并对比三种做法。

正文 · 原文

Another very interesting paper from Google on self-improving agent harnesses.

(bookmark it)

They let an optimizer rewrite the harness around a frozen Gemini 3.8 Flash on SWE-bench Lite and compared three approaches.

  • A prompt rule file lowered held-out accuracy from 5.0% to 2.5%.
  • A full rewrite of the harness reached 42.5% at 451.6k tokens per task.
  • Gated edits to one module at a time reached 40.0% at 46.7k tokens per task.

The gated version splits the harness into four parts (context, control, tools, and verification). Each round it patches one part, checks the patch on held-out tasks under a token cap, and deploys it only if it passes.

The gated harness also transferred to Gemini 4 Argon with no new search, solving 37.5% against 30.0% for the full rewrite. The test set is small (40 held-out issues), so treat the numbers as directional.

The paper also lists failure modes to guard against when a harness edits itself, including prompt bloat, benchmark gaming, and evaluator tampering.

https://arxiv.org/abs/2611.22299

Chat with Paper: https://academy.dair.ai/papers/harnesses-that-improve-themselves-a-unifying-framework-for-self-improving-agent-2611.22299

来源:elvis · x.com