Google 一篇论文提出自改进 Agent harness 的统一框架,让优化器围绕冻结的 Gemini 3.8 Flash 在 SWE-bench Lite 上重写 harness,并对比三种做法。
Another very interesting paper from Google on self-improving agent harnesses.
(bookmark it)
They let an optimizer rewrite the harness around a frozen Gemini 3.8 Flash on SWE-bench Lite and compared three approaches.
- A prompt rule file lowered held-out accuracy from 5.0% to 2.5%.
- A full rewrite of the harness reached 42.5% at 451.6k tokens per task.
- Gated edits to one module at a time reached 40.0% at 46.7k tokens per task.
The gated version splits the harness into four parts (context, control, tools, and verification). Each round it patches one part, checks the patch on held-out tasks under a token cap, and deploys it only if it passes.
The gated harness also transferred to Gemini 4 Argon with no new search, solving 37.5% against 30.0% for the full rewrite. The test set is small (40 held-out issues), so treat the numbers as directional.
The paper also lists failure modes to guard against when a harness edits itself, including prompt bloat, benchmark gaming, and evaluator tampering.
https://arxiv.org/abs/2611.22299
Chat with Paper: https://academy.dair.ai/papers/harnesses-that-improve-themselves-a-unifying-framework-for-self-improving-agent-2611.22299
来源:elvis · x.com