跳到正文
elvis· @omarsar0 · X·· 2 小时前AI 评分60
AI 导读

论文提出 AutoRAGBench,让智能体在无起始代码、无建议方案的情况下,用六小时和一块 H800 GPU 完成九项 RAG 开发任务。GPT-6.1 Sol 搭配 Codex CLI 平均得分 56.1,取各任务最佳已发表方法组成的人工基线为 44.4。当智能体从人工设计的系统出发而非从零开始时,七项任务平均提升 16.6 分;论文还比较了九种智能体配置及其六小时内的开发过程。

正文 · 原文

Can coding agents build better RAG systems than the best published human designs?

AutoRAGBench gives an agent nine RAG development tasks to measure this. There is no starter code and no suggested approach. The agent gets six hours and one H800 GPU to analyze the data, build a pipeline, and refine it from experiments.

GPT-6.1 Sol with Codex CLI averages 56.1. The composite human baseline, which takes the best published method on each task, averages 44.4.

When the agent starts from a human-designed system instead of from scratch, it adds 16.6 points on average across seven tasks.

The paper compares nine agent setups and also studies how each one's development process unfolds over the six hours.

Paper: https://arxiv.org/abs/2611.25582

Chat with Paper: https://academy.dair.ai/papers/autoragbench-a-benchmark-for-llm-agents-automating-rag-system-development-2611.25582

来源:elvis · x.com