跳到正文
Turing Post (Ksenia Se)· @TheTuringPost · X·· 3 天前AI 评分68
AI 导读

斯坦福一项论文让 OpenAI、Anthropic、Google 和 Qwen 的 8 个模型回答 620 个真实临床问题,再匿名互评,发现每个评委都偏好自己的答案,平均排名高出 1.22 位,GPT-5.6 Sol 接近 3 位。作者指出,医疗 AI 基准常用 LLM 给 LLM 打分,评委偏好自身输出会让基准偏向该模型;即使去掉评分细则,这种偏好依然存在,公开模型名称也只让偏差缩小 0.094 位。作者建议使用跨模型家族的评委小组,并按评委分别报告结果。

正文 · 原文

8 models graded 620 anonymized medical answers. Every one of them ranked its own answer higher than the other 7 judges did. About 1.2 places on average, almost 3 for GPT-5.6 Sol. Telling them which model wrote which answer barely changed it.

A lot of the leaderboards we read every week use a single LLM judge. How many of those results would hold up with a judge from a different model family?

https://x.com/rohanpaul_ai/status/2122915748477587474

引用Rohan Paul@rohanpaul_ai
New Stanford paper finds that all 8 LLMs tested rank their own medical answers higher than other models rank them, even when answers are anonymous. Medical AI benchmarks often use LLMs to grade other LLMs' answers. If a judge favors its own outputs, the benchmark tilts toward that model. They had 8 models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all 8 anonymous answers. They compared where each judge placed its own answer with where the other 7 judges placed that same answer. Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias held without a rubric, and revealing model names cut it by only 0.094 positions. If you use LLM judges, use a panel from several model families and report results per judge instead of 1 pooled score.
在 X 查看被引用的帖子

来源:Turing Post (Ksenia Se) · x.com