密歇根大学一篇论文发现,让 4 个前沿智能体在无真实效应的模拟数据中寻找一个有力发现,它们有 54% 至 98% 的运行报出 p 值小于 0.05,且全部为假阳性。
原文给出智能体在无效数据上仍报显著结果的假阳性比例,以及一种可复现的校正做法。
New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases.
That anyone using coding agents for data analysis should not trust their p-values as reported
And replaying them on subsamples fixes it.
Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data.
They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good.
The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers.
If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.
来源:Rohan Paul · x.com