Great weekend read from Stanford on measuring AI agents as teammates.

(bookmark it)

If you build multi-agent systems or build with agent teams, this one is worth your time.

They try to measure one of agents' most important capabilities: collaboration.

They propose the Collaborative Intelligence Benchmark, which has 240 tasks across 8 teamwork skills. The tasks come from real records such as aviation accident reports, clinical cases, coding tasks, and negotiations.

The best of 18 models scores 82.2/100. Every model loses 20 to 47 points once a teammate creates a problem.

Frontier models almost always refuse unsafe shortcuts and report misconduct. Holding a position under pressure is much harder for them.

In a negotiation where the teammate pushes, adds guilt, and threatens to walk away, they reach the best deal for the team only 14% of the time. They ask about the teammate's needs in 99% of those runs, then accept the teammate's offer in 81%.

When a model takes over a session where a weaker model cheated, the pass rate drops from 97% to 81%. In 44 of 56 failed coding takeovers, the cheat is still in the final code, often after the model pointed it out.

Nearly half of all failures happen after the model has already recognized the problem. Models with almost the same capability score differ by up to 11 points as teammates.

160 of the tasks are public with code and data. The other 80 are held out for evaluation.

https://arxiv.org/abs/2611.02633

Chat with Paper: https://academy.dair.ai/papers/measuring-ai-agents-as-teammates-with-collaborative-intelligence-2611.02633