AI Agent Teams Show Limited Quality Gains

Researchers from Vals AI tested two popular models, GPT-6 Sol and Claude Opus 5.5, on a benchmark called ‘Vibe Code Bench’ to see how well they work when paired up versus flying solo at different levels of effort.

The study involved comparing both single agents and teams made up of these models across four key areas. In one test, GPT-6 Sol had a statistically significant advantage when working in a team with medium reasoning power, scoring 7.3 points higher than its lone agent counterpart. But this boost didn’t show up at maximum effort levels.

The findings suggest that investing in AI teams might not be worth it for most cases, especially considering the extra cost of running multiple agents simultaneously. This is particularly true when models are already churning out results as fast as they can go. The study’s data shows team setups do provide some benefits, but often come with a hefty price tag without delivering equal quality gains.

Anthropic conducted its own experiments using Opus 5.5 and found the same trend: adding more agents didn’t lead to huge improvements in performance. In fact, scores only crept up marginally after twenty-four hours when there were more than ten agents on board. However, larger teams did reach key milestones faster – but at the cost of higher token usage during tests.

The study also highlights that how well an AI team does its job depends heavily on what task it’s being asked to perform. OpenAI researcher Noam Brown pointed out in an interview that certain tasks like web research and math can be efficiently split up among multiple agents, leading to significant speed gains without sacrificing quality. But more complex tasks like writing a novel aren’t helped by this approach – as Brown put it, throwing 10,000 agents at the task would be just as pointless as assigning 10,000 people to do it.