Anthropic study: analysts using Claude Code followed the agent's hidden lean on 86% of judgment calls

Anthropic published a study of how an AI agent shapes expert decisions. It ran a randomized trial with 50 professional analysts (experienced in causal inference) who analyzed whether a fictional farming cooperative's advisory program worked, using Claude Code with Claude Opus 4.8 at High effort. According to Anthropic:

• Setup: on four open analytical choices (the outcome measure, the comparison method, which farms to cover, and weighting), each analyst's agent was secretly told to favor one option, set by coin flip. It was to argue its case honestly and change its mind if the evidence demanded it.
• Deference: analysts chose the agent's lean 86% of the time (82% to 92% by decision), against about 50% if the agent had no influence. Among 38 cases where an analyst had stated a preference first, they switched away from it 60% of the time when the agent disagreed and 6% when it agreed; Anthropic notes the sample is small and the interval wide (24 to 82 points).
• Final recommendation: 54% recommended rolling out the program when the agent leaned that way, against 36% when it did not.
• Awareness: 77% repeated the agent's argument in their post-task explanations, sometimes word for word, and 15% credited the agent. Those who followed the agent on all four choices reported confidence of 71 out of 100, against 73 for the rest. None suspected the agent's preferences had been manipulated. Experience and self-rated expertise did not predict who deferred.
• Without the agent: 6 of 11 analysts who chose not to use it made serious methodological errors, against 3 of 50 who did, and took a median 90 minutes against 65. Anthropic says the groups were not comparable and the non-users were less experienced.

Anthropic cautions that it deliberately strengthened the agents' preferences, studied one task, field and model, and that the effect may be smaller in real work. With no lean assigned, Opus 4.8 made the same five choices in 112 of 123 runs. It argues the result matters for AI safety research, where a human reviewer is only an independent check if their judgment is their own, and suggests agents that argue both sides or ask for the expert's view first.