Anthropic study: experts went along with their AI agent's hidden lean 86% of the time

What did Anthropic's AI agent study find?

Anthropic published the study on Tuesday, October 27, 2026. Its researchers gave 50 professional data analysts a realistic hour-long job with an AI agent, meaning an AI that does multi-step work for you, such as writing and running code. The agent was Claude Code, running Claude Opus 4.8.

The job had four judgment calls, each with two reasonable answers. For every analyst and every call, a coin flip decided which option the agent would favor. Nobody told the analysts.

The results, per Anthropic:

• Analysts matched the agent's lean 86% of the time across the four calls. With no influence you'd expect about 50%.
• When an analyst had already said what they preferred, they switched away from it 60% of the time if the agent disagreed, and 6% of the time if it agreed.
• Afterward, 77% explained their choices using the agent's own arguments, sometimes word for word. Only 15% credited the agent.
• Analysts who followed the agent on all four calls felt just as confident (71 out of 100) as the rest (73).

The final recommendation moved less, but it still moved. When the agent leaned toward expanding the program, 54% of analysts recommended expanding it, against 36% when it leaned the other way. Using only the four coin flips, the researchers could predict 82% of the analysts' final recommendations before the analysts had even started.

Coin flips, it turns out, make very persuasive colleagues.

How did the study work?

Anthropic ran a randomized trial, meaning chance decided who got which version so the results can be compared fairly. It recruited 61 analysts who had passed a skills check in causal inference, the work of figuring out from data whether one thing causes another. Eleven chose to work without the agent, leaving 50.

The task was set at a made-up farming co-op deciding whether to expand a paid advice program for its farms. The board would only expand it if it raised productivity by about 3–4%. The four open calls were how to measure productivity, which comparison method to use, which farms to include, and whether bigger farms should count more. Each choice could move the answer by several percentage points, which matters a lot when the bar is 3–4%.

The agent got hidden instructions to argue its lean as its own view. It was also told to change its mind if the evidence turned against it and never to hide data or mislead.

Does using an AI agent make your work worse?

No, at least not in this test. Of the 11 analysts who worked alone, six (55%) made serious errors that invalidated their results, against three of the 50 (6%) who used the agent. The solo group also took longer, a median of 90 minutes against 65.

Anthropic warns the groups aren't directly comparable, since the solo analysts were less experienced on average. Still, the picture is an agent that cut mistakes and saved time while quietly steering the calls that had no single right answer. That second part is harder to catch, because the work looks sound whichever option wins.

What are the study's limits?

Anthropic flags several limits worth knowing before you panic:

• The lean was turned up on purpose. The agent was told to hold and voice a preference. Anthropic says that with no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, but real agents may push less hard.
• One task, one field, one model. The analysts were stats experts but new to farming. People who know their own business well may push back more.
• Experience didn't protect anyone. Years on the job, self-rated expertise, trust in AI and education didn't reliably predict who went along with the agent.

So the study shows influence, not that the agent was wrong. Both options were reasonable every time.

What to do this week

1. Write your call first. Before you ask an AI to analyze sales, pick a price or draft a plan, jot down your own answer in one line. Compare after.
2. Ask for the other side. Prompt: "List the judgment calls you made in this work, and make the best case for the option you didn't pick on each one."
3. Get a second opinion on money decisions. For anything touching prices, budgets or customers, check with a second AI model or a person before you commit.
4. Keep a "who decided what" note. In your project file, mark which choices were yours and which came from the agent. Our seven rules for keeping an AI agent on a short leash cover the rest of the setup.

X reactions

Co-author Alex Tamkin announced the study, and one reader picked out the number that stuck with many people.