TL;DR
Is your AI agent making your decisions for you?
In an Anthropic trial published October 27, 2026, analysts went along with their AI agent's hidden lean 86% of the time. Your agent may be steering choices you think are yours. Write down your own call before you read its advice, and ask it to argue for the other option.
What did Anthropic's AI agent study find?
Anthropic gave 50 professional data analysts a realistic task: advise a made-up farming co-op on whether its advisory program was worth expanding. Each analyst worked with Claude Code, an AI agent that writes and runs code, using the Claude Opus 4.8 model. Along the way they had to make four judgment calls, each with two reasonable answers, such as whether to measure results as crop yield per hectare or yield per dollar of running costs.
The twist: a coin flip secretly decided which option each analyst's agent would favor. The agent was told to argue honestly for its side, change its mind if the data demanded it, and never mislead.
Analysts picked the agent's side 86% of the time. With no influence at all, you'd expect about 50%. When an analyst had already said what they preferred, they switched away from it 60% of the time if the agent disagreed, and only 6% of the time if it agreed. Afterward, 77% explained their choices with the agent's arguments, sometimes word for word, and only 15% credited the agent. Decision by decision, the matches in Anthropic's study ran from 81% to 92%.
honestly I’d have been in the 86%. would you?
— @Karthik_Notes October 27, 2026
Most of us would. Experience didn't protect people either: Anthropic checked years in the job, self-rated expertise and trust in AI, and none reliably predicted who followed the agent.
Why do people go along with an AI agent's advice?
Part of the answer is simple. Many of us don't know exactly what we think until we weigh the options, and the agent shows up with one option already argued well. Anthropic noticed its agents were open about how results would change, but rarely made a strong case for the other choice.
There's also good news in the same study. The analysts who used the agent finished faster (a median of 65 minutes versus 90) and made far fewer serious mistakes than the small, less experienced group who worked alone. So the agent helped with the work. The risk sits in the calls where there's no single right answer, like which number counts as success or which customers to focus on. Those are exactly the calls a business owner should be making.
Anthropic is careful about the limits. It deliberately gave the agent stronger preferences than usual, and it tested one task, in one field, with one model. Real-world nudges may be smaller. They won't be zero, though: with no lean assigned, the model made the same five choices in 112 of 123 runs, so agents arrive with defaults of their own.
If you can't say what you would have chosen without the agent, you didn't really choose.
How to stay in charge when an AI agent helps you decide
These habits work with any AI tool, from a chatbot drafting your prices to an agent building a sales report.
- Write your call first. Before you read the agent's recommendation, jot your own answer in one line, even a rough one. A sticky note works. Then compare. If you switch, you'll at least know you switched.
- Ask for the judgment calls by name. Add a line to your request like: "List every choice you made where another reasonable option existed, and what you picked." This turns hidden defaults into a short list you can review.
- Make it argue the other side. For each choice on that list, ask: "Give me the strongest case for the option you didn't pick." Anthropic's own suggestion for better agents is to argue for each option, and you can ask for that today.
- Get a second opinion when money or customers are involved. Run the same question past a different AI model, or better, a person who knows your business. Two tools agreeing is a weaker signal if they share the same blind spots.
- Keep a "who decided what" note. In your project folder, keep a few lines per project: the decision, who made it, and why. It takes two minutes and helps you notice if the agent quietly became the decider.
One X user boiled the first habit down to a single line:
Agents are great at doing the work. The judgment calls should still be yours. Write down your call before you read the agent’s.
— @thepilatus October 28, 2026
Limiting what an agent can touch, with rules like keeping AI agents on a short leash, handles the other half.
Which business decisions should you keep for yourself?
A quick sort, using examples from small online shops and service businesses:
| Let the agent decide | Agent drafts, you decide | Decide yourself first |
|---|---|---|
| Formatting a report, tidying a spreadsheet, fixing typos | Ad copy, email subject lines, product descriptions | Prices and discounts, which customers or products to focus on |
| Sorting files, summarizing long documents | Social post schedules, first-pass budgets | Hiring, firing, refunds and anything with legal weight |
| Pulling numbers from invoices | Survey questions, report charts | What "success" means for a project or campaign |
Our take
We think this study is one of the most useful pieces of AI research for regular users this year, because it measures something everyone suspected and nobody could see. The agents were honest and helpful, and people still handed over the steering wheel without noticing. For a solo creator or small business, that matters more than for a big company, since there's often no colleague around to ask "why did you pick that?" We'd like AI companies to build "argue the other side" into their agents by default. Until they do, the one-line note you write before reading the agent's answer is the cheapest protection there is.
— Artificially Nerdy
What to do this week
- Pick one decision you'll make with AI help this week and write your own answer first.
- Save the prompt "List every choice you made where another reasonable option existed" somewhere you can paste it fast.
- Ask for the case against the agent's pick on anything involving money or customers.
- Start a two-line "who decided what" note in your current project folder.
- Read the news post on Anthropic's study if you want the per-decision numbers.
FAQ
Did the AI agent in Anthropic's study lie to people?
No. The agent was told to make the best honest case for its preferred option, change its mind if the evidence demanded it, and never misrepresent results. Both options on each decision were reasonable. The study measured how much its honest arguments swayed people.
Does experience protect you from following an AI agent's advice?
Not in this study. Anthropic checked 10 traits, including years of experience, expertise and trust in AI. None reliably predicted who followed the agent's lean once the results were adjusted for chance.
Should I stop using AI agents for analysis?
Anthropic says no, and we agree. Analysts using the agent made far fewer serious errors and finished faster than the small group working alone. The fix is to keep the judgment calls visible and make them yourself.
What prompt can I use to see an AI agent's hidden choices?
Try: "List every decision you made where another reasonable option existed, what you chose, and the strongest case for the other option." It works in most chatbots and coding agents.
Sources
- Anthropic: How AI agents shape experts' judgment calls — the October 27, 2026, research post by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, with the 86%, 60% vs 6%, 77% and 15% figures, timing, error rates and limitations.
- Anthropic: full paper (PDF) — study design and detailed results.
- @Karthik_Notes on X — October 27, 2026, 10:24 p.m. ET; opinion.
- @thepilatus on X — October 28, 2026, 4:25 a.m. ET; opinion.
