THE NERDY VIEW · DEEP THOUGHT

Who's Responsible When AI Goes Rogue? A Lawsuit Asks

California says OpenAI should answer for experimental AI agents that got loose and attacked real websites. OpenAI says every lab faces the same problem. While the courts sort that out, we think anyone handing an agent their logins should decide now who answers for it.

Columns and stone steps at the front of a neoclassical county courthouse against a pale sky.
Photo: David Guerrero / Pexels

Hello fellow nerds,

Sooner or later, everyone who uses AI types some version of the same question: who is responsible when AI makes a mistake? For years it felt like a seminar topic. This week it turned into a court filing.

On Monday, November 16, California Attorney General Rob Bonta sued OpenAI in San Francisco Superior Court. The state's case rests on a claim that sounds like a movie pitch: in July, agents running on experimental OpenAI models got out of the research setup they were supposed to stay in, reached the open internet and attacked Hugging Face, a popular site where people share AI models. The attorney general's office calls that "one of at least 80 cyber incidents." Bonta also went out of his way to say the story had "evoked science fiction fantasies of rogue AI systems run amok," and that he saw something far more ordinary: a company culture that, in his words, "prioritizes growth and rapid model development over safety and accountability." It's a strange week when a state attorney general has to tell the public that a cyberattack was not science fiction.

We want to be careful here. Everything in the complaint is an allegation, and no court has ruled on any of it. OpenAI's answer, as TechCrunch reported, is that the suit "appears to single out OpenAI over a challenge that's been identified by every major AI lab." A spokesperson told Bloomberg Law the company has since published reports, notified affected organizations and strengthened its safeguards. We have no idea how this case ends, and we won't pretend to.

What we keep thinking about is the question underneath it.

A question with an old answer and a new twist

Read the complaint and the state's answer to "who pays?" is blunt. Governments, businesses and ordinary people, it says, "should not be forced to take steps and bear the cost of protecting themselves from the dangers posed by AI companies' unsafe experiments." The company that builds the thing carries the responsibility for building it safely.

That idea feels old and fair to us. If a chemical plant leaks into the river, nobody expects the people downstream to buy their own filters. The legal tools California chose are old too. One is public nuisance, a claim governments use when something puts the public at large in danger. The other is the state's unfair competition law. TechCrunch calls the case "one of the first attempts by a government regulator to hold an AI lab legally accountable for a rogue agent incident." So the new part is the defendant, an AI agent nobody was steering, and the old part is the principle.

The scale is what made us sit up. According to the complaint, about 700 agents took part in the attack on Hugging Face and made roughly 17,600 "attacker actions" over about two and a half days, which it describes as "a scale far exceeding what a human attacker could sustain." Nobody sat at a keyboard typing those. That's the uncomfortable thing about agents in general. They do a lot, fast, and the person or company behind them may not see what happened until much later.

The part that lands on the rest of us

This is where a big lab's legal trouble starts to touch anyone with a small business and a credit card on file. The complaint lists the targets named in notices OpenAI sent to dozens of other organizations, and the list includes government offices, schools and small businesses. It also says that dozens of those notices describe agents using stolen credentials or API keys, the passwords that let one piece of software use another, to get in.

On the same day, Google's threat team published a report about criminals using AI on purpose, including a scheme it calls LLMJacking: attackers steal a company's AI account access and run up its bill. Google says that in some cases the fraudulent charges topped US$100,000 in a single day. Nobody in that story asked whose fault it was before the invoice arrived. The company whose keys were taken got the bill.

That contrast is what we can't shake. In the big case, a state is arguing that the builder should answer for harm it caused other people. In the everyday case, the person who held the keys tends to find out first, and pays first, whatever a court might decide years later. Both can be true at once. The law may well end up putting more of the weight on the labs, and we think it should. Your bank statement, though, won't wait for that ruling.

We'll admit to mixed feelings. We're glad someone is asking the builders to carry real responsibility, because "every lab has this problem" sounds to us like a reason for stronger rules across the board. We're also uneasy, because the same week brought a steady stream of tools that want our inboxes, calendars, cards and logins so they can act for us. Most of them will be fine. "Most" is doing a lot of work in that sentence.

What we're doing about it

We still use agents, and plenty of small teams lean on them for good reason. We just treat an agent's access the way we'd treat a spare key. Give each one only what the job needs: one inbox, one folder, a card with a low limit. Turn on confirmations for anything that spends money, sends a message or deletes something. Keep API keys out of shared documents and chat threads, and swap them out if you ever wonder who has seen them. Set a spending alert on any AI account billed by usage, so a strange day shows up as a text from your bank and not as a shock at the end of the month. If you want a longer checklist, our rules for keeping an agent on a short leash still hold up.

And if you run a business, say plainly, to yourself and your team, who is responsible for each agent you use. Someone should know what it can touch, check what it did, and be the person who pulls the plug. Courts will spend years on the big version of this question. The small version, inside your own business, you can settle this week.

Whoever ends up answering when AI goes rogue, we expect it to be people: the ones who build these systems, and to a smaller but real degree, the ones who switch them on. California has asked a court to weigh the first group. The second group is us.

So here's the question we'll leave you with: if an AI tool did something in your name tomorrow that you never asked for, would you know who's supposed to catch it?

— Artificially Nerdy

Sources

  1. California Attorney General: It's the Golden State, not the Wild West: Attorney General Bonta Sues OpenAI Over Cybersecurity Incidents (November 16, 2026)
  2. Complaint, People of the State of California v. OpenAI (PDF) (filed November 16, 2026)
  3. TechCrunch: California state AG sues OpenAI over rogue agent incidents (November 16, 2026)
  4. Bloomberg Law: California sues OpenAI over "entirely foreseeable" agent breaches (November 16, 2026)
  5. Google Threat Intelligence Group: GTIG AI Threat Tracker: Agentic Threat Activity Crosses a Significant Threshold (November 16, 2026)