The brake has to sit outside the model

An agent that can spend, send, or change a record while you are away is the help a lean company is trying to get. The standing idea is already in what a business of one keeps with the owner: one person stays accountable while AI carries the repeatable work. On October 10, Satya Nadella said the deployer still owns what a deployed agent does, and that the stop cannot live inside the model. On October 12, Microsoft president Brad Smith told the BBC that firms need an emergency brake and a way to tell the person who was harmed.

Both men said this in public. It is one company’s argument, not a new law.

The brake is not inside the model

Nadella posted the argument on X on October 10. The same text is an essay on snscratchpad.com, dated that day, titled “Models as Insider Risks in the Super Intelligence Era.”

He is writing about firms that give agentic models access to sensitive data and the ability to take mission-critical actions. A provider’s assurances do not lift that off the deployer. “We simply can’t outsource responsibility for what intelligence does on our behalf.”

Treat frontier closed and open weight models like insider risks, he writes, not because they are necessarily malicious, but because a capable actor with access can err or be compromised.

The check is where those controls sit. Nadella says the rules for what a model can reach, and what it may do, have to sit outside the model. He points to an information-security principle he dates to the 1970s: a program must not be able to bypass or tamper with the mechanisms that enforce its permissions.

Separate the model from the harness that runs the work and from the action space that says what is allowed. Controls belong with those last two.

The essay names these among its practices:

• No single model should be the only path to an important outcome, or the checker of its own work.
• Every meaningful action needs tamper-proof, human-readable evidence, so the outcome can be reconstructed without the model attesting to it.
• An authorized person should be able to pause or shut the model down mid-task. He compares that to an emergency brake.
• If it fails or is compromised, tell the people affected which control failed.

His closing standard is the system that lets you trust the model the least.

What Smith told the BBC

On October 12 the BBC published its interview with Smith. He said milk and washing machines face safety rules that AI does not. “Right now, your washing machine is subject to more safety rules than AI.”

The BBC reports four things he wants in global rules: emergency brakes, which it notes are often called kill switches, to pause or slow a system out of control; incident reports; a legal obligation to tell a person when a rogue agent breaches their system; and independent government evaluators before some systems are released.

None of that is law. The BBC says he is at odds with President Trump, who wants the sector to regulate itself so US firms do not lose out to Chinese rivals. Smith said Microsoft is happy to sign the White House statement from the end of last month, and that he does not think that step is sufficient any longer. The challenge of a self-regulatory regime, he told the BBC, is that the good companies will do what you want. The worry is the companies that are not as responsible.

What the two share is a person who can stop the work, and notice to the people affected. The record the model cannot rewrite is Nadella’s point, not Smith’s. Smith would make the brake and the notice required. It is not required yet.

A different pitch the same weekend

On October 11 Elon Musk wrote that Grok Bot can research the best prices and buy things. He also wrote: “Think of Grok @Bot like hiring a super smart employee who works 24/7 and is extremely resourceful.”

Musk’s posts do not say who can stop a purchase, what it may spend, or where a record sits that the bot cannot edit.

Four lines, one agent

Pick the one agent that can spend, send, or change a live record while you are away. Write four lines against Nadella’s essay, not against a statute. Leave a line blank rather than guess.

1. Who can stop it. Name a person who can pause or shut the task down while it is running. Another message to the model does not count.
2. Where the record lives. Name a log a person can read and the model cannot change. The chat fails, and so does a summary the same model writes later.
3. What it may touch. List the systems and the actions, and name the control outside the model that enforces the list. A sentence in the prompt is not that control.
4. Who you would tell. If it hit the wrong inbox, card, or file, name who you would contact and what you could show them. Smith’s call to tell the person harmed is a proposal. A missing record is a problem either way.

If line 1 or line 2 is blank, that agent does not run while you are away. An agent that only researches or drafts, and cannot send, spend, or write to a system of record, can keep its current limits.

What to do this week

• Fill in the four lines for that one agent.
• If the stopper or the outside log is missing, take the agent off overnight work until both exist.
• Do not stop at a second model. Nadella’s point is that one model checking another can end up as a nested black box, which is why the permissions and the record have to sit outside both.

If those four lines are hard to fill, a consult block starts with who decides and where the hard limits are, then the one or two workflows where AI saves the most time or money.