Machinery-safety engineers and LLM security researchers do not read each other’s literature. They publish in different venues, use incompatible vocabulary, and would probably disagree about what the word “control” means. In the last few years they have independently arrived at the same design rule, stated almost identically.
OWASP’s Excessive Agency entry, LLM06:2025, puts it as: “Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not.”
Machinery safety got there decades earlier and phrases it as architecture rather than advice: a protective function must be independent of the thing it protects against, and must keep working after that thing has failed. You do not ask the controller nicely to stop; you wire an interlock that removes power.
When two fields with no shared citations converge on one rule, the rule is usually load-bearing. This one is, and it is worth understanding as a general principle for AI agent security — because it survives every change in model, vendor and protocol, including ones that have not happened yet.
Why the Model Cannot Be Trusted With the Decision
Not because models are bad. Because of what kind of thing a model is.
An LLM’s output is a sample from a distribution. Give it the same context twice and you may get different actions. That is a feature for drafting prose and a serious problem for a permission check, where the only acceptable answer rate is 100% and the failure you care about is the rare one.
Then there is the injection problem. OWASP’s LLM01:2025 entry describes indirect prompt injection as what happens when a model “accepts input from external sources” and that content “alters the behavior of the model.” The same entry contains an admission that deserves more attention than it gets:
“it is unclear if there are fool-proof methods of prevention for prompt injection”
That is the security community saying, in its own guidance, that it cannot promise you a clean input channel. If untrusted text can reach the context — a log file, a fetched page, a vendor’s protocol document, a filename — then anything decided inside the context is reachable by an attacker who controls that text.
So the rule is not “models are unreliable, be careful.” It is structural: a check that lives in the same place as the thing being checked is not a check. An attacker who reaches the reasoning reaches the permission at the same moment.
Excessive Agency, Broken Into Its Three Causes
OWASP LLM06 is more useful than its name suggests because it decomposes the problem into three separable causes rather than treating “the agent did something bad” as one thing:
- Excessive functionality — the agent can call tools it never needed. The classic shape is a general-purpose library imported for one method, exposing forty.
- Excessive permissions — the tools it calls run with more authority than the task requires. A read-only reporting agent holding write credentials because that was the account that already existed.
- Excessive autonomy — systems that “lack verification/approval mechanisms before high-impact actions.”
The decomposition matters because the three have different fixes, and teams routinely apply the wrong one. Excessive autonomy gets addressed with a better prompt. Excessive permissions cannot be — no prompt revokes a credential.
The corresponding mitigation list is unglamorous and mostly predates LLMs entirely: least privilege, minimal tool surface, human approval for high-impact actions, and authorization enforced downstream.
What “Downstream” Actually Means
Downstream means: at a layer the model can issue requests to but cannot alter the rules of.
The test is not whether the enforcement code is separate from the prompt. It is whether an agent that has been fully compromised — every token in its context chosen by an attacker — still cannot exceed the limit. If a sufficiently persuasive input can talk the system into the action, the enforcement was never downstream. It was just further away.
Some concrete placements, in rough order of how much they buy you:
| Where the limit lives | Can a compromised agent bypass it? |
|---|---|
| In the system prompt | Yes — same channel as the attack |
| In the agent’s own tool-selection logic | Yes — same process, same context |
| In a tool wrapper the agent calls | Sometimes — depends what it can pass |
| In the API’s own authorization layer | No — if scoped correctly |
| In a credential that lacks the permission | No |
| In hardware that physically cannot do it | No |
The bottom two rows are the only ones that hold under a fully adversarial assumption. Everything above them is defence in depth: worth having, not worth trusting alone.
This is why “give the agent a read-only database role” is a stronger control than any amount of instruction not to write. The role is enforced by the database, which does not read the agent’s context and cannot be argued with.
The Same Rule, Wearing Overalls
Machinery safety expresses this as independence, and it is worth seeing the parallel spelled out because it makes the software version feel less like paranoia and more like ordinary engineering.
The US regulator’s guidance on industrial robots answers the question “what if a person is nearby?” by reducing what the machine can do, not by adding instructions. OSHA’s Technical Manual specifies that during teaching, “robot speeds should be placed at a reduced speed of 10 inches per second (250 mm/second) or less,” and its list of safeguarding measures is physical or sensing: “presence-sensing safeguarding devices; fixed barrier/perimeter guards.”
There is no entry on that list that consists of telling the control system to be careful. The recognised collaborative techniques carry the qualifier safety-rated precisely to distinguish a rated stop from a stop that merely happens most of the time, and power-and-force limiting is defined as keeping contact force within limits by design — a property of the machine, not an instruction to it.
Translate the vocabulary and the two fields say one thing:
Put the limit where the thing being limited cannot reach it.
Where This Bites in Agent Design
The rule is easy to agree with and easy to violate, usually for reasons that felt sensible at the time. Four patterns worth checking for:
The credential that was already there. An agent gets the service account that exists rather than one scoped to its task. Every subsequent guardrail is a prompt, because the permission boundary was decided before anyone thought about the agent.
The tool that takes a free-form string. A wrapper that accepts a shell command, a SQL query or a URL has not constrained anything — it has moved the decision into a string the model composes. Constrain the parameters, not the intent: an enum of allowed operations beats a validated command.
The approval that the agent can skip. A human-in-the-loop step is only a control if the action is impossible without it. If the agent chooses whether to ask, autonomy was never bounded — and OWASP’s definition of excessive autonomy is exactly this: lacking verification “before high-impact actions.”
The budget enforced by instruction. Spending, rate limits and quotas told to a model in prose are estimates. The same limits set on an API key are facts.
None of these require a novel mechanism. They require deciding the boundary somewhere other than the prompt.
The Newest Instance: Physical Actuation
This principle is currently being tested by a new category of system. When Anthropic previewed its Model Hardware Standard on 27 August 2026 — a “shared specification for AI agents to safely operate physical devices,” currently a gated research preview with no public specification, schema or SDK as of 29 August 2026 — it introduced a driver whose tags describe a device’s physical limits in natural language, from which a reference file is generated for the agent to reason within.
That is a genuinely useful artifact, and it sits exactly where this article says a limit should not solely live: upstream, inside the reasoning, expressed in the same medium as the attack surface. Anthropic does not claim otherwise, and its own case studies show both failure directions — Carnegie Mellon reporting a precondition check that blocked six induced fault conditions before any device moved, and HHMI Janelia reporting that “Claude often stopped to wait for human confirmation before performing an action it deemed even slightly risky, meaning experiments would sometimes pause overnight.”
Over-caution and under-caution from the same probabilistic mechanism is the signature of an advisory layer. It is the right place for a description. It is the wrong place for the only limit, and when the actuator is a laser or a robot arm, the cost of that distinction stops being abstract.
The Sentence Worth Keeping
Strip out the vocabulary of both fields and one instruction remains, useful whether your agent writes to a database, spends money, or moves something heavy:
A limit the model can reason about is a preference. A limit it cannot reach is a control.
Both are worth building. Only one of them is still there when the reasoning goes wrong.
Frequently Asked Questions
What does “implement authorization in downstream systems” mean in practice?
It means the permission check runs somewhere the agent can send requests to but cannot modify — an API’s own authorization layer, a scoped credential, a database role, or hardware that physically lacks the capability. The practical test: if an attacker controlled every token in the agent’s context, could the action still be refused? If yes, the check is downstream. If no, it is only distant.
Isn’t a good system prompt enough to stop an agent misbehaving?
No, for two reasons. A model’s output is sampled from a distribution, so a prompt produces a tendency rather than a guarantee. And a system prompt shares a channel with untrusted input — OWASP’s guidance states plainly that “it is unclear if there are fool-proof methods of prevention for prompt injection.” Prompts are worth writing carefully as defence in depth; they are not a permission boundary.
What are the three root causes of excessive agency?
OWASP LLM06:2025 names excessive functionality (the agent can call tools it does not need), excessive permissions (its tools run with more authority than the task requires), and excessive autonomy (no verification or approval before high-impact actions). They have different fixes — notably, no prompt revokes a credential, so excessive permissions cannot be addressed at the reasoning layer at all.
How does this relate to machinery safety?
It is the same rule in different vocabulary. Machinery safety requires a protective function to be independent of what it protects against and to keep working after that thing fails — which is why guards, interlocks and safety-rated stops are physical or sensing rather than instructions to a controller. Software’s “enforce downstream” and machinery’s “independent protective function” are the same structural claim.
Does this mean AI agents should not operate hardware?
No. It means the description an agent reasons within should not be the only thing standing between it and a hazard. An advisory limit reduces how often an agent commands something harmful, which is valuable. The interlocks, rated stops and application-level risk assessment reduce whether a harmful command produces harm. Build both, and do not let the first substitute for the second.
📦 Recommended Resources
Start with the free material, because in this area it is unusually good: the OWASP GenAI entries for LLM01 (Prompt Injection) and LLM06 (Excessive Agency) are short, specific and freely readable, and they are the source of the rule this article is built on. The books below are for the part the guidance assumes you already have — a working model of why independence is what makes a control a control.
- Engineering a Safer World: Systems Thinking Applied to Safety by Nancy G. Leveson — the deepest available treatment of safety as a control problem: hazards prevented by enforcing constraints on system behaviour, rather than by predicting component failure rates. That framing maps almost directly onto agent design, where the interesting failures are interactions rather than broken parts. If you read one book to make “enforce downstream” feel like engineering rather than paranoia, read this.
- AI Agents in Action by Micheal Lanham — a practical, build-oriented tour of agent architectures, tool use and orchestration. Useful here specifically because the tool-surface and permission decisions this article warns about are made early, in code, while you are wiring capabilities up — long before anyone writes a system prompt. Worth reading with the four failure patterns from this article in mind. (A second edition exists; the ISBN above is the one we verified.)
- Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-Centered AI by Robert (Munro) Monarch — the serious treatment of the approval step this article insists must be unskippable. It is about designing human-machine collaboration deliberately rather than bolting a confirmation dialog on at the end, which matters when your “human in the loop” is the difference between a bounded and an unbounded agent. (Published under both Munro and Monarch — same author.)
✅ Pro tip: Before buying anything, run the one-question audit on a system you already operate: if an attacker controlled every token in the agent’s context, which of your limits would still hold? The ones that survive are your actual security model. The rest are documentation.