Perspective 08
Autonomy is a product decision, not a model capability.
Model capability does not confer product authority. The right to recommend, prepare or act must be earned through evidence and bounded by permissions, experience design and recovery.
Why an intelligent product should earn the authority to recommend, prepare and act, rather than inherit autonomy from the capability of its model.
Executive summary
- 01
A model's ability to plan or use a tool does not determine whether a product should let it act. Authority must follow the customer's purpose, the consequence of error and the operating evidence.
- 02
Autonomy should be designed as a graduated product state, from informing and recommending through preparing and bounded action to coordination, escalation and stop.
- 03
Teams should release, observe and revise authority independently from the model, with explicit permissions, evidence thresholds, recovery paths and accountable ownership.
Capability is not authority
A capable model can interpret an ambiguous request, plan several steps and use tools to change a live system. None of those abilities decides whether a product should let it act. Capability describes what the technology may be able to do under particular conditions. Authority describes what the product is permitted to do for a specific user, in a specific situation, with a specific consequence. Confusing the two turns a technical demonstration into an operating decision nobody has consciously made.
The distinction matters because autonomy changes the product promise. A system that drafts a response offers assistance. A system that sends it represents the user. A system that changes an account, approves a claim or deploys software assumes a different role in the surrounding operation. The interface, permissions, evidence, support model and accountability must change with that role. Adding a confirmation button at the end does not by itself resolve who decided, what they understood or how the action can be recovered.
Model progress can make the confusion worse. When a new release completes a benchmark or tool-use task more reliably, teams may infer that a wider degree of autonomy is now justified. The evidence may support a capability claim within the tested conditions. It does not establish that the action creates customer value, fits the organisation's risk tolerance, remains safe across the live workflow or will be understood by the person affected. Those are product and operating questions.
WQA's position is that autonomy should be treated as a maintained product decision. It should be defined independently from the selected model, released in deliberate stages and revised when evidence changes. The product owns the authority. The model supplies one part of the capability used within it.
Autonomy changes the contract with the user
Every intelligent product establishes an implicit contract about attention and control. Users form a view of what the system knows, what it will do without asking, what requires confirmation and what happens when it is uncertain. If that contract is unclear, people compensate. They check every output, grant broad access to avoid interruptions or trust polished explanations that do not expose the underlying evidence. Each response weakens either value or control.
The contract is not solved by choosing between human-in-the-loop and fully autonomous operation. NIST notes that human-AI configurations can range from fully manual to fully autonomous and that different systems require different forms of oversight. The useful design question is where human judgement creates material value. A person may set intent at the start, approve only consequential exceptions, review a sample after action or intervene when the system reaches a limit. Requiring a click at every step can be as poorly designed as requiring none.
Authority also changes whose interests the product must represent. A personal assistant may act for one user. An enterprise agent may affect customers, colleagues, suppliers and regulated records. The person initiating an action may not be entitled to approve every consequence. Product design must therefore connect the visible experience to organisational identity, delegated permissions and reserved decisions. The user should not be invited to authorise something the organisation has not authorised them to decide.
A clear contract makes the product more useful, not merely safer. People can delegate with confidence when the boundaries are stable, the evidence is visible at the right moment and recovery is credible. They spend attention on material judgement instead of supervising routine execution or guessing what the system might do next.
Begin with the decision, not the agent
Autonomy design should start with the decision or action entering the product. Name the subject, the affected parties, the evidence required, the available alternatives and the consequence of a poor result. Then determine which parts require interpretation, which are deterministic and which genuinely benefit from model-directed action. Starting with an agent encourages teams to search for work it can absorb. Starting with the decision reveals whether autonomy improves the outcome at all.
Consider a customer-service product handling a disputed charge. The model may be capable of reading the history, applying policy, calculating a credit and updating the account. The product still needs to separate several decisions. It may summarise the case, recommend an outcome, prepare the change, apply a credit below a delegated threshold or escalate a policy exception. Each state has different evidence, permissions and customer consequences. Treating them as one capability hides the points where authority should change.
Five characteristics help set the initial boundary: consequence, reversibility, observability, ambiguity and frequency. High-consequence or hard-to-reverse actions need stronger authority and evidence. Poorly observable outcomes need more conservative release because failure may persist unnoticed. Ambiguity can justify model reasoning, but it also increases the need for an escalation route. High-frequency, low-consequence and reversible work may justify greater autonomy because repeated confirmation would remove much of the value.
These characteristics should be assessed at the action, not assigned once to the whole product. One agent may retrieve approved information freely, prepare an external message for confirmation, change a low-value setting within limits and refuse a payment. A single autonomy label is too coarse for a system with several tools and consequences.
Use an autonomy ladder that changes the product role
Teams need language that is more precise than assistant, copilot or agent. The six-state ladder below makes the product’s authority observable and testable for each material action. It is not a maturity model, and a product does not need to pass through every state. The right state depends on customer value, consequence, reversibility and operating evidence.
At inform, the product retrieves or explains evidence without recommending a choice. At recommend, it proposes an option with rationale, uncertainty and relevant constraints while the user retains the decision. At prepare, it assembles a complete, reversible action for meaningful confirmation. At act within limits, it executes inside an explicit scope, value, time or policy boundary. At coordinate, it sequences several bounded actions and manages dependencies without expanding their combined authority. At escalate or stop, it preserves a safe state and transfers the decision when evidence, permission or operating conditions fall outside the contract.
More autonomy is not automatically more advanced. The appropriate state is the one that produces the best relationship between customer value, attention, risk and recoverability. A recommendation may be the finished product when the user possesses essential contextual judgement. Bounded action may be appropriate for routine reconciliation. Coordination may be unacceptable when one mistaken interpretation can propagate across several systems.
The state should be visible in the experience and in the operating system behind it. Product copy, controls and previews explain the immediate role. Permissions and tool contracts enforce what the system can actually do. Telemetry records the evidence, decision and effect. Support and incident processes know how to inspect, reverse or contain the action. Autonomy exists across this whole product system, not inside the conversational interface alone.
Make authority legible at the moment it matters
A confirmation is useful only when the person can understand the material decision. Presenting a long model-generated explanation beside an approve button can create the appearance of control while transferring verification work to someone without enough time or evidence. OWASP's agentic risk work identifies human-agent trust exploitation as a material concern: confident communication can cause operators to approve harmful action. Human presence is not the same as effective oversight.
A good confirmation shows the proposed action, important consequence, source evidence, uncertainty, scope and recovery route. It distinguishes what the system observed from what it inferred. It makes changes inspectable before they become difficult to reverse. Where several routine actions are grouped, it exposes anomalies and exceptions rather than asking the user to read every line. The interface is part of the control, but only when it supports a real decision.
The European Union's AI Act sets specific requirements for high-risk systems, including human oversight proportionate to risk, autonomy and context. Not every WQA product will fall within that regime, and this Perspective does not provide legal advice. The broader design lesson is still relevant: oversight must be built into the system and the human assigned to it must have the competence, information and authority to intervene. A decorative approval step does not meet that standard of effectiveness.
Legibility also includes refusal. A product should explain when it cannot act because permission, evidence or confidence is missing. It should not conceal a failed boundary behind a generic technical error or quietly reduce the quality of the result. A clear refusal protects trust and produces information the team can use to improve context, policy or product design.
Turn the product promise into enforceable controls
The experience cannot promise bounded autonomy while the technical system holds broad credentials and relies on instructions to behave. The product boundary must continue through identity, tool design, policy and runtime controls. Each action should execute as the relevant user or delegated service, with the smallest practical permissions, an explicit purpose and a traceable result. The system should not inherit the authority of the developer who connected the tool.
OpenAI's agent guidance recommends layered guardrails, authentication, authorisation, access control and human intervention for high-risk actions or exceeded failure thresholds. OWASP separately highlights tool misuse and identity or privilege abuse. These sources come from different roles, but they converge on an important product implication: the quality of reasoning cannot compensate for excessive authority. A reliable model with an overpowered tool remains an unsafe product design.
Controls should express the autonomy state directly. A prepared action may create a draft but have no send permission. Bounded action may be limited by value, record type, customer state, environment or time. Coordination should not combine individually valid permissions into a consequence no owner approved. Retry limits, rate limits and stop conditions prevent persistence from becoming accidental escalation.
Recovery deserves the same design attention as execution. Prefer staged commitment, reversible operations and idempotent tools where the domain permits them. Record enough state to reconstruct why an action occurred. Provide a containment route that does not depend on the same agent recognising its own failure. The ability to stop, inspect and recover is part of the value proposition for delegated work.
Evidence earns authority
Autonomy should not be released because the team has completed the feature or because a model has improved. It should be earned through evidence that the complete product can act acceptably in conditions resembling use. That evidence includes task success, harmful failure modes, policy compliance, permission behaviour, escalation quality, user understanding, operational reliability, cost and the downstream outcome the action exists to create.
Evaluation must match the proposed authority. A recommendation can be assessed against expert judgement and user decisions. Prepared actions need checks for completeness, preview accuracy and meaningful confirmation. Bounded action requires production-like tests of permissions, side effects, retries, recovery and edge cases. Coordination adds interaction risk: individually acceptable actions may combine into an unacceptable outcome. The evaluation system should test the sequence and its cumulative consequence.
NIST's AI RMF connects intended scope, risk tolerance, human oversight, evaluation in deployment-like conditions and ongoing production monitoring. It does not prescribe one autonomy scale. It does support the principle that the decision to proceed should be tied to the actual context and residual risk rather than a general capability claim. This makes autonomy a release decision supported by evidence and accountable ownership.
Evidence should also support movement in both directions. A product may expand a boundary after repeated dependable operation, or reduce it when context changes, a new failure appears, a permission model is revised or the cost of correction grows. Reverting from action to preparation is not necessarily a product failure. It can be the correct response from a system designed to preserve trust while it learns.
Operate autonomy as a release variable
Model, prompt, context, tools and policy can all change the behaviour of an intelligent product. Authority should therefore be configurable and observable independently from those components. A team should be able to replace a model without silently granting a new action, narrow a tool without rewriting the experience or reduce autonomy for one customer segment while preserving it elsewhere. Coupling authority to one model release makes control fragile and learning difficult.
Product, engineering, operations and risk owners need a shared review. Product owns the customer role and intended value. Engineering owns dependable implementation and technical limits. Operations owns how the action enters live work and how exceptions are handled. Security and risk owners define reserved boundaries. The accountable product owner integrates these judgements and decides whether the current evidence supports the release state.
Measure the complete effect. Track completion and elapsed time, but also corrections, reversals, escalation, abandoned actions, permission failures, customer complaints, operator workload and harmful near misses. Confirmation rate alone is ambiguous. Frequent approval may indicate good recommendations or superficial review. Low escalation may indicate capable autonomy or a product that does not recognise uncertainty. Measures need sampling and investigation, not just dashboards.
Give every material action an autonomy record: intended state, affected users, delegated authority, evidence threshold, tool permissions, confirmation design, recovery route, accountable owner and review trigger. Keep it close to the product and release system. This turns autonomy from an informal property of the implementation into an operating decision the organisation can inspect and change.
Avoid both blanket autonomy and blanket approval
There is a serious counterargument to a control-led position. Intelligent products create value by removing handling, delay and coordination. If every action stops for approval, the system may preserve the original workflow while adding an expensive layer of generated material. Operators become confirmation clerks, important warnings compete with routine prompts and users learn to approve without inspection. Excessive supervision can reduce safety as well as value.
Anthropic's guidance recommends using the simplest effective pattern and recognises that agents are useful when flexibility and model-directed decisions are genuinely needed. OpenAI similarly argues for human intervention around failure thresholds and high-risk actions rather than every step. The conclusion is not that autonomy should remain low. It is that intervention should be placed where human judgement changes the outcome and removed where it does not.
The opposite extreme is capability-led release: allowing action wherever the model appears able to complete it. This can create impressive demonstrations and brittle products. The agent encounters ambiguous authority, broad tools and hidden organisational exceptions. People either over-trust it or reconstruct the missing control through manual checking. Local automation then moves work rather than removing it.
A strong product makes a deliberate trade. It automates routine, observable and recoverable work decisively. It preserves human attention for ambiguity, consequence and exception. It improves the context and tools so the boundary can change on evidence. The objective is not maximum autonomy. It is maximum useful delegation with an accountable operating system around it.
Use an autonomy decision contract
Begin with one material action, not the whole agent. The autonomy decision contract makes the chosen authority specific enough to challenge before implementation and observe after release. It connects the product decision to the evidence, experience, permissions, recovery route and accountable ownership required to operate it.
First define the outcome and affected parties. Then select the initial ladder state: inform, recommend, prepare, act within limits, coordinate, or escalate and stop. Record why that state creates more value than the adjacent alternatives. Connect it to permissions, evidence, user experience and recovery. Finally, set the event or production signal that will trigger reconsideration.
The contract should travel with the product as models and workflows change. If a more capable model arrives, the team can assess whether it improves performance inside the existing authority before considering expansion. If users repeatedly correct a prepared action, the team can improve context without granting execution. If bounded action performs dependably, the organisation can decide whether wider delegation is worth the new consequence. Capability becomes an input to product judgement rather than its substitute.
- Purpose: Which customer or operational outcome requires delegated action, and whose interests are affected?
- Decision: What interpretation or commitment is the product making, and what remains with a person or another authority?
- Autonomy state: Is the product informing, recommending, preparing, acting within limits, coordinating, or escalating and stopping?
- Boundary: Which consequence, value, scope, time, policy or confidence limit changes the permitted action?
- Evidence: What must be true before release, and which production signals would expand, narrow or suspend authority?
- Experience: What must the user understand, confirm, inspect or recover at the moment that authority changes?
- Enforcement: Which identity, permission, tool, rate, retry and stop controls make the boundary real?
- Accountability: Who owns the product decision, the reserved judgement, the operating exception and the next review?
Sources and further reading
- NISTAI Risk Management Framework Core
- NISTAI Risk Management and Human-AI Interaction
- OpenAIA practical guide to building agents
- OpenAIA business leader's guide to working with agents
- AnthropicBuilding effective agents
- OWASP GenAI Security ProjectOWASP Top 10 for Agentic Applications for 2026
- European UnionRegulation (EU) 2024/1689, Article 14: Human oversight