An AI feature becomes dependable when its operating boundary is designed as deliberately as its user experience.

Start with the decision, not the model

The first design question is not which model to use. It is what the product should help someone understand, decide or do, and what happens if the result is incomplete, wrong or acted on too quickly. A summarisation aid, a service recommendation and an agent able to change a customer's account are not three versions of the same feature. They carry different consequences and need different operating boundaries.

Write the proposed responsibility as an action: ‘identify likely duplicate claims for an investigator’, ‘draft a response using approved policy’, or ‘schedule an engineer within agreed constraints’. Then name the affected person or operation, the acceptable uncertainty and the worst credible failure. This gives product, design, engineering, security and operations a shared object to assess.

It also exposes cases where AI is unnecessary. If the rules are stable, the inputs structured and the required result exact, deterministic software may remain the better control. OpenAI's agent guidance recommends agents particularly where decisions are complex, rules are difficult to maintain or unstructured data is central. The product decision comes before the model decision.

SourcesOpenAI

Make context and permissions explicit

Useful behaviour depends on more than a prompt. The capability may need product state, customer information, policy, operational history and tools. For each source, define why it is needed, which records are permitted, how current it must be and how provenance will be shown. Retrieval should narrow the operating boundary, not quietly give the model access to an organisation's accumulated knowledge.

Permissions need the same precision. Separate reading, proposing and acting. An assistant may read an order and draft a remedy without being allowed to issue a refund. An agent may create a deployment plan but require confirmation to change production. Use a dedicated identity, least-privilege access, limits on scope and value, and an audit trail that records the context and tool calls behind a consequential action.

OWASP's agentic application guidance highlights risks created by goals, tools, identities, memory and inter-agent communication. Current cloud reference architectures similarly place identity, logging, monitoring and governance around the agent runtime. The lesson is architectural: the operating boundary lives in permissions and interfaces, not in an instruction telling the model to be careful.

Design evaluation around the real task

A generic measure of model quality does not establish that a product decision is dependable. Evaluation must reflect the task, the operating context and the cost of failure. For a policy assistant, that might include factual grounding, use of the current policy, correct handling of exceptions, appropriate refusal and the time saved by the colleague making the final decision.

Build a representative set of normal, difficult and adversarial cases before release. Record the expected qualities, not only an exact answer. Run the set when the model, prompt, tools or context change. Combine automated checks with structured human judgement where interpretation matters, and keep failures in the evaluation set so the same weakness does not return unnoticed.

Production completes the evidence. Observe acceptance, correction, escalation, latency, cost, failure and downstream outcomes. NIST's Generative AI Profile treats measurement and management as continuing activities across the lifecycle; this matters because a controlled test cannot reproduce every user, data condition or attempted misuse. Release is the start of supervised learning about the product, not the end of evaluation.

SourcesNIST; OpenAI

Design human intervention as an operating path

‘Human in the loop’ is too vague to be an operating model. Define when intervention occurs, who receives it, what evidence they see, how quickly they must respond and what safe state is preserved while they decide. A hand-off that sends an unexplained output to a busy queue transfers uncertainty rather than controlling it.

Intervention can take several forms. Confirmation is appropriate before a consequential action. Exception handling is useful when confidence, policy or tool availability falls outside a known range. Sampling can supervise low-consequence work without delaying every case. A kill switch and recovery path are needed when behaviour at scale creates unexpected harm. The right combination depends on consequence and reversibility.

Ownership continues after launch. Product leaders own whether the capability creates value; engineering owns its dependable operation; domain and risk owners define critical boundaries. Those responsibilities can be distributed, but the decision to expand autonomy should have a named accountable owner and current evidence.

SourcesOpenAI; NIST

Increase responsibility only when the evidence supports it

A reasonable counterargument is that this level of design could make a simple AI feature unnecessarily heavy. It can. Controls should be proportionate. A private drafting aid that cannot act and exposes no sensitive data does not need the same architecture as an agent changing production or customer records. The discipline is to make that judgement explicit rather than assuming every feature is harmless or treating every use as high risk.

Define agent responsibilities from the intended outcome, evaluation evidence and permitted authority. A capability might first retrieve and summarise, then recommend, then prepare an action for confirmation, and only later act within narrow limits. At each stage, expand the operating boundary only when task evaluation, production behaviour and recovery evidence justify it. If users routinely correct an output or escalation cannot meet its response time, more autonomy is not yet progress.

Keep the boundary more stable than the implementation inside it. Models, prompts and orchestration will change. The decision definition, permitted sources, tool contracts, evaluation criteria and accountability should provide a durable product frame within which those components can be compared. This reduces dependence on one model and prevents every technical upgrade from becoming a fresh debate about acceptable behaviour.

Record the rationale so later teams can understand why the limit exists and what evidence would allow it to change.

WQA uses this boundary-first approach to connect AI product design with the production system beneath it. The contribution is not an additional governance layer. It is a design method that makes decisions, context, tools, evaluation and accountable operation part of the feature from the start. That allows teams to move quickly without making responsibility implicit.

  • Responsibility: what may the capability understand, recommend or do?
  • Consequence: who or what is affected if it is wrong, delayed or misused?
  • Context and tools: what can it retrieve, invoke or change, under which identity and limits?
  • Evidence: which task evaluations and production signals justify release?
  • Control: when does it confirm, escalate, stop or recover, and who is accountable?

Sources and further reading

  1. OpenAIA practical guide to building agents
  2. NISTArtificial Intelligence Risk Management Framework: Generative AI Profile
  3. OWASP FoundationOWASP Top 10 for Agentic Applications for 2026
  4. Google Cloud Architecture CenterMulti-tenant agentic AI system