Research status: a source-backed technical design investigation, part of The Product Systems Notebook. The customer scenario and diagrams are illustrative. No end-to-end prototype evaluation or comparative benchmark has been completed.

The moment an agent's success becomes a customer's failure

A customer asks an AI assistant to change the delivery address for an order.

The assistant understands the message, locates the order, calls an update tool, and replies: Done. Your delivery address has been changed.

The next day, the package is still heading to the old address.

This is a constructed scenario, not a reported incident. But it exposes a real systems problem. There are at least three different things an AI application might describe as "success":

  1. The model chose the correct tool.
  2. An API accepted, or appeared to accept, the request.
  3. The business operation took effect in the systems that determine the customer's outcome.

These events are not interchangeable.

An agent finishing its work is not the same as the customer getting theirs done.

I wanted to examine what changes when we treat verified business completion, rather than a model's final message, as the product's unit of success.

What the existing evidence tells us

There is good reason for this question to matter now. In LangChain's voluntary 2026 State of Agent Engineering survey, 57.3% of respondents said they had agents in production, while about 32% identified quality as their leading production barrier [1]. Those are survey responses from a self-selected group, not a representative measure of all companies or evidence that a particular design is unreliable.

The engineering distinction is also well established. Anthropic separates workflows, where code determines the orchestration path, from agents, where a model dynamically decides how to use tools [2]. Its recommendation is not to maximize autonomy by default, but to use the simplest arrangement that solves the problem.

Agent SDKs already provide valuable capabilities: tool calling, resumable runs, tracing, guardrails, and review checkpoints. OpenAI's SDK documentation is explicit that applications can retain ownership of tool implementations, state storage, and approval decisions [3]. These tools reduce the effort of building an agent loop. They do not define what completed order change means for a business.

The missing question is therefore not simply can the model call the API? It is what evidence allows our product to tell the customer that the requested outcome happened?

Define "done" before choosing the architecture

I modeled one deliberately narrow operation: changing the delivery address on an existing order. Assume an authenticated customer, a separate order-management service, and an independent delivery provider. Both services hold an address relevant to fulfillment.

In this design, a change should be considered completed only when the system can establish all four conditions:

  • The requesting customer owns the order and the saved destination address.
  • The order is still eligible to change; an already shipped order cannot be silently rewritten.
  • The customer confirmed the specific order and destination outside the model's text output.
  • Fresh reads from both authoritative systems confirm the intended address.

The third condition matters: a model-generated confirmed: true field is not proof that a customer authorized an action. The application must bind approval to the authenticated person and the specific operation. OWASP's AI Agent Security Cheat Sheet recommends enforcing this at the execution boundary rather than trusting agent context [4].

This led to the first architectural rule:

The model may propose an operation. It should not be the sole authority for approving it, executing it, or declaring it complete.

Figure 1: A proposed execution boundary

  1. Interpretation · untrusted planning

    An authenticated customer request goes to intent extraction and agent reasoning, which produce a typed proposed operation.

  2. Policy + execution · trusted backend

    Ownership and eligibility checks, then explicit confirmation and a recheck. If permitted: a durable operation record with a stable request ID, then a controlled workflow and adapters.

  3. External systems · authoritative state
    • Order systemHolds the order's delivery address.
    • Delivery providerHolds the address the carrier will use.
  4. Independent read-back verification
    • CompletedBoth systems confirmed.
    • PendingOutcome unknown; reconcile.
    • ReviewConflicting, or unsafe to retry.
Figure 1. A proposed reference architecture, not a deployed system. It separates model interpretation from authorization, durable execution and independent outcome verification.

The proposal isn't complicated for the sake of complexity. Each boundary answers a different product question.

Interpretation: What does the user intend? The model can turn conversational language into a typed request for an address change.

Policy: Is this action legitimate now? Server-side checks use trusted identity, ownership, explicit confirmation, order status, and a current version check.

Execution: What should happen if the worker stops or an external call times out? A durable operation record and controlled adapters preserve the state of the attempt.

Verification: Did the intended state actually change? The system reads the records that own the truth rather than treating a tool response or agent message as definitive.

The architecture introduces overhead: additional components, more status transitions, and possible waiting before telling the customer a change is final. It is a hypothesis worth evaluating, not a claim that this is always the best design.

Figure 2: The timeout that creates the uncertainty

Now suppose the order system confirms the new address. The application sends the same update to the delivery provider. The provider applies the change, but the network response is lost.

From the application's perspective, the request timed out. It cannot distinguish "not applied" from "applied, response lost" without further evidence.

  1. 01 · Customer → ApplicationRequest address change
  2. 02 · Application → Order serviceAuthorized order update
  3. 03 · Order service → ApplicationOrder persisted · confirmed
  4. 04 · Application → Carrier serviceUpdate delivery destination
  5. Carrier serviceResponse lost. The carrier may have applied the update.
  6. 05 · Application → VerifierRead authoritative records
  7. 06 · Verifier → ApplicationCarrier outcome not yet established
  8. 07 · Application → CustomerPending verification · no success claim
  9. All participantsReconcile later. Retry only with a proven idempotency contract; otherwise escalate.
Figure 2. An illustrative failure path: the carrier applied the change but its response never reached the application. The safe customer-facing state is pending verification while the application reconciles authoritative records.

A naive retry is risky when a provider does not guarantee idempotent mutations. AWS's Builders' Library explains why a client request identifier and an explicit idempotency contract can make repeated requests safe, and why using the same identifier for different intent must be rejected [5].

For our design, the recovery choice depends on what the provider actually supports:

  • Idempotent write contract available: Retry with the same stable operation identifier and unchanged payload, within the provider's documented guarantees.
  • Status query available but no safe retry contract: Read back authoritative state and reconcile before deciding whether another write is appropriate.
  • Neither a trusted query nor a safe retry path: Leave the operation unresolved and route it for review instead of inventing certainty.

An even harder case occurs when the order system updates but the delivery provider permanently rejects the change. There is no automatic cross-provider database transaction. Any compensation must be guarded by current versions and business policy; otherwise a concurrent shipment event could make an attempted rollback unsafe.

The product insight is straightforward: "pending" is a legitimate outcome, not a conversational failure to hide.

Figure 3: Where autonomy helps, and where it might not

Our original research question compared three approaches. None has been benchmarked yet; the figure below records architectural hypotheses, not measured rankings.

Hypotheses to compare · not measured results

  • A / Deterministic
    • Code determines steps
    • Strong control over the write path
    • Less flexible request handling

    Good baseline for a fixed, well-understood operation.

  • B / Agent-directed
    • Model chooses permitted tools
    • Flexible across ambiguous tasks
    • More variable execution paths

    Requires independent policy even if the model plans.

  • C / Hybrid
    • Model interprets and proposes
    • Code authorizes and executes
    • More integration complexity

    Useful when flexible intent meets high-impact writes.

Decision rule: start with the simplest design that can satisfy the required outcome, then measure the value of added autonomy.

Figure 3. Three implementation patterns to evaluate against the same simulated order-change scenarios.

For a fixed address-change workflow, a deterministic implementation may prove cheaper, simpler, and just as effective. An agent-directed implementation may be useful when a customer request is ambiguous or requires planning across several possible steps. A hybrid design is worth examining when flexible interpretation needs to coexist with strict authorization and execution controls.

We should not assume the hybrid approach wins just because it produces an attractive architecture diagram.

Decisions that would need to survive a design review

DecisionWhy it mattersCost or reversal trigger
Keep write authorization outside model outputThe model's confidence is not a permission boundary.More backend logic. Reevaluate only if domain operations already provide equivalent controls.
Persist an operation before calling providersEnables traceability and recovery after worker failure.Adds storage and state management. Unnecessary for purely read-only requests.
Require independent read-back before COMPLETEDAvoids false success claims when systems disagree.Adds latency and provider calls. May need a time-bounded eventual-consistency policy.
Start with deterministic execution as the baselinePrevents confusing added autonomy with added value.Less flexible for open-ended tasks. Compare against model variants only where warranted.
Decisions that would need to survive a design review.

The strongest technical product work isn't choosing the most sophisticated design. It's being able to say which failure each component prevents, what it costs, and what evidence would justify removing it.

What would count as evidence, not just a persuasive diagram?

I drafted a small evaluation plan with 50 seeded scenarios: successful requests, unauthorized users, ineligible orders, duplicates, timeouts before and after writes, partial failures, and concurrent changes. The plan compares deterministic, agent-directed, and hybrid variants against the same mocked provider behavior.

The metrics that matter most are verified task completion, false-success messages, unauthorized mutations, duplicate side effects, time to resolution, and cost per verified completion. For model-driven variants, we'd separately evaluate natural-language interpretation so we don't confuse language understanding with execution reliability.

One boundary is important to state plainly. The related Agent Reliability Lab repository has passed its first two build phases: the database schema, operation state model, mock identity, seeded fixtures, and independent mock order and carrier services with deterministic fault injection, covered by 78 automated tests that pass from a clean database. It does not yet have the execution worker, the verifier, any model integration, or any experimental runs. There are no performance results to report.

That limits what this investigation can conclude today. It can defend the requirements and proposed architecture, but it cannot yet establish that the hybrid design is safer, faster, or more cost-effective in practice.

The question I would take into the product review

The agent loop is becoming easier to assemble. But the customer's job doesn't end at the boundary of a model call or an SDK.

A product still needs a definition of successful work, an owner for irreversible actions, a way to handle uncertainty, and a truthful user-facing status when systems disagree.

For technical product teams, I think the useful question is no longer just "What can we make the agent do?"

It is "What must be true before we let the product say it's done?"

Sources and methodology

Method: documentary research, original requirements analysis, failure-path modeling, and proposed architecture comparison. The e-commerce example is hypothetical. The diagrams and decision table are original design artifacts. No quantitative experiment has been performed.

  1. LangChain, State of Agent Engineering (12 June 2026). Voluntary industry survey; self-selection limits generalization.

  2. Anthropic, Building Effective Agents (19 December 2024). Workflow versus agent distinction and simplest-effective-design guidance.

  3. OpenAI, Agents SDK and Guardrails and Human Review (documentation checked 3 November 2026). Runtime, tool, and review boundaries.

  4. OWASP, AI Agent Security Cheat Sheet (checked 3 November 2026). Independent tool authorization and binding approval to the exact action.

  5. AWS Builders' Library, Making retries safe with idempotent APIs (checked 3 November 2026). Idempotency, client tokens, retries, and changed-intent rejection.