Mikel Studio · Issue #3
Make AI Systems Prove What They Did
Three ways to make production AI observable, recoverable, and open to challenge.
The next step in production AI is not simply adding more capability. It is making the system’s behaviour easier to inspect, recover, and question. This edition looks at assurance as an operating practice: collect evidence while the system runs, design for interruption, and create enough independence to challenge your own assumptions.
The Signal
Turn assurance into an operating loop
A system card can explain what was evaluated at a particular point in time. It cannot show whether an agent remains constrained after its model, prompt, tools, permissions, or workflow changes.
The practical shift is to treat assurance as a production loop. For one consequential workflow, define the behaviours that matter, test representative and adversarial cases, record meaningful actions, enforce authority at the tool or application layer, and re-run evaluations after significant changes. When an incident occurs, convert it into a new test case, alert, policy rule, or workflow improvement.
A useful review question is: could an independent engineer reconstruct what the agent used, attempted, was allowed to do, changed externally, and where it stopped? If the answer depends on the model’s explanation or a user’s memory, the system is not yet continuously verifiable.
From the Lab
Design for the half-finished task
An agent may look like one product surface, but its work depends on state, files, tools, permissions, queues, and other services. The September 14 Work Mode incident—reported as affecting task start or resume and access to workspace tools and files for some users—makes that dependency visible.
The practical design test is not whether a task works from beginning to end. It is whether it can stop halfway through without creating uncertainty or duplicate side effects. Persist durable checkpoints. Give actions stable operation IDs where appropriate. Bound retries. Show whether work is queued, running, waiting, blocked, failed, or complete. When a dependency is unavailable, preserve a draft or provide a clear handoff rather than silently starting over.
For every consequential workflow, identify the checkpoint, state owner, retry policy, deduplication mechanism, and fallback path. If the recovery plan is “run it again,” the workflow still has an operational gap.
What We Learned
Build in a challenge function
Independent evaluation is useful not because outside reviewers are automatically right, but because teams need a credible way to test their own claims. Anthropic’s proposal for embedded third-party evaluators—with ongoing access and publication rights—points toward scrutiny that is continuous rather than limited to a polished report or launch review.
The same principle can be applied at smaller scale. Give someone outside the immediate build team responsibility to challenge one high-impact workflow. Provide the evaluation cases, relevant configuration, incident records, and logs needed to distinguish a model failure from an integration or control failure. Define which findings require escalation and re-run the review after material changes.
Independence does not require a large audit programme on day one. It requires enough separation, access, and authority for a reviewer to question whether the system’s controls work in realistic conditions—and for the team to act on the answer.