A production-readiness checklist for AI agents
Fourteen questions to answer before an agent touches real systems: access, limits, approvals, observability, quality, and rollout.
evanbyte3 min read
Demos run on happy paths. Production runs on everything else. Before an agent touches real systems, we want a clear answer to each of the questions below. Not every answer has to be 'yes' on day one, but every 'no' should be a decision, not an oversight.
Access
- Does the agent have its own identity? Give it a service account, so its actions are attributable and its access can change without touching anyone's login.
- Does it have only the permissions its job needs? Start with read access and add writes one tool at a time.
- Can you revoke its access in one step? If shutting it out takes a meeting, it takes too long.
Limits
- Is there a maximum number of steps per run? Loops that can't end will eventually run forever.
- Are there time and spending limits, per run and per day? A cap turns a runaway bug into an alert.
- Are duplicate triggers handled once? Webhooks repeat and users double-click. Use idempotency keys on anything that changes state.
Approvals
- Which actions need a person's approval, and is the list written down? Money, deletions, and anything customer-visible are the usual candidates.
- Does the approver see the proposed change and the evidence? 'Approve?' with no context gets approved without being read.
Observability
- Is every run traced end to end? Inputs, tool calls, outputs, timings, and cost.
- Can you replay a failed run? The fastest way to fix a failure is to watch it happen again.
- Do you get an alert when quality, cost, or error rates move? Agents degrade quietly. Something has to be watching.
Quality
- Is there an eval suite that blocks releases when it regresses? See Write the evals first.
- Do real failures flow back into that suite? Every surprise in production should become a test.
Rollout
- Can you switch it off in seconds and fall back to the manual process? A kill switch you've never tested is a hope, not a control.
What each control protects against
Every item on the list exists because something specific goes wrong without it.
| What goes wrong | The control |
|---|---|
| A loop that never ends | Step, time, and spending limits |
| The same action taken twice | Idempotency keys and de-duplication |
| An action nobody approved | Approval gates with the evidence attached |
| Private data in the wrong place | Scoped retrieval, redaction in logs, and least privilege |
| A quiet drop in quality | Evals in CI, sampling of live runs, and alerts |
| A bad release | A staged rollout and a kill switch you've tested |
A staged path to launch
Launch isn't one event. It's a sequence, and each stage produces the evidence for the next.
- Offline. Run the eval suite on real past cases. Nothing touches production.
- Shadow. The agent runs on live traffic but only records what it would have done. People compare its output with what they actually did.
- Approval. The agent proposes real actions, and a person approves each one. You measure how often they agree.
- Limits. Routine actions run alone inside tight limits, with approval above them. Widen the limits only as the numbers support it.
If a stage looks bad, you've lost a week, not a customer. The agent blueprint tool lays out this path for your own workflow.
