What this guide is for
A pilot is a small, time-limited test of one agent on one workflow, designed to produce a decision. This guide gives a four-week plan, the measures to track and a checklist. It is written for owners and managers, and it suits a business that has chosen a candidate task but not yet committed.
The plan is our own. It is built on public guidance: Anthropic advises finding “the simplest solution possible, and only increasing complexity when needed”, and Microsoft advises using “the lowest level of complexity that reliably meets your requirements”. Where we state a step as a recommendation, we say it is ours. For background on what agents are, read our plain-English playbook.
Before day 1: five decisions
Our own suggestion, not a finding from the sources: settle these before anyone builds anything.
- One workflow. Repeated often, easy to define as done, cheap to get wrong and quick to check. Answering customer questions from your own documents and booking appointments are two examples from our published guidance.
- One owner. A named person who answers for the result, as for a member of staff.
- One baseline. Record today’s numbers for the workflow: volume, time per item, error rate, cost including staff time. Without these, any before and after is guesswork.
- One access list. Write down exactly what the agent may read and change. Start with read-only.
- One stop rule. Decide in advance what would make you pause the pilot, such as a wrong message sent to a customer or access beyond the list.
The plan at a glance
| Week | Goal | Output |
|---|---|---|
| 1 | Define and measure | Written scope, access list, baseline numbers, success thresholds |
| 2 | Build and test privately | Working agent tested on past cases, with logging switched on |
| 3 | Run alongside people | Agent drafts, a person approves every result |
| 4 | Limited live run and decision | Results against baseline and a continue, adjust or stop decision |
Week 1: define and measure
- Write a one-page scope: what the agent does, what it never does, and when it hands over to a person.
- Define “done” in a sentence with one example that counts and one that does not. Vague outcomes are the hardest part of agent projects, as our buying guidance on outcome-based pricing notes.
- Pull your baseline from the last month of real work.
- Set thresholds now, not after you see results. For example, the lowest accuracy you would accept and the highest review time per item. The numbers are yours to choose.
- Tell the affected staff what is being tested and why.
Week 2: build and test privately
- Connect the agent only to what the access list allows. Prefer read-only access to a copy or test environment.
- Turn on logging before the first test. If you cannot see what the agent read, called and changed, you cannot govern it.
- Test on past cases where you already know the right answer. Include awkward ones: missing information, an angry customer, a request outside scope.
- If the agent connects through MCP, ask the builder which specification version the servers run and how client-supplied state is verified. The OWASP MCP Top 10, still in beta, is a useful question sheet.
- Check what happens when the agent does not know. It should hand over, not guess.
Week 3: run alongside people
Real work now flows through the agent, but nothing leaves the business without a person’s approval.
- The agent drafts; a named reviewer approves, edits or rejects each item.
- Record every edit and rejection with a reason. This is your best evidence about where the agent is weak.
- Track review time per item. Among developers, Stack Overflow’s 2025 survey found 66% pointed to AI output that is “almost right, but not quite”. Expect to see something similar and measure it.
- Fix instructions, data or access based on the log. Change one thing at a time so you know what helped.
- Do not widen scope this week.
Week 4: limited live run and decision
- Let the agent complete a defined slice of low-risk items without item-by-item approval, only if week 3 met your thresholds. Keep approval for payments, deletions, outbound messages and anything customer-facing.
- Review a sample of results each day, and read the log.
- At the end of the week, compare the numbers with your baseline and write the one-page decision (below).
- Keep the ability to switch the agent off in minutes.
Success measures
Gartner, as reported by W.Media, said over 40% of agentic AI projects could be cancelled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls. Our suggestion is to measure those three things directly.
| Area | Measure | How |
|---|---|---|
| Value | Time per item, volume handled, error rate, turnaround | Compare with the week 1 baseline |
| Cost | Supplier fees, staff review time, setup effort | Total for the month, divided by items handled |
| Control | Actions outside the access list, incidents, items handed to a person | Read from the log, not from the supplier’s summary |
| People | Reviewer confidence and staff feedback | Short check-in at the end of weeks 3 and 4 |
Measure outcomes, not enthusiasm. That advice comes from our post on how developers use coding agents: track review time, defects and delivery speed before and after, not how often the tool is used.
The day-30 decision
- Continue: thresholds met on value, cost and control. Widen scope one step at a time.
- Adjust: promising but a threshold missed. Name the fix, run another two weeks.
- Stop: thresholds missed with no clear fix, or the workflow turned out to be better served by plain automation or a booking link. This is a valid, cheap result.
Write the decision on one page: what you tested, the numbers against baseline, what went wrong, what you would change. If you continue, pick the next workflow with the same five decisions.
Checklist
- Workflow chosen, with a written definition of done
- Owner named
- Baseline recorded from real work
- Thresholds for value, cost and control set before week 1 ends
- Access list written; read-only to start
- Logging on before the first test
- Tested on past cases, including awkward ones
- Handover to a person defined and tested
- Reviewer assigned for week 3
- Approval kept for payments, deletions, outbound messages and customer-facing steps
- Switch-off tested
- Day-30 decision written on one page
Need help running one? We build custom AI agent development, and you can book a call.
Sources
- Building effective agents, Anthropic, December 19, 2024
- AI agent orchestration patterns, Microsoft Learn, Azure Architecture Center, February 2026
- Over 40 percent of Agentic AI projects could face the axe by end of 2027: Gartner, W.Media, June 27, 2025 (reporting Gartner’s June 25, 2025 press release)
- 2025 Developer Survey: AI, Stack Overflow, 2025
- MCP Top 10 project, OWASP (beta, 2025 edition labels)
- AI Tooling for Software Engineers in 2026, The Pragmatic Engineer, March 3, 2026
- AI Agent Pricing: Per Seat, Per Task or Per Outcome?, Zunkiree Labs (buying guidance, not market data)
- AI Coding Agents in 2026: How Developers Actually Work Now, Zunkiree Labs
- Why AI Agents Are Getting Their Own Infrastructure, Zunkiree Labs, October 2, 2026
Frequently asked questions
How long should an AI agent pilot last?
Thirty days is a workable length for one narrow workflow: enough time to set a baseline, test with a person reviewing, run a limited live period and decide. A broader scope needs longer. The aim is a decision, not a launch.
What should the pilot measure?
Gartner, as reported by W.Media, links cancelled agentic projects to escalating costs, unclear business value and inadequate risk controls. Our suggestion is to measure those three things: cost, value against your own baseline, and control (access, logs and incidents).
Who needs to be involved?
A named owner who is accountable for the result, a reviewer who checks the agent's work, and a supplier or builder who can answer questions about access, logging and failure. Staff whose work changes should hear about it in week one.
What if the pilot fails?
That is a useful result. A pilot that ends with a clear reason to stop, such as unclear outcomes or too much review time, costs far less than a full rollout that gets cancelled. Write down what you learned and choose a different workflow or wait.
What you get:
- A four-week plan from scoping to a go, adjust or stop decision
- Five decisions to make before day 1
- Success measures for value, cost and control
- Guardrails to keep in place during the pilot
- A 12-point checklist you can print