Skip to content
FocalIDFocalID

How to measure an AI agent pilot: the numbers that matter

A pilot exists to answer one question: is this agent worth putting into daily use? You can only answer it if you decide what to measure before the pilot starts. This guide lists the numbers that matter and how to read them.

By the FocalID team in Colombo, , 2 min read

Hands typing on a laptop that shows an analytics chart.

Measure the old way first

Before the agent handles a single case, record how the work is done today. For two weeks, note how many cases arrive, how long each takes, how long customers wait and how many contain errors. Without this baseline, every result from the pilot is an opinion.

Six numbers to track

MetricWhat it tells you
Completion rateThe share of cases the agent finishes without a person taking over
AccuracyThe share of finished cases that were correct when checked
Hand-over rateHow often the agent stops and asks a person, and for what reason
Time per caseHow long a case takes from arrival to done, compared with the baseline
Response timeHow long the customer or colleague waits for the first useful reply
Cost per caseRunning cost of the agent plus the staff time still needed, per case

Completion rate and accuracy must be read together. An agent that finishes everything but gets one case in ten wrong is worse than one that hands over a third of its cases and is right on the rest.

Check quality by sampling

Do not rely on the agent’s own report of its success. Each week, have someone who knows the work review a random sample of finished cases and mark each as correct, acceptable or wrong. Review every hand-over too, because the reasons show you what to improve next.

Before launch, test the agent on past cases where the right answer is already known. Keep that set and rerun it after every change, so an improvement in one place does not quietly break another.

Decide: scale, fix or stop

  • Scale when accuracy matches or beats your team’s, hand-overs are falling week by week and the cost per case is clearly below the baseline.
  • Fix when most errors fall into a few causes, such as missing data or an unclear rule. Solve those and measure again.
  • Stop when the work turns out to need judgement the agent cannot be given, or when volume is too low for the saving to matter. A pilot that ends this way has still done its job.

Keep the pilot small enough to judge

One workflow, a few weeks and an agreed set of numbers is enough. At FocalID the first step is a fixed-price, four-week AI agent pilot on one workflow, which ends with a results report and next steps.

Common questions

How long should an AI agent pilot run?

Long enough to see a few hundred real cases. For most workflows that is a few weeks.

What accuracy is good enough?

Compare it with how your team performs on the same work today, and with what a mistake costs. The right target differs between a draft reply and a payment.

Who should review the agent’s work?

Someone who does the work today. They know what a correct result looks like and will spot errors quickly.

Related