How do you tell whether an AI agent is actually saving time?
Most reporting around this measures activity because activity is easy to count. The number that matters is smaller, harder to get, and worth the trouble.
Questions and answers
What should I measure?
Minutes returned to a named person on a named task, and how often the output needed fixing. Everything else — runs, tokens, agents deployed — is activity. Activity goes up whether or not the thing is working, which is exactly why it is the number people report.
How do I get a baseline if nobody timed the task before?
Ask the person to time it for three days before anything changes. Three days of imperfect measurement beats a perfect estimate, and it stops the argument later about whether the task really took that long.
What is a realistic saving?
For a well-chosen task, most of the reading and drafting and none of the deciding — often half to three-quarters of the elapsed time, with the remainder being review. Anything claiming to remove the whole task is describing a task nobody was accountable for.
How do I know if the quality is holding up?
Count edits, not complaints. Have the reviewer note whether each output went out unchanged, lightly edited, or rewritten. A rising rewrite rate is the earliest signal that something drifted — usually the inputs rather than the agent.
When should I switch an agent off?
When review takes longer than the task did, when the rewrite rate stops falling after the first fortnight, or when the person it was meant to help has quietly stopped using it. The last one is the most reliable and the least reported.
How long before I judge it?
Two weeks of real use. The first few days are unrepresentative in both directions — novelty inflates it, unfamiliarity depresses it. By the second week people have stopped thinking about it, which is when the number means something.