Key takeaway: Faster task completion is not the same as more output. Measuring the task rather than the system is why reported productivity gains do not appear in delivery.
Why Self-Reports Mislead
Surveys asking how much time a tool saved reliably produce enthusiastic numbers. Those numbers do not survive comparison with observed data.
People estimate against a remembered baseline they never measured, notice the visible fast part while overlooking added verification, and answer in a context where the expected answer is obvious. None of this is dishonesty; it is a known limitation of retrospective self-assessment.
The Bottleneck Moves
A task that took two hours now takes forty minutes. Throughput should triple. It does not, because the constraint on delivery was never that task.
If code generation accelerates and review capacity is unchanged, work accumulates before review. If drafting accelerates and approval is unchanged, drafts queue for approval. Speeding up a non-bottleneck step increases work in progress, not completion.
This is why organisations report substantial individual time savings alongside unchanged delivery metrics. Both observations are accurate.
Measuring the System Instead
| Level | Metric | What it reveals |
|---|---|---|
| Task | Time to complete | Whether the tool works |
| Flow | Lead time end to end | Whether the system improved |
| Flow | Work in progress | Where the queue formed |
| Quality | Rework and defect rate | Whether speed cost correctness |
| Outcome | Delivered value | Whether any of it mattered |
Lead time from request to delivered is the honest number. It captures queueing, review and rework — everything a task-level measurement excludes.
Rework rate must be measured alongside speed, because the most common way to appear faster is to produce output that needs more correction later. A team shipping twice as fast with three times the defect rate has not improved.
Designing a Credible Comparison
Randomising by task and comparing groups is the strongest available method in most organisations. Assign incoming work to assisted and unassisted paths, then compare lead time and quality on comparable items.
The alternative, comparing before and after, is confounded by everything else that changed — team composition, project difficulty, seasonal load. Those effects are frequently larger than the tool’s.
Duration matters too. Early measurements capture novelty and learning-curve effects in both directions. Four to eight weeks is a minimum for a stable signal, and the direction sometimes reverses within that window as initial enthusiasm meets accumulated review burden.
What to Do With the Answer
If task time improved and lead time did not, you have located your bottleneck precisely. That is a valuable result — it identifies where the next investment should go, which is usually review capacity or approval process rather than more tooling.
The Bottom Line
Measure lead time and rework rate rather than self-reported task savings, compare randomised groups instead of before-and-after periods, and expect the bottleneck to move. A speedup that does not shorten lead time has told you where the real constraint is.
