Story Points and AI: Measure Flow and Outcomes Instead

Story points estimate effort. AI has made effort unstable, different for every person and shrinking month by month. So a metric built on effort now tells you less than ever. My advice: stop arguing about points and move your measurement to the two things AI cannot distort, how work flows and what it changes for customers.

The Assumption Story Points Depended On

Story points were never meant to measure productivity. They were a planning shortcut. Instead of debating whether a feature would take 12 or 15 hours, a team agreed this work was bigger than that. Over a few sprints, velocity (the points a team finishes per sprint) gave a rough forecast of capacity. That was the whole job.

It worked because of one quiet assumption: the effort behind a point stayed roughly stable. A five-point story this quarter cost about the same effort as one last quarter. The baseline held, so the forecast held.

Then organisations started misusing velocity as a performance metric. Teams got compared, targets got set. A mistake, but tolerable while the baseline held: the numbers at least meant something consistent.

AI Did Not Break the Tool. It Broke the Assumption

Same Five Points, Half the Effort

A story your team sized at five points last year involved writing code, tests and documentation by hand. Today an AI assistant drafts most of that. The business value is identical. The human effort is not.

So what is it now? Five points, and points drift away from effort? Three, and your history becomes useless? Neither works. The unit has come loose.

Effort also varies by person now. A developer fluent with AI and a colleague who uses it sparingly spend very different effort on the same story. One estimate, two realities. A shared baseline cannot survive that.

Velocity up 30% Tells You Nothing

Here is the trap. A team adopts AI tooling and velocity jumps 30%. Leadership celebrates. What has probably happened is that the measuring stick shrank. The team may be delivering the same value with less effort, which is good news, but the chart cannot separate better performance from better tools.

The data backs up the caution. Faros AI studied over 10,000 developers: teams with high AI adoption merged 98% more pull requests (bundles of code changes awaiting review), yet company-level delivery did not improve at all.¹ In METR’s controlled trial, experienced developers believed AI made them 20% faster while the stopwatch showed them 19% slower.² METR’s 2026 follow-up suggests developers now go faster, but so many refused to work without AI that METR is redesigning the study.³ Even the people who measure effort for a living can no longer pin it down. Either way, feelings about speed are not evidence.

Measure Flow First

Flow metrics measure what happened, not what someone guessed. That one property makes them immune to the problem that broke story points. No estimate, no baseline to erode.

I ran flow metrics across 17 data engineering teams at a UK retailer, using plain cycle time and throughput data. Nothing foreign. The discipline those numbers enforced mattered more than any chart: keep work small enough to reach production in one to three days, and let the data settle arguments that opinions could not.

If your teams code with AI, three flow signals earn a place on the wall. They turn a noisy delivery system back into something you can read, which is my whole job: order in complex work.

Cycle time and Lead time

Cycle time is how long work takes from started to live. Lead time is how long from requested to live. If AI speeds your teams up, these numbers fall and you can prove it. If coding time shrinks but cycle time does not, you have found your real bottleneck, and it is not the typing. In most AI-adopting teams it is code review, where the extra volume piles up.

Work Item Age

The earliest honest warning you can buy. An item that has been in progress for eight days is a fact, whatever the velocity chart says. Ageing items point at where work waits. AI produces more change per week, so queues form faster and this signal matters more, not less.

Throughput and Work in Progress

Count finished items per week. No sizing, no ceremony, no baseline to argue about. Pair it with a limit on work in progress (how many items sit unfinished at once) and you get forecasting that survives AI, because it never depended on effort. An agent that spawns ten parallel branches is still ten open items ageing against one reviewer’s attention.

How the three families compare:

Then Measure Outcomes

Flow tells you the machine is healthy, not that it is pointed at anything useful. AI sharpens this second question, because when building gets cheap, building the wrong thing gets cheap too.

Quality

Speed that returns as bugs is borrowed, not earned. Watch defect rates, production incidents and rework. DORA’s research found AI raises throughput but degrades stability where small batches, good tests and fast feedback are weak, and added rework rate to its core metrics for this reason.⁴ If your cycle time falls while rework climbs, you have not gained speed. You have deferred it.

Customer Impact

Adoption, engagement, retention, satisfaction. These answer the question velocity never could: did the work make a difference? No customer ever cared how many points a sprint burned, only whether the product solved their problem.

This is the part I found hardest. At the UK retailer, flow data was cheap to collect and paid back within weeks. Outcome measures took months, because for internal data platforms “adoption” needs defining before it can be counted, and nobody owned the definition at first. Expect the flow half to take a month, the outcome half two or three. Start both anyway; the value is in the outcome half.

What Story Points Are Still For

I am not telling you to ban them. A sizing conversation still surfaces misunderstanding, risk and hidden complexity before work starts, whoever writes the code. Keep points for that if your teams find them useful.

But retire them from every other job. Flow data forecasts better. Points were never a performance measure. And comparing velocity between teams was always nonsense. AI is not creating this weakness, it is making it impossible to ignore.

What I Would Do on Monday Morning

1. Pull cycle time, throughput and work item age for the last eight weeks. Jira and Azure DevOps already hold the data.
2. Set a working agreement on item size. One to three days to production. Small batches were good practice before AI. Now they carry the whole system.
3. Put work item age on the daily board. Ask about the oldest item first, not the newest.
4. Pick one outcome measure per product. Adoption, retention, whatever fits. Agree who owns the definition, then review it monthly next to the flow data.
5. Tell leadership the velocity chart is retiring. Explain why before someone celebrates a 30% jump that means nothing.

The teams that handle AI well will not be the ones with the highest velocity, but the ones who can show, with evidence, that work flows quickly, quality is holding and customers noticed. Agile asked us to value outcomes over output from the start. AI is forcing us to live by that at last.

Takeaway: story points measured effort, and AI made effort meaningless as a unit. Measure flow to see how work moves, outcomes to see if it mattered, and keep points only for the planning conversation they were born for.

1. The AI Productivity Paradox Report 2025, Faros AI, July 2025. ↩︎
2. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, July 2025. ↩︎
3. We are Changing our Developer Productivity Experiment Design, METR, February 2026. ↩︎
4. DORA: State of AI-assisted Software Development 2025, DORA / Google Cloud, September 2025. ↩︎