Why Most AI ROI Metrics Are Measuring the Wrong Thing
Most AI ROI metrics measure activity, not value. Tokens burned. Monthly active users. Seats provisioned. None of it tells you whether the AI is actually helping someone do their job better.
This is a problem that shows up repeatedly in organisations that have deployed AI tools — particularly in the public sector, where procurement decisions are made by people several steps removed from day-to-day operations. The people holding the budget rarely see the work. The people doing the work rarely hold the budget.
That gap is where bad decisions live.
The Proxy Metric Trap
When organisations evaluate their AI investment, they reach for numbers that are easy to collect: how many people logged in this month, how many queries were processed, how much compute was consumed. These metrics feel rigorous. They are measurable, reportable, and fit neatly into a dashboard.
But they measure the wrong thing entirely.
A tool can have fifty registered users and forty-nine of them never open it. That tells you nothing useful. Meanwhile, one person might be using it every single day to do work that directly affects the organisation’s ability to function — and that one person’s usage will barely register in an aggregate report.
The question is never “how many people are using it?” The question is “what happens if we switch it off?”
A Story About a Subscription That Lapsed
We had a customer where the IT Director and Finance Director reviewed their Bidwriter subscription and decided they weren’t seeing sufficient user numbers to justify the cost. They let the subscription lapse.
Within an hour, the phone rang.
The person calling was the one who writes bids for the organisation — the person responsible for winning new work, bringing in new income, and ultimately contributing to the financial position that funds the roles of the IT Director and Finance Director who had just cancelled the subscription.
She couldn’t access KnowledgeFlow. She used it every day. It was embedded in how she worked.
The subscription was reinstated.
Who Should Be in the Room When You Evaluate AI?
This story isn’t unusual. The pattern repeats: a tool is evaluated by people who don’t use it, measured by metrics that don’t reflect its value, and cancelled by a decision that looks rational on paper and is immediately revealed as wrong in practice.
The fix is straightforward, even if the politics aren’t. When you review your AI investment, ask:
Who actually uses this, and what are they using it for? Not who has a licence — who opens it, and what do they do with it?
What would those people do instead if it disappeared tomorrow? If the answer is “they’d have to do it manually and it would take much longer,” that’s your ROI calculation.
Have you spoken to those people before making the decision? Not their manager. Not the team lead. The person doing the work.
The Squeal Test
There’s a blunter way to frame this. Switch it off and see who squeals.
The people who squeal are the people whose work depended on it. The silence from everyone else tells you something too.
Good AI ROI isn’t measured in usage statistics. It’s measured in the disruption caused when the tool disappears. If nobody notices, the investment probably wasn’t working. If someone is on the phone within the hour, you have your answer.
Measuring What Actually Matters
The better metrics are harder to collect but more honest: time saved on specific tasks, quality of outputs compared to what came before, whether the people using the tool feel it has changed how they work.
For organisations deploying AI in the public sector — whether in housing, local authorities, education or the NHS — the stakes are higher than most. These aren’t luxury tools. They’re being used by people trying to do more with less, under increasing pressure, serving communities that depend on them.
Getting the ROI evaluation wrong doesn’t just waste money. It removes something that was actually working.
If you want to understand how KnowledgeFlow fits into the work your organisation is already doing — and what it would take to measure its impact properly — you can find out more here.
