Still not that into Copilot

Back in September 2024, we published a blog called Why aren’t we in love with Copilot?. Nine months on, I’ve been trying to work out if my feelings have changed. The short answer is: I’m still just not that into it.

Nine months is a long time. Long enough to grow a whole new person. To put that in tech speak, it’s the time it took for OpenAI to put out five new ChatGPT releases.

It’s not that Copilot isn’t useful, and it has definitely improved. I know people — especially in large organisations — who say it’s saving them time, and I believe them. But, for the price, I’m not convinced it’s giving people what they need. Or at least, not what we should expect to get as a return on gen AI investment.

If Copilot (like its natural dad, Clippy) were free, I’d say it’s fine. Even quite good. But it’s not free (for clarity, I am referring to MS 365 Copilot). And once you start looking at what’s being spent – and particularly how much public money is being spent – you have to ask: are we making good choices?

Your taxes

In late 2023, the UK Government’s Crown Commercial Service signed a five-year deal with Microsoft. It covers the usual suspects (Office, Teams, Azure) and it also includes Copilot. The idea was to give the public sector discounted access to Microsoft’s newest AI features.

(I’m going to pause here and point out that I could write an entire blog post asking how the UK procurement sector can lead more organisations to vendor lock-ins and big tech suppliers. We’d reflect on how or if it helps the public sector to be dynamic and innovative and how it supports small and medium sized businesses, or not. But we don’t have time for that right now and we can agree these questions are implicit, yes?)

Even with negotiated discounts, the total UK government spend in this space is big. Just one department — the Department for Work and Pensions — is estimated to have spent around £250 million on Microsoft licences over five years. That’s more than £600 per user per year by some estimates. HMRC spent even more: £265 million since April 2021. Across central government, total Microsoft-related spend reached an estimated £1.44 billion in 2023–24.

Paying a fair price for licenses from trusted suppliers to use reliable tools, minimising procurement friction, trying to deliver some degree of operational consistency across a fragmented system, and managing risks around data security is no small task. No one expects these kinds of investments to be small or for them to land with untested suppliers – arguably, you want it to be reassuringly expensive. We’re not here to tut about investing in enterprise-grade tech in public services: I can’t think of a better place to invest in good tech. But the spend figures quoted above were up to March 2024, before a lot of public sector organisations started buying into generative AI solutions in a serious way. Copilot is running on top of that spend and, because it’s marketed as a comprehensive AI assistant for Microsoft 365, it also means a lot of places buy into it then stop shopping around for other AI-driven solutions almost entirely.

What’s working

The big headline from the UK government Copilot pilot was this: civil servants using Copilot saved about 26 minutes a day. That adds up to nearly two weeks a year of regained time. Not bad. And in that trial, over 70% said it helped with boring or repetitive tasks. The overwhelming majority said they didn’t want to give it up.

We shouldn’t trivialise the efficiency saving – and I don’t want to knock civil servants; they are good, smart people, doing important work. These kinds of savings, across thousands of people, make a difference. And the UK the public sector has lived through a series of real terms pay cuts, so the least they deserve is some decent tech that reduces time spent on tasks that are just drudgery.

There’s good evidence from enterprise settings that matches up to the public sector experience: Copilot users are getting faster at writing, less time is spent wrangling documents and emails.

So yes, it’s helping. It is unarguably better than nothing. But is that really the choice?

There are two questions that anyone who’s ever written a business case is probably already asking, as they read this: is Copilot offering good value in itself, and how does it stack up against other options?

What is the alternative?

First off, if most people are using Microsoft 365 Copilot for meeting summaries and email writing (and most users I talk to say it’s the meeting notes they are most attached to), it’s worth pointing out: you can get a lot of that with Teams Premium, which costs about a third of the price. It’s possible this wasn’t mentioned in your Copilot demo.

It’s hard to establish exactly what rate-card the public sector is accessing at this moment, and we know they are accessing the kinds of discounts that tend to materialise for customers who are prepared to sign very, very long-term deals, at scale. But we do know that in the private sector (and, awkwardly, much of the voluntary sector), price estimates are:

• Microsoft 365 Copilot: approx. £23/user/month (£276/year).

• Teams Premium: approx. £7.70/user/month (£92/year) — includes meeting transcripts and AI summaries, two of Copilot’s most popular features

Some people will be reading this and thinking that maybe there isn’t much choice, in practice, when you’re bound by law to process all your data in (in the case of the UK government) the UK, and you already have well-established cloud infrastructure on Microsoft’s Azure. The EU applies similar restrictions on the public sector. There’s a strong case that buying into Copilot to access generative AI potential is a pragmatic inevitability. And it is true that a lot of solutions are off the table for the public sector once you really test your suppliers on where the data is stored and processed and what rights suppliers have to use your data in training their next AI.

There are still choices, even when you need – or just prefer – to apply residency controls to your data. AWS and their cloud architecture, Bedrock, also enable you to choose where your data is housed. Google too. On Bedrock, you can use Anthropic’s Claude locked into your own secure environment in the same way that Microsoft enables you to use OpenAI’s ChatGPT in Azure. Those are the features that save you wondering what happens if the data goes offshore for processing, where another government might have very different rules about who can access it and how. And you can have both - running a solution on Bedrock while your SharePoint is humming away in Azure is not a problem.

That’s already quite a lot of flexibility outside the Microsoft bubble, so if you want help with general writing or information look-up or even basic coding, there are simpler, cheaper, more focused tools on the market that do a great job, using various of those underpinning systems. A secure, standalone GPT-type assistant — for example (ahem) one that’s plugged into your policies or knowledge base, like the kind we build — can get you a long way for surprisingly little spend.

Then, of course, there’s the access to free tools. You can help your users understand when it is okay to use those, which is easily achieved with a moderate amount of policy setting and internal comms. And when you think about it, there’s quite a lot of instances when you don’t need to worry about your intellectual property and there may be nothing in your data that needs protecting. Producing public information is a prime example.

What still isn’t working

The more impressive parts of Microsoft 365 Copilot — embedded AI assistance in Word, PowerPoint, Excel — are still patchy. People regularly report vague, half-finished content. In Excel, especially, it’s often not usable without major cleanup. And despite all the rebrands, it’s still unclear to many users exactly which Copilot they’re using and what it should do.

Meanwhile, on any given day, I – like a lot of people – toggle between a bit of Napkin (just glorious for turning text into infographics), Perplexity (unbeatable, imho, for web searches and source referencing), our private GPT (generating and summarising text, working out ideas and plans – and any general language-based task where I want to know precisely where my prompts and responses are stored), Gamma (getting better all the time for slide decks, branded reports… incredible) and a few others.

Microsoft has faced criticism over the Copilot branding — especially around user confusion due to very different experiences across tools carrying the same name. And even now, the sales pitch doesn’t always seem to match the experience – albeit some of the criticism is probably a bit unfair because we’re still all learning to prompt and often comparing it to ChatGPT, which is particularly good at telling you how great you are. It’s possible that affects your views of whether you had a good experience using it (“You’re great, ChatGPT!”, “No, you’re great, Leading AI!”).

In reality, I’ve spoken to too many teams who were genuinely excited to have full-fat Copilot at first then, once the novelty wore off, settled into using it for just a couple of tasks (meeting notes, mainly). Some kept using ChatGPT on the side, whether sanctioned by their employer or not. Others quietly stopped using it altogether. This week, for the first time, I met some people who were using it more and more and were generally pretty happy. But they admitted the issues we’d seen were common amongst their colleagues, and they hadn’t tried a lot of other AI tools to compare.

Copilot, it’s not you, it’s… well, maybe it is you

I suspect this all sounds atypically grumpy for one of our blogs. If you’re wondering what set me off, it was this post on LinkedIn. It takes a lot of time and effort for someone like a hard-pressed SEN school to shop around for tech. It also takes some skill and knowledge. They have to, therefore, lean heavily on both government and their trusted suppliers to guide them. In an ideal world, one where all schools are funded to research and deliver the best possible education money can buy, they might shop around, buy two or three things, run them side by side for a while, have someone independently evaluate the impact and comparative value, then purchase something for the long term good of staff and students. But in practice they have to quickly buy something that should work, from someone reliable, in the context of: they can only invest so much time in the decision-making and purchasing process itself.

This isn’t a hit piece, which you can tell because I’m literally telling you to buy Teams Premium. There’s real value here, and it’s getting better. And we’ve written before about excellent choices we see the public sector making to exploit the potential of AI – a scroll through nominees for the various public innovation awards gives you a sense of a pretty exciting landscape. But as things stand, Copilot still feels like an expensive way to tick AI off your list.

But we want to hear from you:

• If you’re using Copilot in your organisation, what are you (mainly) using it for?

• Have you identified the pain points and found a way to solve them with specialist AI?

I genuinely believe I could love Copilot one day. Or at least be thankful for it. It might become like an arranged marriage where, on paper, everything is a good fit, you apply a positive attitude to making it work and then you really can fall in love and become a strong, happy unit over time - even if there wasn’t an immediate emotional connection. Maybe the real issue is that I’m a romantic and feel entitled to be more delighted by new things. But, for now, I’m not quite ready to settle for 'fine'.

‍