Apparently I am supposed to have a view on the probability that AI will kill us all. It’s now the first question a lot of my friends ask me. They used to just ask me how work was going.
The problem is I don’t have an answer. If someone gives me a percentage likelihood of an AI-powered apocalypse, I have no sensible way of checking their working. I can’t dismiss the warnings from people who understand these systems far better than I do, either. I can see how the doomsday scenarios might work. I also tend to look for the points at which a person could notice something was going wrong and intervene. And I have quite a lot of faith in people. What I’m less sure of is whether we’re giving them enough information, authority and time to do the right thing.
Where are the edges?
What I do find fascinating is that some of the people building the most powerful AI systems are asking for guardrails. Anthropic’s chief executive, Dario Amodei, has called for a slower pace of development and for independent evaluators to have access inside AI companies. Anthropic has since announced a partnership to start doing that, although the details are still being worked out.
That’s confusing, right? Aren’t the people racing to build AI supposed to be against anyone telling them what to do?
But I think we misunderstand our relationship with rules. We love to complain about red tape, nanny states and straitjackets. Then we get near something that might actually hurt us and become rather more keen for an expert to tell us where the line is. It’s the realisation we’re near the edge of a cliff and it’s now too dark to see it. We want someone to pop up a little sign, warning us about the risk of death. Or even install an actual barrier, if we think it’s serious enough.
I don’t want the person who designed a new lift to tell me that safety inspections would stifle innovation. I want to know someone has checked it before I step in. Ideally, I’d like them to have considered what happens when a part fails, rather than waiting for the first passengers to find out the hard way.
That doesn’t mean big tech companies should be allowed to write all their own rules. They have expertise and evidence the rest of us need, but they also have products to sell and competitors to beat. An independent evaluator is interesting because they might be able to see more than a well-practiced demo. Whether Anthropic’s arrangement provides meaningful independence will depend on how it works in practice.
Elon Musk’s response to Amodei was simply, “Dario is right.” I’m glad he said it, although I’d be interested to know what he intends to do about it. I wrote earlier this year about Grok and their conscious choice to market fewer safeguards as part of its appeal, even as people were using AI image tools to harm women and girls. If we’re asking the industry to take responsibility for risks that might emerge in future, we should also ask how it deals with harms people can point to now.
Naomi Klein and Astra Taylor’s new book on tech billionaires and the end of the world has been on my mind while writing this. We are spending a lot of energy imagining a future in which AI becomes too powerful for humans to control. Meanwhile, a remarkably small number of humans have an enormous say in how powerful it becomes, what it is allowed to do, and which harms count as urgent. It feels like those decisions need to be open to more than just the people who own the companies.
Kieron White and Neil Watkins discussed the current warnings on the Leading AI podcast. Kieron was asked about a claim that there is a 10% chance AI will wipe us out. His answer was, essentially, that neither he nor the person asking was qualified to assess that number. Because that’s the truth. You can take a risk seriously without pretending you know its precise probability.
What happens when we hand over the keys?
I also wonder whether we spend so much time arguing about the speed of the big AI companies that we miss another speed altogether: how fast the rest of us are giving these systems authority.
An organisation starts with an assistant that can search its documents and answer questions. Someone suggests giving it access to customer or client records, so it can provide a more useful answer. Then it gets access to a live system. Perhaps it can update a record, send a message, book an appointment or approve a routine request. Each step might be perfectly sensible. Together, they change what happens when the system gets something wrong. They also change what happens when it pursues a perfectly reasonable goal by a method nobody intended. I wrote recently about why agents need principles, not just personality: telling an agent what you want is only part of the job. You need to tell it what it must never do to get there.
Organisations need to go a step further. Who decided the assistant could take that action? What information can it use? When does it have to ask a person? Can someone see what it did and undo it? And would the person affected know they were dealing with AI in the first place? These aren’t questions for a distant future in which machines run everything. They’re questions for a housing association answering a complaint, a business handling a customer account, a college advising a student, or a team using an agent to manage its own workload.
The mistakes we nearly make
There’s a useful idea in management theory about giving people room to experiment within clear boundaries, and learning from the mistakes they nearly make. Harvard Business School professor Amy Edmondson calls it intelligent failure: when you’re trying something new, you set boundaries around the experiment so a mistake teaches you something without causing damage you can’t put right. Her work also makes the case for learning from near misses — all the things that nearly went wrong, but were caught in time. That only works if people know where the boundaries are and feel able to say when they’ve come close to one. Long story short: the boundary tells you what you can try; the near miss tells you where the boundary may need to move or become stronger.
AI needs both. If an assistant produces a convincing but wrong answer and an employee catches it, that is useful information. If an agent tries to take an action outside its authority and a safeguard stops it, someone should know. A system that prevents one mistake but teaches nobody anything leaves the next team to discover the same problem for themselves.
I’ve often argued that we should worry more about what people choose to do with AI than about the AI itself. I still think that’s true. But I’m conscious of how close it can sound to “guns don’t kill people, people kill people”. We can hold someone responsible for what they do with a tool and decide that the tool is powerful enough to need rules about who can have it, what it can do and what checks must happen first. Those rules will look different in different places. The principle is that human responsibility doesn’t begin and end with the person pressing the button.
None of this is an argument for leaving a useful tool untouched. Neil and Kieron have also talked about an organisation where staff rarely used an AI tool that could give them a sourced customer summary in seconds. The tool worked; the organisation had not made enough room for people to change how they worked. Not getting people across the threshold has a real cost too.
So I’m wary of treating “speed up or slow down?” as one question with one answer. You know us better than that. But we should pay attention when the people at the frontier say they need more oversight, and insist that the oversight is credible. Inside our own organisations, we can make a more immediate choice: give people useful tools, be mindful about the authority we give those tools, and learn when something nearly goes wrong.
I still can’t tell you the probability that AI will kill us all; no one can. But I think we should be able to say who gave it permission to act, who is watching, and whether they can still tell it to stop.
