The world will invest a trillion dollars in artificial intelligence this year. The question Nick Bostrom has been asking for twelve years is no longer philosophical: AI governance is now decided in boardrooms.
By Ángel Bonet, founder of ImpactCo
When Nick Bostrom published Superintelligence in 2014, many boards read it as well-written science fiction. Twelve years later, the Swedish philosopher sat down with Scott Galloway and Ed Elson on Prof G Markets. There he explained why the warnings from AI researchers themselves are genuine. The episode title captures the shift: the industry has started listening.
That shift is no accident. In recent weeks, the chief executive of a major laboratory publicly asked the sector to slow down and spend more time on safeguards. A researcher at that same lab then resigned. No company, he argued, is acting responsibly enough. A debate that lived in Oxford seminars for a decade has reached the place where capital gets allocated.
AI governance and the numbers behind an ungoverned asset
Let’s put the data on the table, because this is where the debate stops being a matter of opinion.
- $1 trillion. Global AI investment forecast for 2026, of which $581 billion falls in the United States. (Goldman Sachs Research)
- 102%. The share of their cloud revenue that Amazon, Alphabet and Microsoft will plough back into capex this year. (UBS)
- 10%. The median probability that 1,580 AI researchers assign to human extinction or a permanent loss of control. (ESPAI 2024, published September 2026)
- 72%. Those same researchers who want society to prioritise safety research more.
UBS expects the big cloud operators to invest roughly $4.1 trillion between 2026 and 2028. That is more than three times what they spent in the previous six years. Clearly, this investment cycle has no precedent. It also moves faster than any regulator can follow.
Against that figure sits the longest-running survey of AI researchers, published this month by AI Impacts. For the first time in the series, the median researcher puts the risk of extinction or equally irreversible disempowerment at 10%. The mean reaches 18%. Meanwhile, only 22% think it likely that by 2029 we will understand the real reasons a system makes a given decision.
Now translate that into the language of an investment committee. We are deploying capital at record scale into an asset nobody can fully explain. Not even the research community can. Half of those researchers attach a tail risk that no board would accept in an industrial project. No CFO would approve that without asking who governs the asset and what it is for. That question is the heart of AI governance.
The false dilemma: accelerate or brake
Public debate has settled into a comfortable trench: accelerationists against doom-mongers. However, the data doesn’t support it. Asked what pace of progress would make them more optimistic, researchers split almost into thirds. Some 34% prefer to go faster, another 34% want to go slower, and 29% would keep the current speed.
Bostrom himself breaks the caricature. In recent work, he argues that developing general intelligence sooner could extend the life expectancy of people alive today. That holds even under high probabilities of catastrophe. He also warns against readers who cast him as an enemy of AI. His position is more uncomfortable and more useful: the technology may be the greatest opportunity in history or its final mistake, and speed is not what decides which.
Speed without direction isn’t progress. It’s inertia with a budget.
Direction is the decisive variable. Moreover, the direction of any technology is set by whoever funds it, whoever governs it and whoever answers for its consequences.
AI governance starts on the share register
Here comes the part that matters most to those of us working in the purpose economy. In 2025, Bostrom published a governance proposal for artificial general intelligence that avoids both a state-run Manhattan Project and a CERN-style body. His model, Open Global Investment, places the lever somewhere very specific: capital.
The proposal puts advanced AI inside listed companies with broad international shareholder bases. Those companies would run governance above the Fortune 500 standard, with public benefit corporation charters. Ideally, foundations and non-profits would hold a significant share of their stock. Their mandate: to benefit humanity as a whole. Bostrom adds an argument any chair will recognise. The risk of public backlash gives directors business reasons to provide public goods, even in a purely for-profit company.
So one of the most cited thinkers on existential risk concludes that the safety of the most powerful technology in history rests on three things. It depends on who sits on the cap table. It depends on how the board works. Finally, it depends on whether purpose is written into the articles of association or only into the annual report. In other words, AI governance is corporate governance.
That is the distinction I have defended for years: capital that grows roots against capital that extracts. Extractive capital tries to capture the value of AI fast and pass the risk to others. Capital that grows roots sees it differently. An asset that reshapes the work, health and information of billions is only sustainable if it shares value and answers to those it affects.
When intelligence is abundant, purpose becomes scarce
There is a second Bostrom book that boards should read before the first one. In Deep Utopia, he asks what happens in a world where technology has solved the instrumental problems. What is left to do once machines work, calculate and create better than we do? His answer points straight at the question of meaning.
Applied to business, the reasoning is devastating. Once intelligence becomes a raw material you rent by the hour, competitive advantage will no longer come from knowing more or processing faster. Instead, it will come from knowing what for. Organisations without a clear reason to exist will end up automated from the inside. They will optimise, with enormous efficiency, towards goals nobody chose. Those with a clear purpose will use AI to grow and create economic, social and environmental impact at once. As we have argued before, corporate purpose has become a survival strategy.
Three questions for your next board meeting
No board will decide the future of superintelligence. Yet every board owns its own AI governance. Each quarter, it decides how the company buys, deploys and supervises the AI it already uses. Start with three questions.
- What do we use AI for? Not which processes it automates, but which purpose it serves. If the answer is only cost-cutting, your supplier is designing your strategy.
- Who decides, and to whom do they answer? You need real human oversight, traceability for critical decisions, and someone on the board who understands what they are approving. That is where AI governance either exists or doesn’t.
- Who receives the value it creates? Employees who reskill, customers who gain trust, territories where productivity grows.
Bostrom’s warning has finally reached the boardroom. The answer, however, won’t come from a laboratory. It will come from whoever decides what this technology is for. That decision is what AI governance really means.
Tech & AI Trust Architecture is ImpactCo’s new practice for chairs, boards and investors who want artificial intelligence to advance their purpose rather than replace it. To discuss how to design trust into your organisation’s technology, write to us at tribu@impactco.es.