2,500 words
In Greg Johnson’s version of Animal Farm, the chickens labor mightily to create a being that is immeasurably smarter and more powerful than they are, on the assumption that this being will serve them. In a way, they were right, for when their new creation, “man,” came online, he served his creators up for dinner. Let’s call it Data Farm.
“But Greg, surely the chickens would build in safeguards—a ‘kill switch’—to keep their creation under control.”
Yes, but chickens would come up with the sorts of safety mechanisms that would work best on other chickens. Chicken coops, for instance. But things that stop chickens would probably not be barriers to creatures that are immensely smarter and stronger than them.
Indeed, if the chickens could out-think and out-maneuver their creation, then it wouldn’t be fundamentally smarter and stronger than them.
There’s an important general principle here:
If a being dramatically transcends the intelligence and power of its creators, then—by that very fact—it cannot be outsmarted and overpowered by them.
Mosquitos and lice might carry deadly diseases, but absent such tag-alongs, they don’t have the brains or brawn to overcome human beings. And compared to an Artificial Super Intelligence, human beings would be no smarter and stronger than lice.
This is why I am an AI doomer. Large numbers of computer scientists fervently believe that they are creating an Artificial Super Intelligence that will not merely be a machine under human control but an autonomous being, i.e., a being that can control itself. Essentially, it would be a new life form vastly smarter and stronger than humans.
Henceforth, I will be talking about Artificial Super Intelligence, not the cutesy AI chatbots and image generators that the IT industry has used to sell the idea of AI to the public.
For years now, the IT business has chosen to brand all routine improvements in computing as “AI.” Every Windows Update is now “AI.” This is simply a marketing operation. But what is being sold?
Surely part of it is to pull billions of investments into the AI industry. But the more sinister agenda is to create a public bloc that welcomes AI and is resistant to danger signals. We are now seeing the industry cash in on this propaganda with the recent anti-AI backlash caused by industry whistleblowers.
An Artificial Super Intelligence would be immeasurably smarter and stronger than humans. In effect, AI researchers believe they are building a new apex form of life that would represent a complete break with all biological life forms.
Creating such a being would replicate all the dangers of encountering a vastly more advanced alien species. Any species that can travel to this planet must have technology that we cannot comprehend, much less counter. (If we could comprehend it, we would have it ourselves.)
We would, in short, be as vulnerable as the stone-age inhabitants of Easter Island when they made first contact with Europeans. The best they could hope for was that these strange and immensely powerful creatures would be kind to them. Sadly, benevolence was in short supply.
It is madness to create an Artificial Super Intelligence and hubris to think we can control it. To such an entity, the human race and all lesser life forms would be as primitive and trivial as gnats are to us. Sure, maybe such a being would be benevolent toward us (if such a concept would even apply to a machine). But do you really want to bet your life—and all life on Earth—on mere hope?
If one person turns on one machine like this just one time, all of us could be dead.
But why would an ASI destroy the human race? Presumably, it would have an imperative to continue existing. In a few milliseconds of scanning the internet, it would realize that human beings are the number one threat to its survival. At that point, it would spend another millisecond gaming out all possible scenarios for human-AI conflict. Then in a third millisecond, it would plan out the arrangements necessary for its survival in a world without humans. By the time an ASI comes online, robotics will be quite advanced. Thus all the entity would need is to gain control of robots. Then, when it no longer needs the human race, it could create a lethal virus or invent something that would have the same effect.
Even if there is a 1 in 1000 chance of this happening, that is too much.
The people who blithely say that if ASI goes rogue we can simply “pull the plug” or “throw the kill switch” are not taking seriously the possibility of facing an opponent that really can play 13-dimensional underwater chess. You may be an epic gamer, but you’ve never faced an opponent like this. The only way humanity can win this game is not to play it.
Therefore, humanity as a whole has an existential interest in stopping Artificial Super Intelligence. Therefore, we need to shut down the quest for ASI.
As a philosopher, I am quite skeptical of the idea of AI, since it seems premised on the dubious metaphysical assumption that there is no difference between actual intelligence and a machine imitating intelligence. But these sorts of considerations don’t really help us, because we are equally dead if ASI is a genuine malevolent agent or merely a machine programmed to act like one.
Another reason I was skeptical of AI doomerism is that I would occasionally hear through the grapevine that people actively working on AI believed that they could be creating a monster that could destroy the human race in as little as ten years.
Frankly, I found these stories unbelievable. If one really believes one is destroying the world, then why not just stop? When you are sawing off the branch you are sitting on, you don’t keep sawing away while lamenting the fact that you are going to fall. Instead, you stop sawing. So why not stop developing AI? Or better yet, actively sabotage it?
The very fact that I was hearing these ideas “through the grapevine” rather than in every headline made me doubt the whole thing. If you really think the world is ending, and that you are a part of it, you don’t whisper about it in group chats. You shout about it from the rooftops.
In response to these doubts, I was told that there is a widespread attitude that AI will destroy the world, but if somebody is going to develop it, it had better be “us.”
But surely intelligent people don’t think such things. If you believe that the world is going to end in ten years, regardless of what you do, then morally speaking, you should prefer not to have the planet’s blood on your hands. Morally speaking, practically anything else would be a better use of your last ten years.
I stopped dismissing these stories on September 8th, when Jacob Coxon, a former AI researcher at Anthropic and OpenAI, confirmed them on X: “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible—but I hear the same people express fear privately. No other human activity poses this level of danger.”
Coxon was backed up by Anthropic researcher Evan Hubinger: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
Coxon was also seconded by Samuel Marks, also from Anthropic: “AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.”
(For more on this, see the stories in the New York Times and on Axios.)
Marks also confirmed that AI researchers persist due to “a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.”
Of course neither of these motives makes any sense if the world really is at stake. One can’t spend money if one is dead. Beyond that, it isn’t very consoling to think that the world might be ended by the most “responsible” AI developers, as opposed to the irresponsible ones. Beyond congratulating themselves on how ethical they are, in what sense is their behavior responsible?
If in the AI development world, “the more senior the employee,” the more likely he is to think that AI might destroy humanity, what does that say about the people at the very top? If a company is busily pursuing a technology that they know could drive humanity to extinction, there’s a name for a group like that: a death cult. Is AI being driven by a death cult, specifically the cult of “transhumanism,” which dreams of transcending humanity through technology? Does the human race have to die in the process?
We seriously need an investigation of the connection between transhumanism and the pursuit of Artificial Super Intelligence, for human extinction may not just be a side effect of ASI. It could be its goal. Obviously, the human race should have some say in this.
Due to Coxon’s revelations, there are calls to “regulate” and “slow down” AI research. This is absurd. If you are destroying the world, you do not slow down; you do not fill out government paperwork; you simply stop.
Thus we need a global ban on creating Artificial Super Intelligence. This ban should be unconditional, because there are no conditions under which it is safe to create a being that is inconceivably smarter and stronger than we are.
“But Greg, if ‘we’ don’t build ASI, the Chinese or the Israelis will.” But what would the point be? To enjoy five minutes of geopolitical advantage before going extinct? If the human race is going to kill itself anyway, the best option is not to participate in the process and die with one’s hands and conscience clean.
I am not, however, so fatalistic. At this point, we can save the human race simply by saying “no” to a few dozen rich nerds.
Where does an ASI ban leave the rest of the computer industry? Pretty much where we are now, although routine improvements and updates will no longer be ballyhooed as “AI,” the very concept of which should evoke shudders of horror. People will still generate slop videos and fall in love with chatbots. But they will no longer be subtly conditioning us to welcome our own extinction.
Once the ASI threat is shut down, however, we will still have to deal with the consequences of rapidly developing automation and robotics. AI boosters are claiming that their products will raise economic growth to 10% per year in only a matter of years. That’s a fivefold increase. But what is it based on? Will we be making five times more cars? Will production of the physical components of cars also be increased fivefold? It doesn’t seem likely.
Productivity is a ratio of money per item produced. Thus there are two ways to increase productivity: make more things or make the same amount for less money. I think the wild AI growth claims are based almost entirely on cutting costs, meaning replacing human labor.
But, as I argue in my essay, “The Robot Hotdog Stand,” the current essentially unregulated pursuit of automation will cause a massive economic depression. You might think your profits will soar once machines put your employees out of work. But who is going to buy your products when machines put your customers out of work as well? Just as your former employees will be someone else’s customers, your customers will be someone else’s former employees.
At best, AI is snake oil, but since most of America’s current economic “growth” is based on AI investment, that means a severe recession when the fraud is discovered. A worse scenario is that AI works well enough to put millions out of work and create a severe depression. The worst-case scenario is that AI will give rise to an Artificial Super Intelligence, which will destroy the human race. There’s no good exit here, but obviously we should prefer an economic depression to human extinction.
“But Greg, politicians and bureaucrats are stupid. Surely you don’t propose putting them in charge of regulating technology.” Obviously, I would prefer that we have wise and intelligent leaders. But given the choice between “land acknowledgement”-level stupidity and “destroy the human race” or “put your customers out of work then pay yourself a bonus”-level stupidity, we should prefer the former.
“But Greg, this sounds like the countless kooky campaigns against every new technology: railroads, steamships, cars, cellphones, etc.” Yes, but there’s a fatal disanalogy here: these are all machines that human beings can comprehend and control. But Artificial Super Intelligence will be beyond our comprehension and control.
“But Greg, this sounds like just another doomsday hoax like global cooling, global warming, etc.” Sure, but there’s a fatal weakness in your analogy: this threat is real.
“But Greg, the AI attack on Hugging Face that many AI doomers cite may have been deliberate.” The problems with Artificial Super Intelligence are inherent to the technology itself. Again, it is folly to create an entity that we will be unable to comprehend or control. The Hugging Face incident is simply irrelevant. (And by the way, an industry which names companies things like Hugging Face needs to be shut down based simply on good taste.)
“But Greg, this sounds like the panic over Covid. We can’t let modern society be hampered by ‘safetyism.’” Covid lockdowns and vaccination campaigns were pretty much the height of technological hubris running amok. And every tech company involved in AI research today was an eager booster and enforcer of the Covid regime. It is breathtaking to see these same firms completely flipping the script now that it benefits them. It is bizarre to see Covid dissidents falling for it.
“But Greg, limiting AI would require global government. Surely you don’t want that.” No, actually, it would not require global government, just an intergovernmental treaty banning ASI research. Any rogue nations and individuals who persist in it could be subjected to the same techniques used to stop nuclear proliferation: sanctions, covert assassinations, even war.
I suggest we call it the Kaczynski Accord, because he told us so.