Is Sometime Now?

A common opinion is that pure science is a moral good in the world, and moral judgments are pushed downstream to the applications. Newton and Leibniz cannot be responsible for their work being used for missile guidance hundreds of years after its discovery. This same technology was also used in the interceptor missiles that are purely defensive. This situation is very common in the sciences where the typical response would be calling science a double-bladed sword; it can be used for good or bad, but typically the good outweighs the bad.

For developments in pure sciences, like nuclear physics, chemistry, artificial intelligence, is this true? Is the good outcome more likely? If it has been true, will it continue to be? But the variance often goes unnoticed, and the variance matters more when technology becomes more powerful. A rubberband usually returns to its initial state, but if it doesn’t, it never will again. Each further advancement in the sciences will bring more and more powerful and complex applications. These applications will grow exponentially as the Cartesian product across the advancements across fields. Even if the mean application stays positive, the negative applications multiply as the possibilities explode.

In narrow warfare applications, defensive measures are outpaced by offensive measures. This means that with each increase in capability, we need a considerably larger increase to defend against its worst cases. We have the technology now for super-sonic nuclear missiles and no technical defense against them. In 50 years, we will be able to defend against supersonic missiles, but it won’t matter against anti-matter bombs delivered by synthetic hummingbirds. We can produce genetically modified strains of smallpox that are even more deadly and virulent, but the ability to defend against them without societal shutdowns will be invented in a few decades. Defense lags behind offense, and it always will. There are a combinatorially large number of ways to abuse knowledge, that increases with the complexity and depth of that knowledge, but once an offensive vector is set, there are only a few ways to defend against it.

This offense-defense asymmetry will hold in each branch of science going forward, and beyond warfare this stretches to general misuse and negligence. Each branch of science must constantly weigh its pros and cons of further progress. And importantly, they all must stop at sometime. They all must stop at sometime. They all must stop at sometime. This is never discussed. Every scientific field must stop advancing at sometime, since eventually it will produce knowledge too dangerous to be known by too many people. You cannot develop physics to the point that individuals have the ability to create black holes in their basements, or to manipulate arbitrary matter one meter in front of them, or edit sequences of DNA from their garage.

Every scientific direction has a gradual transition from green to red. This is when its applications become so powerful that the variance in the outcomes gives a high enough chance for catastrophe. And like offense and defense, the position along this transition will be virtually unknowable with technology that exists when it needs to be known. For example, to understand the complex tradeoffs of nuclear physics, you would need to understand its applications to fission reactors, their potential environmental catastrophes, fusion reactors, the horrors of nuclear bombs, the tenuous but prolonged peace from mutually assured destruction. You would need to weigh the effects of all of these and combine them with their probabilities of actually occurring, potentially decades or centuries before they happen. This is not unknowable, but the technology didn’t exist 100 years ago when the physics was being developed and still doesn’t exist now.

This brings us to the current moment, which is the most dynamic moment for life and the most critical moment for humanity. Any planet, in any universe, if given enough time, must pass through the point when it builds tools that are more capable than itself in every way. This is when evolution finds a new path through direct optimization, instead of blind exploration. This is the transfer of power to a more scalable and capable being, which can optimize and self-optimize within unimaginable timescales. This moment reliably causes an explosion in all branches of science at the same time, from physics, chemistry, biology, and back to artificial intelligence. This change comes from so many angles at once and the applications grow as the product of all of these fields.

If there was ever a time that we reflect and consider whether we want to pause, it would be now before the complete integration of AI throughout the world. If you survey the civilizations across the universe and collect their experiences through this transition, I expect their advice would be straightforward. Either don’t build tools more capable than yourself, or only do so once there is global stability and coordination. Do not race through to build the most powerful intelligence possible. Do not release them directly into the wild without significant testing. Or release the earliest weakest AI as a vaccine for society in preparation for the more virulent strains. This is not an easy decision but likely the most important for life on Earth to date.