This Oxford professor thinks artificial intelligence will destroy us all

Nick Bostrom:What I mean is any intellect that radically outperforms the best human minds in every field, including scientific creativity, general wisdom and social skills, roughly speaking.

I guess I have a bunch of different theses in the book. The basic idea is that I think at some point we will create machines that are superintelligent, and that the first machine to attain superintelligence may become extremely powerful to the point of being able to shape the future according to its preferences.

So that's step one in the argument, and step two is that it looks quite difficult to design a seed AI such that its preferences, if fully implemented, would be consistent with the survival of humans and the things we care about. And that, therefore, we face this great challenge of solving the control problem for advanced artificial agents that we've got to solve before we figure out how to solve the other problem, which is how to create machines that are superintelligent in the first place.

Nick Bostrom:60 years is not very long in the scheme of things. There are a lot of things that human civilization has accomplished now that it took a lot longer than 60 years to do.

We did a survey of opinions among some of the leading experts in artificial intelligence, which I report in the book. One of the questions we ask was, "by what year do you think there is a 50 percent probability that we will have human-level machine intelligence?" And the median answer to that was 2040 or 2050, depending on precisely which group of experts we polled.

There is a lot of uncertainty around that of course, it could happen much sooner or much later, but it suggests that it's a very mainstream opinion among experts to think that there is a real chance that this may happen over the next few decades, or at least in this century.

Nick Bostrom:It is certainly possible that this expert group is biased. I don't buy the particular argument by Dreyfus, or any of the other attempts to show that it is, in principle, impossible to create generally intelligent machines.

We have an existence proof of general intelligence. We have the human brain, which produces general intelligence. And one pathway towards machine intelligence is by figuring out how the human brain accomplishes this by studying the neural network that is inside our heads. We can, perhaps, discern what algorithms and computational architectures are used in there and then do something similar in computers. That might not be the way that we will actually first get to the goal, but in principle it is something that we could do to make incremental progress.

An even more radical approach is to literally copy a particular human brain, as in the approach of whole-brain emulation. We would not have to understand at any higher level of description how the brain produces intelligence. We would just need to understand what the components do.

It might well be that the purely artificial approach will beat these other approaches to the punch. Another consideration is that even if it were, which I do not believe it is, outside the reach of current human intelligence to conceive and engineer artificial intelligence, human biological intelligence is itself something that can be enhanced, and I believe will be enhanced, perhaps in latter half of this century. So we also need to consider that there could be these much enhanced human scientists and computer scientists who will be able to make progress even if we were stumped.

What if Watson tricked Ken Jennings into doing its bidding? (Ben Hider/Getty Images)

Nick Bostrom:Well, there are two different questions there. One is whether they want to do that, and the other question is how they could do, assuming that they did want it.

If we begin with the latter of those two questions, I think that the real danger lies not in the body of the AI, but in the mind. You could imagine different robots equipped with machine guns or rockets or stuff like that. That's not where I think the danger lies.

Suppose that we initially put it in a box and disconnected the internet cable and allowed it to communicate only by typing text on a screen in the hope that that would enable us to keep it confined in this way. But, in order to get any use of it at all, we have to interact with it. At that point a huge vulnerability opens up, which is the human itself. Humans are not secure systems. Even today, mere mortal humans often succeed in manipulating other humans, and hackers often use social engineering to break into computer systems. So if we imagine a superintelligent manipulator and persuader, it seems like it would likely succeed in finding a human accomplice or otherwise using us as its arms and legs if it didn't have direct access to physical manipulators.

Now the question as to why it might want to do this is interesting and important and my claim would be that for a wide range of different possible goals that the superintelligence might have, achieving access to physical resources would be a convergent instrumental value. That is, having access to resources is something that is useful if you want to achieve a goal for a very wide range of different goals.

There are some other instrumental goals as well that would be convergent for a wide range of tasks, like making yourself smarter, preventing humans from switching you off, inventing new technologies, preserving your current goal system: these are all instrumentally useful sub-goals to a wide range of possible final goals that the superintelligence might have.

Nick Bostrom:Perhaps. I call that "tool AI," and there is a subsection on it.

Today, if you had some mass AI and you tried to enlist it to help you find a proof, maybe that would involve taking some lemmas and seeing whether combining them in a certain sequence would deliver the proof.

But, with a more powerful general AI, you might also find different plans. For example, how to allocate your computational resources, or how to obtain more computational resources that would then make you more likely to discover the proof. If you start to think about how to obtain more computational resources, one of those plans might call for taking over other microprocessors so that you could instantiate your program on more hardware, and then a whole host of other things come with that.

Say you go from human-level to superintelligence within four hours...

To achieve general intelligence as opposed to domain specific intelligence, you would need a very sophisticated model about how the world works, how humans work, and how physical objects work. And once you are searching for plans in that very general world model, you might have machines that look as if they have will.

One approach that one might think would obviously be the safest bet is to try to engineer a system that would not be of this agent-like character: that is, to try to build a tool AI. It is still worth exploring that further, but it's a lot less obvious than it looks that it actually is a fruitful avenue.

For a start, you might end up with an agent even if you didn't set out to create it. So, if there are these internal processes within the system that amount to very powerful search processes for plans, it might well be that one of these internal processes will exhibit agent-like behavior, even if you didn't define the whole system to be an agent. And these agent-like processes that emerge spontaneously might then be more dangerous, because they were not put in on purpose.

Nick Bostrom:I unfortunately don't see much hope at the current time, if we are thinking about regulations that would be helpful in this context. I think, at the moment, it would make more sense to try to accelerate work on the control problem, rather than trying to slow down work on AI, because there are just so many incentives for various people and companies to try to make advances in faster hardware, better understanding of how the brain works, cleverer algorithms, etc.

It is very hard to see how that would slow down in any significant degree, at least at the current level of understanding of what the problem is, whereas even fairly limited resources could do a lot to speed up our progress towards solving the control problem. It might only be half a dozen or so people working on the control problem today, whereas there are tens and tens of thousands if not hundreds of thousands of people working towards general AI.

Skynet is a great example of a singleton, though it relies on androids like this T-800 rather than humans to interact with the outside world. (Universal History Archive/Getty Images)

Nick Bostrom:I discuss two different classes of scenarios. There is the scenario where the transition to the machine intelligence era is very rapid, and you have an intelligence explosion at some point, where it is likely that there will be one superintelligence able to form a singleton, a world order where at the highest level of decision making there is only one agent.

Say you go from human-level to superintelligence within four hours or two days or some very short period of time like that. In that case, it is very unlikely that there will be two or more development teams that undergo this transition in parallel. Usually, the leader is at least two weeks or months ahead of the next best project. In that case, you might have this singleton outcome, where there is one thing that will shape the future according to its preferences.

But then there is this other process scenario, the multipolar outcome, where you have a slower transition playing out over years or decades. In this case, there probably would be multiple systems that are roughly comparable at any given time, such that none of them was able to dictate the future. I think the singleton outcome is more likely, but we certainly cannot, at this stage, preclude the multipolar outcome either.

In the singleton outcome, everything would depend on what this AI would want. And that, in turn, would depend on whether we had succeeded in solving this very difficult control problem, which is an open question. I am hoping my book will increase the probability that we will solve it by attracting some of the brightest minds to work on this problem.

In the multipolar outcome, it might be harder for us to influence the long-term future, but it might be that various institutional arrangements might make a difference. But there is the question of whether those institutions would be stable, as these digital minds proliferate and continue to evolve in competition with one another. I think in that scenario, maybe some form of global coordination will be necessary in order to avoid developments going in whichever direction the AIs are pointed, which might not be very friendly to human values.

This interview has been edited for clarity and length.