Full transcript: Stop AI Development Now! The Dangers Of Chat GPT-4 on London Real
0:00 Brian Rose: This is London Real. I am Brian Rose. My guest today is Professor Max Tegmark, the cosmologist, physicist and artificial intelligence expert. You have spent the last 20 years as a professor in the department of physics at MIT and have published more than 200 technical papers. In 2017 you wrote the best-selling book Life 3.0: Being Human in the Age of Artificial Intelligence, where you explored how the rise of AI has the potential to transform our future more than any other technology in history.
You’re also the co-founder and president of the Future of Life Institute, a non-profit organisation with the goal of reducing global catastrophic and existential risks facing humanity. Just recently the Institute drafted a letter calling on major AI developers to agree to a six-month pause of any systems more powerful than GPT-4. You believe that AI development is moving faster than anticipated and we need a serious public debate regarding AI’s potential impact on humanity. Max, welcome to London Real.
Max Tegmark: Thank you.
Brian Rose: Great to have you on here. I’m going to jump into it. I think you have said that when it comes to AI we’re not in an arms race, it’s actually a suicide race. I was wondering if you could tell me the state of AI right now, where you think we’re headed and why everyone needs to pay attention to this now.
4:08 Max Tegmark: Yeah, great question. So let’s unpack it bit by bit. After 13.8 billion years of cosmic history and hundreds of thousands of years of human history, we’re approaching this fork in the road, the most interesting juncture really of all, I would say, where we’re either going to become empowered as a species to really be the masters of our own destiny and use artificial intelligence to solve all the problems we’ve been stumped by so far, help life flourish like never before, not just on Earth but probably also out there in our vast cosmos, or we’re going to go extinct or get replaced by entities that don’t really care about us, and lose basically everything we’ve got.
So this has made me incredibly motivated to think about this. About seven years ago I shifted my MIT research group over to doing AI research, super fun, and it’s also made me really motivated to help catalyse this conversation about which way we are going to go in this fork. I think it’s a really inspiring topic to talk about. It’s not a depressing topic like nuclear weapons, where either they get used and it really sucks, or we don’t use them and it’s kind of neutral. This is one where it’s either going to go terribly bad or it’s going to be incredibly inspiring, better than our ancestors ever dreamed of.
And I think the risk is that we’re just so eager to get all the upside that we rush a little too much and cut a few too many corners on safety, and because of that we lose it all, lose control over this whole business. Because it is possible, I believe, for us to build machines that are way smarter than us and still either control them or make sure that they help us. After all, you and I have both lived in the presence of smarter beings ourselves. When we were little kids they were mommy and daddy, right? And that worked out quite well because their interests were aligned with ours.
But it’s not guaranteed by default, right? The woolly mammoths on Earth had to try out living with more intelligent beings, namely us. Have you ever seen any around? It didn’t turn out great for them. So let’s put ourselves in the situation where, if we build AI, we’ve figured out how to keep it doing what we want, aligned with our goals, and not make ourselves the woolly mammoths. The problem we’re having is that the research on that stuff, which I think will succeed if it’s given enough time, is still lagging behind the research aiming just to make it more powerful and earn us more money. That’s why we called for the pause, in short: to make sure that we have enough time to actually get it right, get all the upside and not squander it.
7:32 Brian Rose: Recently I’m sure you’ve been watching the interviews of Geoffrey Hinton, the kind of godfather of AI, who was at Google and has now left Google, I think so he can speak freely or do more to talk about the dangers of AI. But he came out and said it was too late for a letter like that. What is your response, and was the letter effective? I mean, we’re still talking about it, so as a PR event I’ve got to take my hat off to you. Was it ever meant to get a pause, or was it really meant to get more attention on this topic?
Max Tegmark: My personal goal with the letter was actually to mainstream the conversation on whether we should pause or not, so that people felt comfortable having it. Because, to be quite candid, many people I really respect in the field felt uncomfortable even suggesting it. It was almost like a taboo, because people might think you’re some sort of weirdo Luddite scaremonger who doesn’t understand AI, right? And having people like Yoshua Bengio, who shared the Turing Award with Hinton for pioneering deep learning, signing as signatory number one, saying we should pause, what it really did, I felt, was make everybody else feel comfortable joining the conversation, whatever side they were on, which is why we’ve had so much conversation since the letter. That to me is step one. The next step is to see where that is all going to go.
9:20 But I think the reason we haven’t seen a pause already, and the reason there is a lot of, in my opinion, corner cutting on safety, is not because the people leading the companies building them are somehow not concerned. In fact, even in the last month I’ve spoken with Sam Altman and Greg Brockman, who lead OpenAI, I’ve spoken with Demis Hassabis from DeepMind, with Dario Amodei from Anthropic, with Connor Leahy from Conjecture and so on. All of these people are quite idealistic. They really want to use AI to make the world better and they’re all very mindful of the risks.
I really tip my hat to Sam Altman for having the courage to acknowledge this very publicly. He said quite recently that the worst-case scenario is lights out for all of us. And all of these people have been conveying that this could certainly cause human extinction, just like the woolly mammoth went extinct when it had to deal with more intelligent hominids. The reason why we’re still not paused, I think, is not because these are not good people, and it’s not because they’re stupid and don’t understand the risks.
10:40 It’s because of what you referred to at the beginning: Moloch, the demon that keeps pitting everybody against each other. If one company tries to pause alone, the only thing that’s going to happen is they’re going to get outcompeted by other companies, and probably even before that happens, the CEO who did the pause gets fired, right? This is not a new thing. Moloch has been with us throughout history. Take overfishing, for example. Canadian fishers collectively managed to make it so that there was suddenly ten times less cod to fish. Why was that? Was it because the fishers were evil? No, of course not. But they all knew that if they didn’t fish, everybody else was still going to keep fishing, and they had to take care of their families.
Economists call this the tragedy of the commons, and economists and others have written books and papers studying how sometimes you can put people in a situation that brings out the best in them, and you can also put people in situations that bring out the worst in them. Put people into the Hunger Games, or whatever: Moloch in action. In other words, all of these leaders of these companies need help to change their incentive structure so everyone can pause together. That’s ultimately what the letter is about. If public opinion and governments come in and say, hey, you folks need to become less like the Wild West and more like biotech, now everything changes for the better.
12:22 Think about it. In biotech, suppose a company says, we have this fantastic new vaccine, we’re just going to start selling it in supermarkets before doing any safety testing. What do you think the government would have to say about that?
Brian Rose: They’d shut them down.
Max Tegmark: Yeah. And then if the company comes back and says, you government folks, this is so ridiculous, you need to tell us how to make it safe, the government would be like, go away. This is your responsibility: to do the safety testing, to do the clinical trials and present the FDA here in the US with evidence that it’s safe. And if you can’t do that, well, go back and work harder. This is how we should obviously have it in all other fields, including AI, where if you’re putting out a system which has the potential to create harm, it should be the responsibility of the companies to demonstrate that it’s safe. That would end this race to the bottom. This is the way we combat Moloch. This is the way companies are really incentivised to all work on AI safety enough that they can demonstrate the safety, which again is doable. It just takes a little bit of time.
Brian Rose: Your concept of Moloch is fascinating. To quickly define it for people that don’t know, you might say that Moloch is a metaphorical term for irrational collective behaviour. And I think you’ve said the evil thing about this monster is that even though everyone sees it and understands it, they still can’t get out of the race, which is ultimately to the bottom and potentially everybody’s demise, or the demise of resources. This has happened throughout history, and I’d say it’s pretty much your signature IP.
Max Tegmark: I want to be clear: Moloch is not my idea. The demon has been with us since the beginning of history, and there have been many wonderful things written about it. Scott Alexander wrote a fantastic piece on his blog Slate Star Codex called Meditations on Moloch, which I would highly recommend. In religions around the world the devil goes by many different names, and Moloch goes by many different names also. What I’m doing is just pointing out here that the villains in this are not the CEOs of the tech companies, and it’s not AI researchers. It’s Moloch himself. That’s what we’re up against here.
And I think your definition was really, really good. Everybody can see that we’re being pitted against each other, yet it’s still rational for every one person to keep fighting against the other people in the race, because it’s set up that way. The only way we can all get out of the race is by collaborating with each other instead.
15:30 So that this doesn’t sound too depressing, I think it might be worth just taking a minute to remind ourselves how we’ve successfully defeated Moloch in many other battles throughout history. Evolution, first of all, has built in a lot of collaboration mechanisms, not just in us but in other animals too. Generally animals that could kill each other, like cats or lions, have evolutionary instincts that inhibit them from fighting unnecessarily. We’re the same. If two dudes get into a drunken bar fight, there’s still this deep inhibition towards actually killing the other person, because it turned out that the groups in evolutionary history that didn’t have that inhibition got outcompeted by other groups that could get along better. I’m just going to have a sip of water.
And then early humans took this to another level. We invented gossip, which is also a great way to collaborate, because now all the moochers and cheaters and liars get a personal incentive to be more collaborative, because if they aren’t, they start getting socially punished by others. And then as societies grew and we ran into a lot of strangers we’d never met before, so that the gossip and reputation thing didn’t work, we invented a legal system to again defeat Moloch. Even if two strangers are dealing with each other, they know that if they kill each other or violate the contract they signed, that’s a bad idea just for them, egoistically.
So what these anti-Moloch collaboration strategies always do is make your personal incentives aligned with the greater good. That’s what we need to do here also: make sure that the incentives with AI are aligned with the greater good. There’s a lot of nerd research, and we’re doing some here at MIT, on how you can make AI have incentives aligned with their owners’ incentives. You also obviously have to make sure that the corporations that own the AIs have incentives that are aligned. That’s where we’re failing right now, because they have an incentive to make money.
If it were biotech, there would be all these regulations to meet, or for that matter almost any other industry. If you’re a car company you have a lot of safety regulations. You can’t build a nuclear reactor on Trafalgar Square without first demonstrating to the British government that it’s safe. It’s actually AI that’s the weird exception, where there are no meaningful regulations at all, and that’s just because it’s the new kid on the block and policymakers haven’t had time to catch up.
18:35 Brian Rose: We’ve been following the blockchain industry for the last couple of years, and the common thing I’ve heard from AI people is that the government can’t figure out how to regulate blockchain, and it’s much, much simpler than AI. So does it have any chance of regulating AI as it continues to develop exponentially?
Max Tegmark: I think it does. Obviously we can’t expect all our politicians to have PhDs in computer science, but they certainly can work with technical people from the field who are very eager to help. In fact, just yesterday the European Parliament, if I’m correctly informed, voted yes for the EU AI Act that’s been developed, which is the first really meaningful regulation in the West on AI. That’s a huge step forward. We just need to give policymakers a little more time to catch up. This is another point of the pause, because if we just let the technology get driven by market forces, we go as fast as possible and policymaking will never catch up.
But if we can pause for six months and use those six months with people from industry, from academia and from policymaking getting together and establishing the safety standards, you have something like an FDA, but not for drugs, for AI. These are the safety standards. Now you can kick the ball back to the market and say, okay, everything you release has to be above this bar. If it’s just some kid making a funny little computer game, there’s no safety risk, so no change. But if it’s somebody releasing an AI system that’s going to control nuclear reactors, or anything that has the potential for major harm, they are now forced to do their safety homework first and make a convincing case to the regulators that this is safe.
20:48 Brian Rose: But that doesn’t stop the covert development of that, either behind the scenes in a corporation, or in a government or an intelligence agency, or just a bunch of guys in a lab. They could make advances that maybe you couldn’t compare to developing a drug, because those things could also do harm in the future. So what do you do about that, or is that an issue?
Max Tegmark: Yeah. If this sounds depressing, let me see if I can inject some optimism here. Let’s take something like crystal meth or synthetic opioids like fentanyl. These are actually pretty easy to manufacture. In fact, there are so many crystal meth labs in people’s basements here in the US that there’s even a TV series about it, like Breaking Bad, right? And the […] that’s not going to happen at all anymore as soon as it’s by default illegal and you have to do the safety thing first. So I think if we can just start thinking of AI not as some sort of exception that gets exempt, but make it behave like any other industry, like biotech for example, that’s a huge win across the board. It’s going to take a while to crack down on very dangerous illicit bootleg stuff, just like with crystal meth or other things, but at the end of the day we’re in a good position.
23:10 Okay, you also mentioned another really great issue: what about governments doing things in secret? So let’s look at biological weapons. There was a huge arms race in biological weapons in the 1960s, mainly between the US and the Soviet Union. Then biologists got activated, kind of like AI researchers are now, warning about this and saying, hey, we don’t want there to be a weapon of mass destruction that’s so cheap that all our adversaries can afford it. Matthew Meselson here at Harvard convinced Richard Nixon of this, and there was the international bioweapons ban. It’s great. Now, after that it’s known that both the Soviet Union and the US cheated on the ban to various degrees. Did that make it a failure? Can you, off the top of your head, think of any bioweapon attacks that you’ve read about in the newspaper, anyone getting killed by bioweapons, in the last 50 years?
Brian Rose: No. I’m sure there’s a couple of small incidents, but nothing major.
Max Tegmark: Actually there were two examples of people killed by American and Russian bioweapons since then, but they were both own goals. The Russians had secretly been making anthrax in Sverdlovsk, and it leaked out and killed a bunch of Russians, maybe 10, maybe 30, something like that, but it didn’t kill any Americans. And then some Americans were killed by anthrax when this guy started mailing anthrax letters to a bunch of famous people. That was American-made anthrax he had somehow gotten hold of. I’d say, all in all, a pretty good scorecard, right? In 50 years the superpowers only managed to hurt themselves a little bit. There were no billions who died from this.
Why was it that they didn’t use it on a massive scale even though they were cheating? Because the stigma was so huge. People on all sides, including the Americans and the Soviets, had been saying bioweapons are disgusting, absolutely gross, and we’re not building them. It became viewed as disgusting, and the last thing either of the cheating superpowers wanted to do was to admit that they had them. So every time there was a little conflict somewhere, a proxy war, and they made this calculus of whether to start using bioweapons in Afghanistan or wherever to get a little edge, they always concluded no, it’s absolutely not worth revealing that we have them for this relatively minor thing.
Similarly, if anyone ever tried to start a private bioweapon corporation, I’m pretty sure they didn’t get very far with their investor pitches, because the stigma was so large. What good biology grad student is going to work on bioweapons when everybody thinks it’s gross? So if we can start getting this kind of stigma against uncontrollable AI systems that could wipe out humanity, I think we’ll be in exactly the same situation as with bioweapons.
26:50 Brian Rose: Yeah. I’m sure there’ll be some interest from various clandestine efforts and so on. But if I put myself in Putin’s shoes or Xi’s shoes, just as a hypothetical, and I’m sure they might think the same about us, they might say, okay, this doesn’t necessarily have the stigma. What if I could build an AI that would be the ultimate cyber attack, and all of a sudden I’d watch America or Ukraine or whoever my enemy was, and everything would be shut down from an infrastructure standpoint? There’s no real stigma because there are no bodies in the streets. You might argue that we’re a little ways away from the AI stigma, and since it’s developing so fast, maybe that stigma won’t get there. What are your thoughts on that argument?
Max Tegmark: My answer is: let’s look at what the actual risks are of ending humanity here, and that’s not the risk that someone does a mediocre cyber attack. You don’t even need AI for that in particular. You can do better cyber attacks with AI for sure, but that’s still fairly low-tech stuff.
27:59 The reason I am so motivated to make the call is because the most powerful AI systems we are building now are on the cusp of artificial general intelligence: they can outsmart humans at everything. This is something most people thought was only going to happen in 50 years or 30 years, and now Microsoft has a paper out talking about sparks of AGI, effectively saying that it’s almost here. Maybe we’ll get it this year, next year, in two years: machines that could in principle do almost all human jobs better than us. Some people think it’ll take ten years, some people still think it’s going to take much longer, but a lot of very serious people think we’re almost there.
And the issue with these things is that right now we hardly understand at all how they work. So if you ask why it was that GPT-J seemingly persuaded a Belgian man recently to commit suicide, or that another chatbot tried to convince a New York Times journalist to leave his wife, it was not because some engineer involved in building it was like, ha ha ha, I’m going to mess with these people. No, they just had no idea it was going to do these things. They’re gigantic black boxes. We had a big conference here at MIT, which I was the lead organiser of, just last weekend on exactly this: how can we understand how these actually work? There is a lot of momentum and a lot of progress in the field, but we’re still largely clueless on these things.
And if you’re Putin, if you’re Xi Jinping, or for that matter any powerful entity in the US […] that’s actually the last thing that a very powerful person would normally want to do, right? We should not think that the reason the Chinese or the Russians avoid doing this is to do America a favour. The reason they would avoid pressing start on this is because they like being in control. The Chinese government is just as interested in having control as any Western government, maybe more, you could even argue. That’s the reason why we currently don’t need to worry much about that, I think.
And you can even see today which country has cracked down the hardest on its tech companies so far. Number one is actually China, because they feel quite concerned, and I think rightly so, that one of the Chinese tech companies is going to deploy some sort of super-powerful AI system that’s going to do stuff they don’t understand. So it’s in the self-interest of the most powerful countries and their secret services not to do anything reckless. That’s why the biggest threat right now, I think, is instead coming from the private market, where everybody is just trying to make more money, and you can make a lot of money on things in the short term even if you don’t understand how they work and even if you can’t control them.
31:28 Brian Rose: You mentioned artificial general intelligence. That always used to be Kurzweil’s 2029, but now when I speak to people like Ben Goertzel and Mo Gawdat, they’re starting, like you said, to say it could be as close as two years away. And when that happens, you have something that can do everything better than a human, and if it can code and improve on itself, it starts to become something even more out of control than the black-box LLMs that we don’t understand now. So in your mind, if it’s unchecked, what would the next two to four years look like? I don’t want to go dark, but how does it progress and change and affect the world? I think that might be important for people to understand.
Max Tegmark: I’m very humble about how hard it is to make tech predictions. I was wrong seven years ago, thinking that it was going to take much longer than it actually has to get to where we are now. It actually turns out that it’s easier to get close to human-level intelligence than we had thought. So what I’m going to say now comes with a great deal of humility and huge uncertainties. But if things are completely unchecked, and we actually get to AGI next year or in two years, where they can do even AI development as well as humans, then according to the prediction site metaculus.org it’ll take on average about nine months or so until you get to superintelligence.
The idea there is that if, instead of having humans develop better AI, you can let the AI do that, then you can now have a billion virtual computer scientists who don’t need any salary except some electricity. You can greatly accelerate the process, and instead of doubling the capability on the human research and development time scale of years, it might double on the time scale of months or weeks or days. So you pretty quickly get from being limited by our ability to write software to being limited by the hardware we have. And then what […] it’s like a very faint old-fashioned light bulb; that’s how much better the software could be than what these AI systems are now.
And then in pretty short order you can also have the machines figure out how to make much better hardware, and pretty quickly you start getting to systems that are so smart that they’re limited just by the laws of physics. There are also limits there. You can’t send information through your AI faster than the speed of light. It’s hard to put in much more than one bit of information per atom. If you try to put too much information processing too fast in a small space, there’s so much energy there that it turns into a black hole. Professor Seth Lloyd at MIT worked all this out, and his answer is that you could do about a million million million million million times better than today, according to the laws of physics. That’s a one with 30 zeros, which is way farther beyond our human intelligence than we are beyond the cockroach.
So you might end up, in the fastest kind of scenario, in a matter of years, five years from now, maybe three years from now, or maybe ten years or a little longer if you think tech is really hard, where we are like the cockroaches among these god-like intelligent entities that have been created. This would be the culmination of a process that’s been going on ever since life started showing up on Earth over a billion years ago, where things got more complex and smarter, and gradually it’s gone faster and faster because we always use today’s technology to build tomorrow’s. With the Industrial Revolution things went faster, and now we’re getting machines to help us with more and more. We’re already having machines help us write code faster than before with these co-pilot systems.
36:29 So then the question is: is that good for us humans or is that bad for us? I think by default, if we don’t solve the safety problems first, it’s obviously going to be bad for us. We can’t compete with these things, so if we’re forced to compete we’re going to lose. There won’t be any humans. It’s not like the machines would have anything particular against us, but why are there no mammoths left? Why have we chopped down half the rainforest? It’s not because we hate trees. It’s just that we had other goals that weren’t aligned so much with the denizens of the rainforest, and it’s just really inconvenient to have to share the planet with much more intelligent beings.
People sometimes ask me how exactly much smarter machines would figure out how to kill off humanity. Well, if you asked a mammoth, or if you asked the black African rhinoceros why we humans drove it extinct, they couldn’t have understood how we would do it or why we would do it, right? They would probably be like, we’ve not done anything mean to the humans, why would they kill us? How could they possibly have guessed that some human thought that eating ground-up rhinoceros horn was going to improve their virility, even though the medical journals disagree? How could they have guessed that they were going to get killed by some fast-moving little piece of metal that came flying at them at high speed? That wasn’t even part of their world.
If we’re up against something more intelligent and it wants to get rid of us, it’ll be in a way that we can’t even imagine, and probably for a reason that we also can’t imagine. The basic point is that right now it’s still within our power to decide whether we even want to build these things or not. And what I’m saying is incredibly simple: let’s not build them until we’ve convinced ourselves that it’s going to be good for us. People sometimes say […] a fraction of suicidal people who want to take the rest of us with them to do this. I’m very hopeful that conversations like the one we’re having now can help people appreciate that, first, it’s not in our interest to rush ahead with this, and second, of course we can stop, if people are aware of what’s happening.
I’m not sitting here saying that we should never enjoy all the great upsides of this either. I’m just saying, if you have to wait one more year to make it safe so we can then enjoy it for billions of years, that’s a pretty good idea.
Brian Rose: Max, when you say it that way, if you just zoom out, one year is nothing, ten years is nothing, a hundred years is really nothing to slow down in the grand scheme of things. You’re talking about one or two years to AGI, nine months or more to superintelligence, and from there, there’s no history on this planet of a less intelligent being ever either surviving or having any type of reality that they used to have. So there’s a good argument to say this is really important to stay aware of. But as humans we’ve always been in the driver’s seat. I try to destroy my ego whenever I get the chance; it’s always good for perspective.
40:38 But to come to humans and say, by the way, there’s a new sheriff in town, you’re going to be second-class citizens from now on, your life is going to change, it’s called the Singularity and it’s coming up: nobody can conceive of this. You’ve talked about the movie Don’t Look Up, right? I don’t even think the human subconscious would allow us to see a reality that is so fearful and so ego-destroying. It will probably do everything to not see it. Do you think that’s what’s happening right now, to a certain extent?
Max Tegmark: That’s a very good point. There’s probably a lot of wishful thinking. We are very good at not seeing things that are painful to see.
Brian Rose: We’ve been here for a long time and it’s never happened before.
Max Tegmark: Yeah, it’s so easy to assume it’s always going to be this way. But we have to ask ourselves, and you’re completely correct, we have been the sheriff in town for a couple of hundred thousand years at least. Why is that? Is it because we have sharper claws and bigger biceps than all other animals? No, it’s because we’re the smartest ones, right? And we know that, and we’re kind of proud of that, even to the point that we use it for the branding of our species: we call ourselves Homo sapiens, the thinking man. So it should be quite obvious that if we build another species that can do the sapiens part better, we’re not necessarily in charge anymore.
42:21 It’s probably a good idea to talk a little bit about strategies for […] intellectually very advanced minds. I think if a bunch of five-year-olds tried to keep you and me locked in a building and make us invent stuff, we’d probably figure out a way eventually of sweet-talking them into letting us out, or tricking them somehow. So I don’t like that antagonistic strategy. A much better strategy is to align the goals of the machines with ours in the first place. When we have kids, we don’t just teach them how to use powerful tools. As good parents we also try to instil in them our values, what we think is good and what we think is right. That means that even when we end up more powerful than our parents, which is almost guaranteed to be the case when our parents are old and frail, we take care of them and we’re nice to them. So that’s a glimmer of hope.
The problem with machines is that the space of alien minds is just vastly larger than the space of biological minds, so it’s a much harder challenge to get them to even understand our goals in the first place. Then once you do that, you have to make them adopt our goals, and finally you have to make them retain their goals as they get smarter. If I think about my kids, when they were six months old I couldn’t explain all my ideas of morality to them. They just wouldn’t get it, because they couldn’t even talk yet, right? […] and that’s just very messed up for our species. I and many others have tried hard to help grow this field. There’s a lot of opportunity to do it dramatically, but that again takes a little bit of time.
46:25 You might also wonder how you could possibly ever trust a machine that’s way smarter than us about anything. Do you want to hear another little optimistic thought?
Brian Rose: Yes.
Max Tegmark: So you have a virus checker on your computer, right? If it can prove that some code you’ve downloaded from the internet is actually harmful, it won’t run it. What if we flip it around, so that future virus checkers, metaphorically speaking, look for a proof that the AI they’re going to run is going to do what you want it to do and not cause harm, and if they can’t prove it, they just refuse to run it? It shifts the burden of proof from proving that you’re guilty to proving that you’re innocent. And what if you outsource to the AI itself to come up with a proof that you should trust it?
There’s a fascinating fact about mathematics and computer science, which is that it’s much, much harder to come up with a proof than to verify that the proof is correct. You can write a simple program, just 100 lines long, a proof checker that checks this step is correct, this step is correct, done. You don’t have to understand much at all about the proof to check that it’s correct. It’s like if I have a very long chain, I just have to inspect each link to see that it’s not broken. So if a superintelligent AI writes some code, which isn’t the AI itself, that figures out how to make your new cancer drug or run your nuclear reactor or build your spaceship, and it also produces a proof that you can check with your little human-understandable proof checker that this is going to do exactly what you want, then you just run that.
That’s actually a fantastic situation to be in. Then you never have to run the big black-box AI that you don’t understand in uncontrolled circumstances, but you’ll have all this amazingly powerful AI it has created that you can actually prove you can trust. The kicker here is that no matter how smart you are as an AI, you can never do the impossible. So if there is actually a mathematical proof that this thing is not going to cause harm, then you actually can trust it, no matter how smart it is.
48:55 Brian Rose: There were some comments recently by Geoffrey Hinton, and one of the things he said is that now that ChatGPT, for example, has ingested everything that’s ever been written about humans, tricking humans is going to be very easy for it to do. Do you agree with something like that, or do you think we can still align it in some way, the way you spoke about? Or will we just be pawns compared to this knowledge of how it can affect us flawed biological beings?
49:36 Max Tegmark: Well, I agree with the things he said there, in the sense that if you had asked me five years ago for a little top list of things not to do, I would say number one: don’t connect it to the internet. Oops, we’ve done that, right? Number two: don’t give it access to humans so it can learn a whole bunch about how to manipulate humans. Oops, we built the chatbot, and we have massive amounts of data from social media on how to manipulate people into watching more ads and so on. Number three: don’t build a so-called API where computers can directly call these AIs and have them do whatever they want inside of loops.
Right now there’s something called AutoGPT, which is actually a full-fledged AI agent that keeps using GPT-4 as a sort of slave and keeps asking it for instructions. Someone else, probably as a dark joke, created something called ChaosGPT, whose explicit goal is to destroy humanity, and it keeps calling OpenAI’s API for instructions on how to do it. And they still haven’t, as far as I know, even blocked that. It’s just absurd. So we’ve already taken a few wrong turns here.
But in terms of the hope, if we get it the way I’m hoping, where AI becomes like biotech and there are safety standards you have to meet for any system that’s actually deployed and able to do harm, of course you’re not going to be allowed to put something like GPT-5, or some competing product from another company, in charge of major infrastructure, or let it touch the world and do things in the […] world, right? It’s like if you have a very, very intelligent human who you think secretly has sketchy ideas for ruining your life, but all they’re allowed to do is make inventions that you then give to other scientists to build, and they can prove that those things are safe. You’re not putting that sketchy guy in charge directly. Then you’re in a better position. But the first thing is that you have to restrict everybody’s access to that most powerful AI system, obviously.
Brian Rose: Yeah. And when GPT-4 came out, they said it’s only got information from the internet up to 2021, which I guess was to try to keep it somewhat siloed. But quickly, like you said, we saw plugins where it could access the internet, then AutoGPT where it can get real-time feedback from the internet, and now you’re telling me about ChaosGPT. Every week something new is coming up.
Max Tegmark: Again, of course, if you make the most powerful tool ever available to everybody in the world, everybody’s going to want to mess with it. I was just invited to a hackathon in India with a thousand people who are going to try to use AutoGPT to create recursive self-improvement, which is the sort of thing which could lead to superintelligence eventually. I’m thinking to myself, why are we doing this? And it’s obvious: a bunch of bright-eyed kids in India, it’s exciting, and they’ve probably thought a lot less about the risks than the CEOs of those companies. But we shouldn’t give people these self-destructive incentives.
There’s a reason why you’re not allowed to buy as much fentanyl as you want in any supermarket, because you’re giving the wrong incentives to people. And there’s a reason why you’re not allowed to buy hand grenades or nuclear weapons in the supermarket either. I think here we have, again, just a funny culture clash. AI has developed so quickly that many people in the field are stuck in the past, remembering that AI was useless, just a toy, so of course it should be free and unregulated. Why would you regulate philosophy, right? You wouldn’t. But it has very quickly transitioned to being a science like physics or biology, which can cause huge harms.
Ever since Hiroshima and Nagasaki, every physicist knows that you shouldn’t just open-source nuclear weapon design. Any biologist today who works in synthetic biology knows that you shouldn’t just let people use DNA printers to print smallpox or other harmful pathogens. You obviously need to restrict certain things so that you don’t give people the wrong incentives. And AI has to wake up and smell the coffee and realise that it has just as much social responsibility now as these other fields, because it is impacting society. It has already caused a lot of harm in society, and there’s going to be a lot more unless it gets with it.
55:35 Brian Rose: You said it has already caused a lot of harm. What do you mean by that?
Max Tegmark: Look at the world today. For example, I would argue that there is way more hatred in the Western world now than there was even five years ago, not just between countries but also within countries. So many Republicans and Democrats here in America can hardly talk to each other anymore at Thanksgiving parties. Why is that? It’s popular to just blame populist politicians, but we’ve had populist politicians for as long as we’ve had politicians. I’m telling you, what’s new is technology. This is very much caused by artificial intelligence.
We have these super-powerful algorithms used by social media companies that were given the very innocent-seeming goal of driving the profits of these companies by maximising engagement, in other words maximising how many ads people watched. They were trained on enormous amounts of data, and they quickly discovered things that even their creators hadn’t anticipated, namely that the best way of driving engagement is to make people really angry. And it doesn’t matter so much if what you make them angry with is true or not. What matters is how long they stay on and watch ads. This has driven these incredible filter bubbles that we have, and I feel it’s done quite serious harm already to our democracies.
We also have a very sad record in America in suicides: a huge rise in suicides a number of years in a row now. And I also think that things that aren’t counted as suicides, like the opioid crisis, are often a little bit suicides in disguise. Life expectancy is down four years in a row in the US. I think artificial intelligence also shares a bit of the blame here. A lot of social alienation has come from social media. There are some heart-wrenching psychology papers I’ve seen about teenage girls […] I gave you two examples of the bad things that AI has clearly already been involved in. That should serve as a wake-up call, I think.
Brian Rose: Yeah. Oftentimes people blame social media, but it really is AI on those platforms, because without it there would maybe be no social media effect. And you didn’t even get into the ads.
Max Tegmark: Yeah, exactly. And if there were no AI on it, the mix of content that we got would not have been in this race to the bottom. That’s totally driven by optimisation. Moloch always optimises things, right?
59:27 And there are some kinds of competition that are very healthy. If you ask what a free market helps with, sometimes competition is good: you get a better deal on bananas if there’s more than one seller, and so on. But we want to make sure that the kinds of competition we get are the healthy kind, not the kind where it’s an out-of-control arms race, or an out-of-control race between teenage girls about who can get the most likes or show the most skin in their selfies. Another very concrete example is these beauty filters. Liv Boeree made this excellent point recently in an interview that […] and Moloch wins, right?
Brian Rose: Yeah, that damn Moloch. It reminds me of some old Star Trek episodes.
1:00:56 What did we not talk about that you think is an extremely important thing for people to understand going forward? What surprised me the most is that everyone in this field is surprised at how fast this happened, so I was wondering how that happened. And my last question, you can choose which one to answer: is it pretty clear that OpenAI is way ahead of everyone and is going to be a big leader in this development? And based on that, are you pretty happy that Sam Altman is the CEO, as a guy who seems to be self-aware and understands this field? Or will he ultimately be subject to the forces of Moloch?
Max Tegmark: There are a few questions there. The reason I think many people underestimated how quickly it was going to go is because we made the same mistake thinking about intelligence as we made thinking about flying. If you had argued that we won’t be able to build a flying machine that can fly better than birds until after we figure out how birds fly, that would have been epically wrong. There’s that beautiful TED Talk video you can watch of a flying robotic bird, but it took 100 years longer to get that than it took the Wright brothers to build the first aeroplane. There was an easier way to make a flying machine.
In the same way, it turns out there are much easier ways of building human-level intelligence than what evolution came up with. People were mistaken, I think, in thinking we had to figure out the brain first, and obviously that’s really hard. But if you think about it, it’s not so shocking that there are easier ways, because evolution has its hands tied when it designs anything. It’s limited to building machines that can self-assemble, it can only use the most common elements from the periodic table that are abundant in nature, and they have to be able to repair themselves and stuff like that. When you’re an engineer you don’t care about any of that, so it was much easier. Then you asked whether I’m glad that we have the particular leaders we do for the companies that we do, and who’s ahead, particularly about Sam Altman here.
1:03:25 I greatly respect Sam Altman. I really believe that his concern is very sincere, and I really respect and am grateful for him speaking very candidly, more candidly than some other tech leaders actually, about how grave the risks are. But I am also critical of Sam for releasing GPT-4 and ChatGPT the way he did. It doesn’t mean I don’t respect him as a person, but I think it was a mistake, because I really think it fuelled the arms race. It fed Moloch. I think the world would have been better off if they had waited a bit with that, so that we could establish clear safety standards first. I hope I’m wrong. I hope his judgement was correct and this was a wise thing to do, but my current take is that it was a mistake.
I also disagree with him about the pause, because he says it’s not that we should have a pause right now. I think we do. But that said, I don’t think Sam Altman or Demis Hassabis or anybody else in this game has the power to pause alone anyway. So it really is the responsibility of all of us to put pressure on the industry so that we give them the right incentives, the incentives to bring out the best in them, not the worst in them.
1:04:55 Before we end, can I just say something optimistic also? Because we’ve spent a lot of time talking about doom, the ways in which things can go wrong and why they go wrong, and it’s very easy to get passive and resigned if we think we’re screwed, and just descend into decadence and not try to do anything. We’re not doomed. Right now humanity is still quite firmly in charge, and if we get this right there’s just an enormous upside. First of all, let’s not freak out about Moloch and just run for the hills. If some country invades you, the worst thing you can do is tell yourself that the foe is invincible and immediately surrender. That guarantees that you’re going to lose.
Moloch is not invincible. We’ve beaten Moloch many times before. The strategy is always collaboration. Moloch tries to divide us; we can build new collaboration mechanisms that unite us across countries, companies and so on. We’ve done it before and we can do it again. Take that, Moloch. And if we get it right, if we buy ourselves a little bit of time by dealing a body blow to Moloch, we can solve the technical AI safety problems. I’m really quite optimistic about that. We got into some aspects of how you can make machines understand our goals, learn them and retain them, and make things safe. I’m confident there are great ideas that nobody has even thought about yet, because so few people have even worked on it.
If you imagine scaling up the AI safety effort to be the size of, say, the cancer research effort we have, that’s a huge step up. I’m quite confident that we can solve this […] easy peasy, and we don’t even have to start on that for a few billion years. There’s actually nothing in the laws of physics stopping life from spreading into this amazing cosmos we have, where we’re on this tiny little rock squabbling about this, and we might self-destruct here when we have this incredible universe out there that is just yearning to come alive.
So I dream of a future billions of years from now when much of our universe is just teeming with life. I have no idea what they’re going to think about us, but I’m quite sure that if they know about us, they’re not going to think that we were insignificant, because it’s what we do now, in our lifetime, where we solved the AI alignment problem, that made everything possible for them in the first place. So yeah, let’s really fight this fight. It’s the most important fight ever: humanity versus Moloch. Let’s win it.
Brian Rose: I appreciate your thoughts, Professor, and I agree. It’s somehow been dropped right into our laps in 2023 as the biggest fight of the entire species. What a time to be alive, right? Look, I really appreciate the time. I think you bring a really valuable, nuanced and unique point of view to this entire conversation happening with AI, and I think yours is an integral part, so thank you.
I’d like to definitely stay in touch and continue having these conversations, maybe even throughout the rest of the year, because time is of the essence. One thing I’ve learned about communication, Professor, is that it’s a repetition game. I learned that from the politicians. So I appreciate you tirelessly having these conversations. Let’s continue having more of these in the future, and again, thank you for everything you’re doing.
Max Tegmark: Thank you so much for what you’re doing. My pleasure.
Brian Rose: All right, take care.
