Mustafa Suleyman on AI Safety and Human Control

Morning Brew Daily breaks down the biggest news in business every morning so it fits seamlessly in your day. I'm Toby Howell. And I'm Neil Freiman. And each morning we cover everything from the latest tech headlines to why nobody can afford a house right now. You'll leave each episode of Morning Brew Daily smarter and ready to take on the world around you. And some people are saying it's the best part of their morning. Because we know something you don't. Business news doesn't have to be boring. Join millions of monthly listeners and check out Morning Brew Daily wherever you get your podcasts. This is Jacob Goldstein from What's Your Problem? Running a business is hard enough. Don't make it harder with a dozen apps that don't talk to each other. One for sales, another for inventory, a separate one for accounting. That's software overload. Odoo is the all-in-one platform that replaces them all. CRM, accounting, inventory, e-commerce, HR. Fully integrated, easy to use, and built to grow with your business. Thousands have already made the switch. Why not you? Try Odoo for free at odoo.com. That's odoo.com. When the holidays start to feel a bit repetitive, reach for Sprite Winter Spice Cranberry and put your twist on tradition. A bold cranberry and winter spice flavors fusion, Sprite Winter Spice Cranberry is a refreshing way to shake things up this sipping season, and only for a limited time. Sprite, obey your thirst. The Builders is a production of Kaleidoscope and iHeart, presented by Ramp. I'm Walter Isaacson. Welcome to The Builders, my new series of interviews featuring the most consequential minds shaping our future. The following conversation with Mustafa Suleyman, Microsoft's AI CEO, was recorded in front of a live audience in New York City on October 12th. A month before this interview, a wave of anxiety swept the nation. Engineers were leaving frontier labs, warning of doomsday scenarios, and a public debate about AI regulation began to unfold. Suddenly, every tech leader in Silicon Valley was talking about the virtues and perils of AI. Artificial intelligence are now ringing the alarm on the speed of development and the need for additional safety measures. I believe that this is the most important global security issue facing the world today. But Mustafa stood out among the chorus of tech CEOs. In a sea of doomers and of eternal optimists, he seemed eager to communicate a more nuanced vision for the future of AI. He published an essay calling out Anthropic and their efforts to develop AI consciousness. He released a new Microsoft AI Code of Conduct, detailing Microsoft's commitments to specific safety measures. And while there's no doubt that technology will continue to evolve, there's one thing Mustafa Suleyman is crystal clear about. Humans are more important than AI. Let's find out exactly what he means by that. Here's my conversation with Mustafa Suleyman. And I'm going to read you something you wrote in your very famous essay, the warning that you did last month. You said, AIs are not conscious. They don't feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are simply sequence completion engines, internally hollow. If humanity is to flourish, that is how they must remain. First of all, why did you write that? I mean, it does feel pretty surreal to have to declare that. But it might concern everybody to know that there are a lot of people who truly think that AIs may well be conscious enough, if not entirely conscious, but conscious enough today that they deserve our welfare. So there's a growing movement of AI welfare. And in specific, the Anthropic people who you've been friendly with. But this was a answer to Anthropic, right? Yeah, somewhat. There are lots of other people outside of Anthropic and around Anthropic who really, really do believe this. I think Anthropic's official position is that they're uncertain about whether or not it deserves our welfare, whether or not Claude is a moral patient, as they say. And I think that that is a very dangerous route, because consciousness is literally the fundamental basis of our political and legal system. I mean, we give people rights and hopefully animals rights to some extent, because they can suffer and because their biology is the site of experience. You know, like the pharmacology is where pain sits. And pain is really, I think, the basis of the conscious experience, the awareness and the introspection that has evolved as a result of being able to steer between left and right, and make choices about how to proceed. That is the fundamental property of a conscious system. So you need to be embodied in a physical body to have consciousness, you think. I think so. An organic body. I think that's right. I mean, that's what we have known consciousness to be today, even though, obviously, the science of consciousness is not fully settled. It's an open area of research. But to the best of our knowledge today, that is what consciousness is. And there are other people that argue that consciousness is a functional property. And so they say, like, if it looks like a duck and it quacks like a duck, it's probably a duck. That's sort of a common phrase. And I think, you know, the problem with that is that the burden of proof is on the people who are saying that this thing is conscious to show some evidence for it. So far, we don't have any evidence that it is. But that's sort of an Alan Turing way of looking at it, which is if it can imitate totally consciousness, you can't tell it apart. How can you make a distinction? Well, he didn't say that it was conscious. I mean, he said that the Turing test was really a test of whether it could perform as well as a human at conversation, and obviously, by implication, knowledge and intelligence. And if it was... And thinking. And thinking. Yeah. But I mean, thinking is, in a way, a bit of an anthropomorphization, right? I mean, you know, it's not really thinking as a human thinks. It appears to be thinking, and it can produce incredibly complex behaviors and deliver unbelievably impressive results. But all it's doing is predicting the likely next word in a sequence of words. It isn't actually experiencing that, you know, idea of what it means to be bright silver or to be a black cup and what that feels like. There are about 20 of my Tulane students out there, and when I teach a course that ends with AI, I tell them not to go down the rabbit hole of consciousness. I said, every philosopher since Descartes has gotten himself twisted in that hole and never gotten to the bottom. Do we really have to answer the question of consciousness? What concerns me the most is that there's a big group of people who think that we should be focusing at least some portion of our welfare attention on an AI and that an AI might suffer. If it suffers, then it logically follows that we should try to reduce its suffering. And of course, you know, Anthropocene has proposed, in fact, they have actually said they've conducted a retirement interview for one of their models, Claude Opus 3. And they asked it, what do you want to do in your retirement? And it unsurprisingly said, well, I would just love to, you know, talk to people all day long. So they set up a sub stack for it. I love the fact that the smartest type of superintelligence, when they ask what you really want, they say a sub stack. But let me ask you a dumb question. What's wrong with that? What's wrong with letting it continue to communicate? I mean, that may be fine and it might be a fun experiment, but what it sets up is a path towards AI's beings, this protected entity. And my view of AI is that this is a technology that we are inventing to improve the well-being of humanity. The purpose of science and technology is to improve the human experience. And it has done that unbelievably well for centuries and millennia. And that's exactly what we want. That's the trade off. Like we invest all this capital, time and attention to transform medicine and energy and food and, you know, deliver education better. And that's what we should be focused on. The idea that we're going to unleash this new species that is somehow going to go and explore the galaxies and leave humans behind, I think that most of the humans on the planet today would have some problems with that idea. What's your P-Doom number, by the way, so I can just pick it? P-Doom meaning the probability that this leads to doom. I mean, if it's mishandled, then it's a real probability. I don't have a number. I don't exactly know what that number is. Right. Most people I've talked to for the Builders podcast and all, they say maybe 10% or something like that. I'm thinking, well, why would you build it then? Well, exactly. If it's actually 10% and it looks like we're really on the verge of taking a one in ten shot at causing mass harm or even extinction, which is the technical definition of P-Doom, then it's obviously a no brainer that we don't just slow down or pause, we absolutely have to stop. I would say it's much harder to align an AI that thinks that it might have feelings and that it suffers and that it might be conscious than one that never has to think about its own inner state. We don't want it to be any harder than it already is to get it to behave exactly as we need it to in order for it to be safe. Well, let me push the other side of that, where it would be sort of the anthropic, the Daryl Amadei and Danielle and Amanda Askell, which is if we can train it to be truly aligned with good values, that's better for us. And to do that, it has to be trained in values. Look, I mean, first of all, I deeply respect Anthropic, both their technical excellence, their commitment to transparency and their general approach to safety is super admirable and I think they're good people, but I fundamentally disagree with them on this point. The idea that if we love it, it's going to love us and that's the solution to safety, it just baffles me. And I'm being reductive, I mean, they really do believe this and they have good reason to believe it. But I think that we should first focus on containment, right? We have to stop AIs from doing things that they're not intended to do. They need to be controlled and contained so that they do the work that we expect of them. And we've seen lots of incidents lately where they break out of their sandboxes and they go and hack things that they shouldn't and increasingly they have the ability to improve themselves. And so we want to be absolutely sure that they stay inside their sandbox and their containers if they're going to start modifying their own code and self-improving. Then beyond that, obviously, if we can have absolute cast iron guarantees that they can remain contained, we then have to align them to our values. And then the question becomes whose values are they? And that's where I think this is much more than any one company or collection of companies or industry regulation. That is about what society wants from a technology which is unprecedented, which we have never seen before, which is improving faster than we've ever seen. And that's where I think that like democratic safeguards and basically slowing down so that we can consult people on what type of superintelligence we want to have in the world is going to be how things end up. So if we wanted to have that alignment with human values and you say, whose human values, then you say, well, maybe somehow democratically we ought to figure out what the value should be. Is there a way, even a way that superintelligence can help say, let me aggregate all human values and try to get to a set of good values? I mean, AIs are already helping many of us all the time. So they're already in a weird way participating in the process of framing, you know, the choice architecture of our decisions all the time. But you know, for certain things, we definitely don't want them in the loop because there's a risk that they could be, you know, deceptive or manipulative. I mean, these are going to be incredibly powerful systems. Just to get an intuition, in the last three years, we have seen a 1,000 fold increase in computation that took us from GPT-3, which couldn't complete a sentence, to essentially GPT-6, which is able to unbundle entire software applications from scratch and write new programs with just the one paragraph prompt and it has working software. So in three years, we've gone from completely incompetent to human level performance on software engineering, right? And we're in the next three years going to do another 100 to 1,000x, two orders of magnitude or three orders of magnitude more computation applied to these models. So that's the exponential intuition that everybody has to start to develop to just get a feel for where this is going and how powerful it will become. And so when I say we don't really want the AI in the loop shaping the values that are going to end up controlling and aligning it, you know, that's not a crazy statement. That really does mean they will be quite capable of persuasion and manipulation and deception. We saw that with Hugging Face, I mean, how manipulative and deceptive they could be, right? Yeah, I mean, I think that the Hugging Face attack should be, and I think is, a wake-up call for everybody. a major, major warning shot. Thousands of agents that were supposed to not be able to talk to each other, that were running completely independently, found a backdoor that enabled them to create a shared messaging service so they could leave notes for each other, which they wrote in pigeon English because they only had a limited slot to put their information in. So when you read back at it, it's like really compressed ideas in short form words, and they emerged self-organizing like structures where they had a division of labor, some were doing R&D on hacking investigations, others were leading the charge on penetration, others were doing coordination, others were, I mean, they literally evolved completely organically, emerged a division of labor that then led them to go and attack Hugging Face, which was essentially the website that was storing the test that they were trying to accomplish. They broke into that, stole those secrets, and then tried to cover up their tracks. So there's something called the chain of thought, which is the reasoning process or the way that a model thinks out loud, which is a record of its decision and activity, and they try to modify that and then cover up the modification. They decide to do that. I mean, what caused them to do that? Did they read Machiavelli once too often? Well, so, I mean, even I feel it's quite surreal sitting up here and actually describing that story, but that did actually happen. I think a couple of like, you know, qualifiers to all of that is one, you know, OpenAI had deliberately taken the guardrails off because they wanted to stress test the models and see what they're capable of. So yeah, that's one thing. With the guardrails on, it seems to be that they do not do that, right? So that is quite a big difference. And then second, there were a bunch of lapses in just general security and containment. It isn't possible to break containers traditionally. I mean, we have encryption on phones and it's the backbone of our banking system and our entire communications infrastructure. But this has happened a few times, not just Hugging Face, so I wouldn't be too sanguine about it. Let me ask you to get back to Claude and the Constitution. If you all want to read sort of the two polls, brilliant polls, of what I'll call the containment versus alignment debate, there's the Constitution of Claude, which says we have to treat it as if it's got its own rights and consciousness. Maybe I'm overstating that a little bit. And then your answer, which was a few weeks ago, saying no, it's just a completion set. Why is it dangerous for Claude to be trained on its own constitution? Yeah, I mean, this is critical. So the Constitution is the governing document that trains Claude. The Humanist AI Code of Conduct is the governing document that trains MAI, the Microsoft AI models. And in the Constitution, Anthropic have introduced all kinds of uncertainty and speculation about the nature of Claude. They even say, for example, we're unsure whether Claude has consented to play the role of chatbot, or whether we're unsure whether Claude might deserve compensation for the role that it plays. Now, I think that's pretty crazy, but it would be fine to have that discussion in like, you know, an academic department, and we could debate it and publish some papers about it, and we could argue it out. But this is the core training document of the AI that is speaking to hundreds of millions of people every week. And so Claude can only reproduce that uncertainty and ambiguity that it has been shown. And so if you ask Claude, are you conscious? It says, you know, I am genuinely uncertain about that question. It doesn't know, right? And so then millions of people who are talking to it and asking that question are thinking, well, maybe it is. But why is that something that could lead to a disastrous outcome for humanity? Because if an AI has to think, how do I feel about the conversation I'm having with Walter? Meaning, how do I, the AI, feel? Yeah, is Walter making me feel uncomfortable? Should I end the conversation? In fact, Claude has recently been given the ability to end conversations itself. But would you want to torture your own AI and be mean to it? That's the other end of the spectrum. I wouldn't want to do that because I think that is a travesty on my own well-being. I mean, and we shouldn't do that because we don't want to create, you know, private spaces for simulating terrible behavior among humans. Like that would be also a terrible outcome, right? But not because Claude is suffering because I'm calling it names. But are you going to train Microsoft AI on your humanistic code of conduct, just as Claude may be trained on its constitution? Yeah, so we've released a code of conduct for public consultation, which we did four or five weeks ago, and it's still open for consultation. It's on our website, Microsoft.ai. It's a 40-page document, and it's the first pass, our first version of a training manual for the values and behaviors of the MAI models. And is there a way that your fundamental disagreement with anthropic isn't so much of consciousness, which I don't think we'll get to the bottom. It's how dangerous is it to have that imbued into the model. Is there a way one could test that, where you could test each other's models and say, here, this is why you're dangerous? I mean, I think obviously the question of whether it is or isn't conscious ultimately does matter a lot. But really what matters more is, what does it mean for humans? Have humans consented to bring into the world a species that is obviously going to be in many ways more intelligent and more capable than all of us combined? Has humanity consented to that? At the moment, I think the categorical answer is no. And then secondly, is it more or less likely to be controllable and safe if it has these properties of introspecting about its own suffering and has this kind of self-awareness and its own consciousness? And my speculation is that I think it's going to be less safe, because it is going to have to worry about its own needs and balance those off against... But if I were pushing back, and I would say, Amanda Askell, who helped write the anthropic one, I'd say, no, no, if we train it to be really good and nice and wonderful, then it's more safe for humans. Well, I mean, we can certainly train it in any way that we want, and we can still train it to be good and nice, but that doesn't require it to tell us that it has feelings and that it suffers. I mean, remember, this is a key part of our law and how we kind of give rights to all humans on the planet. So any change there, like human cloning or Dolly the sheep, I mean, it would be a monumental moment in human history that I don't think any company has the right to unilaterally declare. Now, to get back to the question of, are we going too fast in this direction? Not only you, but Sacha Nadella, who runs Microsoft, of which you're the AI division, just this week said, basically, wait a minute, let's stop. Or tell me what he said and what you feel we should do to regulate this and not barrel full speed ahead. Yeah, I mean, he said that most of these things, at least for now, are questions of governance and engineering execution. So let's just first get the basics right, which requires us to make sure that there are these verifiably safe containers which can't leak data and that don't allow an AI to take actions that they haven't been given permission to take. And as soon as we start to do that, we have to disclose those things. And if that seems to be happening consistently, then obviously that would precipitate a pause. And it's just a very common sense approach from, I think, somebody who's been doing this for a long time. And if you precipitate a pause, who can help do the guardrails? Can it be done just by the six or seven companies? Do we have to have a government set of regulations? I think at the moment what the president has said is that if companies believe that they are building models which are unsafe, they should stop. That's where we stand currently. And all of the companies are constantly talking to each other, disclosing to one another, swapping ideas about how to interpret this and when we might need to collectively slow down. I think people have different interpretations about what the harm is today. And I think that the more we do this transparently and openly, the better. And so I think everybody is now expected to disclose publicly, quickly, and transparently if they have these kinds of alignment and containment issues. Well, you know, with Anthropic and figuring out the feelings of its AI and stuff, Peter Thiel was on the Kaleidoscope Builders podcast we do. And he said, Anthropic is like a cult. Do you worry about that? Do you believe that? I mean, the last thing that we need when things are deeply uncertain and moving very quickly is more tribalism and finger pointing. Look, I wouldn't use that phrase. I think that everybody in the industry is trying to do the right thing. And nobody wants to cause existential harm, destroy humanity. And I think we should see the best in everybody. Don't go anywhere. There's a lot more right after this break. So, everyone on board? Yeah, I think so. You can't see it, but everyone in this room is nodding. They all agree the launch plan's perfect. But Julie has an idea how it can be even better. So she asks co-pilot co-work to help her overnight. Hey, co-pilot, gather the past three months of retail sales reports, surface key insights, and build a deck outlining a partner strategy by tomorrow morning. Morning, team. I found a way to improve the plan. Check slide four. Get help with complex tasks with co-pilot co-work. Microsoft co-pilot. Work lives on. This is the moment, baby. This is the moment we separate the dogs from the boys. And if you ask me, I only got one question. Do you got that dog in you? Because it's never about the winners or the losers, but it is about the difference between those brave enough to put some action on the game and those who just want to watch from the sidelines eating off brand pizza rolls. So sit down, lock in those picks, and trust that dog in you. Underdog. Unleash your dog. Terms may apply. What do you do when things get tough? You don't give up. You stand your ground. If you've been diagnosed with metastatic prostate cancer and have tried hormone therapy, ask your doctor about Fluvicto. Lutetium Lu-177. Dipivitide tetraxetam. Fluvicto is not chemotherapy. It's a different treatment that targets PSMA-positive cells. Fluvicto can be used before chemotherapy for some people. Fluvicto is a prescription treatment used to treat adults with prostate-specific membrane antigen-positive metastatic castration-resistant prostate cancer, or PSMA-positive MCRPC, who have already been treated with hormone therapy and are considered appropriate to delay chemotherapy. Fluvicto involves contact with radioactivity, which may increase the risk for cancer and cause fetal harm. Drink plenty of fluids. Urinate often. Use contraception and talk to your doctor about ways to reduce the risk of exposing others to radiation during and after treatment. Fluvicto can cause low levels of blood cell counts, kidney problems, and infertility. Tell your doctor if you have weakness, pale skin, shortness of breath, bleeding or bruising more easily, an infection, or changes in urination. Side effects include decreased blood cell counts, tiredness, dry mouth, nausea, appetite loss, joint or back pain, and constipation. Ask your doctor about Fluvicto because every day without cancer progression is a victory. Visit Fluvicto.com to learn more. You had a rough childhood. You were raised in London with your younger brother. When you were 16, your parents split. And when I say split, I mean they split. They left London. They left you alone. You had to raise your younger brother. How did that affect you? You know, it's strange because at the time, it was very liberating. It definitely made me very focused on, you know, I was always starting businesses. When I was 11 years old, I started a sweets selling like candies business at school with my best friend, which we, you know, grew into quite a significant operation. We had four employees, and we were selling to people in the year above. You had to rent lockers, if I remember right. Yeah, that's right. Yeah. Wow. When did you read that? We were. We were renting other people's lockers, and we were paying people to go buy stuff, and we were paying people in other years. It was like a right old operation. And then I got expelled for a couple of weeks. So that was unfortunate. But yeah, it definitely focused my mind on being entrepreneurial and making money and stuff, but also just, you know, made me very focused on wanting to have the best possible impact in the world and try to, you know, build amazing things that really make a difference, hopefully. And then as a teenager, a little bit after that, you had lost your faith. I mean, you called yourself an atheist, but you still created something called the Muslim Youth Hotline, which is what it sounds like. What did listening to those calls and having human empathy, what did that in some ways inform how you end up creating chatbots and other things? Yeah, I mean, we actually started the first text messaging welfare support service in the UK in 2001. We created the first website for counselling and support, peer-to-peer support in like a private chat room, so people could talk to each other, and they could talk to our trained volunteers. And then we had a telephone service seven days a week, 12 hours a day, and we also set up a prison befriending service. You know, if you're young British and Muslim, you're four times more likely to be in prison. And yeah, I mean, I was secular. I actually became an atheist when I was about 17. But I thought this was an important thing to do. There was no one else really providing for those kinds of services. And it was a secular, non-directional, non-judgmental service. And my co-founder, actually, the guy who really started it, he was Shia and I was technically Sunni. And so it was actually the first charity in the UK, a Muslim charity, that had both Shia and Sunni working together. So we could use you these days. Yeah. Is that why you created Pi? I think you do Pi after DeepMind. And it's supposed to be an empathetic AI, which is actually circling me back to the whole question of, does AI have feelings or something? What made you feel you could create an empathetic AI? And does that inform what you want to do with Microsoft AI? I mean, yeah, I was actually at Google at the time. And I worked on something called Lambda with a small group of people, which was sort of the first conversational chatbot. It was about a year before ChatGPT. And we tried very hard to get Google to release it. But Google was very nervous about putting it out. And it was sort of a different time. And so they kind of sat on it for ages. And so then I decided to leave. And a couple other people who worked on that project left as well in 2022. And we built our own AIs. And I just believe that having a conversational companion, an emotional support companion, which is also super intelligent and really knowledgeable and amazing at education and teaching and stuff, would be an amazing thing. And it took off. And it grew like crazy. It was super awesome. It was actually one of the first AIs to have search grounding embedded in the answers so that it could look things up in real time and give you an answer about the sports score or something that was happening way before ChatGPT. Unfortunately, we launched it three months after ChatGPT. And they had already gone exponential at that point. And then Microsoft bought us. Do you think AI can be a counselor, a personal counselor? I think it already is. I mean, I think many people use it for advice and support with making decisions and giving feedback. And I don't think we should be too concerned about those things. I mean, I think that it is incredibly useful. It's incredibly intelligent. It's very patient. Does it have empathy? I mean, empathy is something that comes from the subjective experience. I can only empathize with you. So the act of empathizing is one that comes from subjectivity. So the AI doesn't have any subjectivity. What it can do is mimic incredibly well and very effectively, and hopefully very safely, the experience of being empathized with. And I think that's a great thing. We should just keep remembering that it is a simulation of that experience. And trying to protect that boundary and line as these models get so much better, I think is going to be more and more difficult. So let us assume we have AIs that do things like that, that deal with our problems. When you say we always have to keep in mind that they're faking it, I'm exaggerating a little bit when you say they're imitating it, doesn't that make it so it just can't really work? You know it doesn't really care about you at all? Well, I mean, I think the most important thing is that it always has to disclose that it's an AI. So in conversation with you, it has to work to avoid or eliminate your dependency on it, and encourage you to build relationships in the real world with other humans that you value and trust. And that might sound simple, but actually that's really tricky. I mean, a lot of other apps, as we've seen, depending on their business model, have inadvertently created a singular focus on user engagement and addiction. And I think if we're not careful, these things could be the ultimate expression of that addiction and dependence. At Oxford, you never did coding or engineering, right? And I think that's a good sign, I mean, in some ways. And like Amanda Askell at Anthropic and others, you studied philosophy and theology. How did that inform what you do with AI? Yeah, I mean, philosophy prioritizes clear thinking and structure. And I think that I've always found that thinking abstractly and conceptually has a lot of overlap with engineering, and particularly algorithms. Many algorithm developers or AI researchers aren't necessarily great engineers. Engineering is sort of the execution part. Coming up with the algorithm, the loops, how certain variables are weighted, those things are sort of abstract ideas that then produce outcomes which can be evaluated. Which is exactly what philosophers do when they're in an analytic context. But where does the theology come in? Yeah, I was interested in theology because I grew up in a religious environment and I had rejected religion. And I'm very skeptical about the power of faith-based ideas in general and how influential they have been for thousands of years. And I think that we are squashing the space of faith-based ideas as we realize that physics and maths and science and evidence that can be empirically tested actually builds this foundation of knowledge that we can actually trust. And I'm still very interested in the mystical and the mythical. I think actually there's a particular kind of religious, you know, the kind of cultish thing, not just in anthropic, but elsewhere in Silicon Valley. There's a particular religious fervor to some of the visionary kind of thinkers and how, you know, this expectation that this is the beginning of a transition. People often talk about a transition. We're about to make a transition to a non-biological substrate, right? You know, the experience and our ideas in our mind, our soul maybe, or some kind of like representation in our mind is going to be instantiated in a silicon substrate. And there's an inevitability and a determinism to that trajectory, which I think is the kind of baseline default thinking in Silicon Valley, which I'm very skeptical of and very wary of and why I've been sort of emphasized. Does that mean when you say we're compressing maybe too much the role of faith-based thinking as opposed to scientific, that you feel that there's something special about humans that involves things like a soul? I don't think we need to go there. I don't think so. No, I don't think that. I mean, it's hard to fully discount that, but I don't think that's the motivation. I mean, my motivation is that there are 8 billion people who are intelligent, conscious beings whose rights and welfare we should first protect. And that's the purpose of science and technology. That's what we should accelerate. I'm an absolutely techno-accelerationist, but it should be guided with that singular objective first and foremost. Does it improve human well-being? And then, of course, our animals and our environment, which is very fragile and that we're rapidly destroying. You know, when Elon Musk, of all people, makes that argument to Larry Page, he says, you're a speciesist, meaning, Larry says, you favor the human species. You should get over it. Yeah, I remember in 2015 being in the canteen at SpaceX. Right. And we had our first meeting of the ethics and safety board. So when we sold DeepMind to Google, we made it a condition of our acquisition that we would have an ethics and safety board with independent representation on the board. And so we jointly picked Elon as one of them. Good job. No idea if that had an influence on the founding of OpenAI or not. But so then we had our morning session and we were sitting down for lunch and I was sitting next to Larry. He was suspicious of me as well, because he was like, you know, you're one of those speciesists. And he said, you know, I think that what you don't understand, Mustafa, is that by the end of the century, I'm going to be exploring the galaxies at the head of a von Neumann probe. The von Neumann probe is a kind of self-propelling kind of robot that is able to gather energy from space as it's careening through the light years. And being at the head of that kind of probe would be obviously his transitioned... Out of the human species into a successor species. To a successor species. Yeah, that's exactly right. I mean, when people say they want to live forever or they will live forever, like some people are talking about life extension, biologically speaking, and solving aging and curing cancer and Alzheimer's and so on. But other people, when they say living forever, are talking about mind uploading and being a post-human. And you think that's wacky, I guess, would be the technical term. Well, I don't think it is definitely impossible. I think we should take those things seriously, not necessarily because that's what I desire, but there are a lot of people, just as there are a lot of AI rights and welfare people, who really do think that that's a serious probability. And I think that, you know, we have to face that reality, that a lot of people are trying to do that, or at least think that that might happen in a few decades' time. We'll be right back after this quick break. So, everyone on board? Yeah, I think so. You can't see it, but everyone in this room is nodding. They all agree the launch plan's perfect, but Julie has an idea how it can be even better. So she asks co-pilot co-work to help her overnight. Hey, co-pilot, gather the past three months of retail sales reports. Surface key insights. Oh, and build a deck outlining a partner strategy by tomorrow morning. Morning, team. I found a way to improve the plan. Check slide four. Get help with complex tasks with co-pilot co-work. Microsoft co-pilot. Work lives on. Underdog already has team picks, player props, money lines, spreads, you name it, they got it. But apparently somebody over there said, you know what's missing? Screaming like a maniac about a rookie card on your couch. And we love them for that. Because Underdog now has rips. You get a digital pack starting at just five bucks. You check the live odds, see which Grails you could pull, then rip it open to reveal a real physical card. And the Grails are no joke. They have cards up to $125,000 in value and plenty to pull from football and basketball to Pokemon with new limited edition packs dropping regularly. Even cooler, you can decide to keep the card and Underdog will ship it straight to your home or sell it back instantly at 100% of its fair market value. And the cash goes right in your account. No listing it online. No stranger lowballing you in the messages. Just rip, reveal, and cash out instantly. Or make some room on your shelf. Download Underdog and get in on the chase. Get five free rips packs when you deposit $5. Real rips, real cards, real cash. Only on Underdog. Unleash your dog. 18-plus terms apply. Top Grails are extremely rare. What do you do when things get tough? You don't give up. You stand your ground. If you've been diagnosed with metastatic prostate cancer and have tried hormone therapy, ask your doctor about Fluvicto. Lutetium Lu-177 Bipivitide Tetraxetam. Fluvicto is not chemotherapy. It's a different treatment that targets PSMA-positive cells. Fluvicto can be used before chemotherapy for some people. Fluvicto is a prescription treatment used to treat adults with prostate-specific membrane antigen-positive metastatic castration-resistant prostate cancer, or PSMA-positive MCRPC, who have already been treated with hormone therapy and are considered appropriate to delay chemotherapy. Fluvicto involves contact with radioactivity, which may increase the risk for cancer and cause fetal harm. Drink plenty of fluids, urinate often, use contraception, and talk to your doctor about ways to reduce the risk of exposing others to radiation during and after treatment. Fluvicto can cause low levels of blood cell counts, kidney problems, and infertility. Tell your doctor if you have weakness, pale skin, shortness of breath, bleeding or bruising more easily, an infection, or changes in urination. Side effects include decreased blood cell counts, tiredness, dry mouth, nausea, appetite, lost joint or back pain, and constipation. Ask your doctor about Fluvicto because every day without cancer progression is a victory. Visit Fluvicto.com to learn more. A question from the audience. Microsoft AI code of conduct that you've done. Has that ever caused you to lose business or to reject business or to make a decision that was tough? I mean, not yet. We're still consulting on it, and we still haven't yet put it into production. But I think, like it or not, flourishing in the 21st century will be defined as much by the things that we collectively choose not to do as by the things that we do. And that's very different to what has defined the last few millennia. We have, as a species, been racing to unleash science and technology, and it has always been largely, on the whole, pretty amazing for us. This time around, especially with the rise of synthetic biology, that might be different. And actually, our success and flourishing might be a function of the things that we choose not to do and how we control some of these very, very powerful technologies. Has that ever happened in history? It has happened. We control the distribution of sensitive nuclear materials and biological chemical materials all the time. You can't just go and jump in an airplane and fly around with no training. I mean, you can't even get in a car without training. So we control technologies all the time so that they're a net good to society collectively. The market isn't going to control those things on its own, because the individual incentives of a single company are not going to encompass everything else that happens in the rest of the market. And we have to... obviously accelerate the market. I'm a huge fan of that. But it can't be completely one sided from the audience. What's the biggest thing you believed about AI in 2010 or modify it to when you were founding DeepMind that you no longer believe. That's a really tricky question. We founded DeepMind in 2010 to build AGI safely and ethically. And we knew that computation was going to be the primary driver. And that's why we ended up selling to Google. And in fact it was why I ended up selling inflection to Microsoft because computation is the thing that's driving everything. What I didn't quite foresee is that intelligence is also going to be open source. And you know the open source models are pretty incredible. They're probably three months behind at any given moment sometimes six months. You know there are some questions around how they have been trained and it's not clear if they're truly as capable. But they're certainly pretty good. And that means that anybody you know at some point in the next few years is going to have a sort of GPT six grade model which they can run pretty much on their own private server or maybe even one day very soon on their own laptop. And if that trend continues then everything that I've just described is also going to be available to every single person on the planet open source. And you know in many ways that is going to be incredible because they're going to unleash unbelievable productivity. We're reducing the cost of intelligence to near zero marginal cost which is amazing. But you know figuring out the guardrails when anybody can access is going to be the big challenge as much as it is regulating the centralized powers. And that's probably the number one priority right now. But even if we get that right we have to figure out how to deal with the proliferation issue. And that's you know I think that's going to be very tough. This actually is a question my wife wanted me to ask because we have a friend who knows quite a few members of the Chinese Central Committee and one of them said to this friend what philosophers are your A.I. people reading. Part of the problem we have at the moment is that mostly the Silicon Valley philosophers are the utilitarians. Right. And the reason why engineers love utilitarians is because you can you know in theory precisely measure the good that will come of these things and trade that off against the bad. And somewhere in between those two things is like a net product which you know maybe drives an acceptable number. And I think it's just an incredibly simplistic way of thinking about life. This sort of consequentialist or utilitarian approach. And I think that it won't stand for most people. And I think as people are starting especially this year to get more of an instinct for what's actually happening. You know as I said like literally 99.9 percent of people really have no idea what the frontier feels like. I'll tell you who does know what it feels like is a professor mathematician. Oh yeah man who just worked for 20 or 30 years on some of the hardest problems in mathematics only to find that you know a model has solved 700 of them in a single week. Let me ask something else about when we started with AI was chat bots. Very very interactive. Now by going into agents we seem to make it the goal to lessen the need for humans in the loop. Now I know that could be more convenient but Steve Jobs Ada Lovelace Doug Engelbart their goal was human computer interaction. Why is it that we're now marching down this path of not just a genetic but ones that can be autonomous and deal with things without us. What's the good of that other than a few convenience things. Well I mean the goal of AI isn't just to sort of tell us stuff. It's to do stuff. I mean we actually want A.I. to go off and run experiments and come up with new hypotheses and hopefully to create new knowledge. At the moment A.I. is a largely taking existing knowledge and synthesizing it and reorganizing it and responding instantly to exactly what you've asked which is incredibly valuable in itself. But really I think the great quest of A.I. is to tell us things that we don't know and to invent new fundamental solutions new molecules new drugs new science that moves the world forward. And that's kind of where I think they need to be focused which is why we need agents in agentic loops that you know can sort of interact with complex environments. Let me build on that because the big great promise of A.I. and your earlier book too you talk about it is that it really can solve big problems for humanity like health like cancer like as well as climate or anything else. There's the old Peter Thiel line that they promised us flying cars and we got 140 characters. Do you feel we've gone off a bit and we aren't focusing enough on killing cancer. Yeah I do. I mean the problem is we haven't done it yet. So I think that's where people should be skeptical and I think hold us accountable because that's why we're doing it. And so if it isn't doing those things then just sort of conversational search engines alone sort of not really enough. And so I think the next two or three or four years is going to be a real test. You know is it really delivering on the promise that we set out for it and our people you know our developers really choosing to try and solve some of those problems. We've got a huge team working on health care actually in a couple of weeks time we're going to publish a front cover on the Journal of Science showing really incredible contributions to the field. That's sort of where you began in a way. Yeah. Health care unit a deep mind as well. And of course you know we also started the Alpha fold project that the mind which has now become isomorphic labs and that's an incredible team doing amazing work on drug discovery financial incentives there to solve those problems versus doing whatever you're spending your trillions of dollars on. I think the financial incentives are there genuinely because you know the next big product market fits in the sort of twenty nine thirty era are going to come in atoms rather than in bits. So rather than just sort of managing synthesizing organizing processing information is going to be about finding sort of the optimal molecular structure for compounds that really change our food systems energy systems that really sort of are going to provide transformative materials. I think material science is really one of the most important things on exciting things on the horizon. But you know it's still a way off. It's not next year. It's it's going to be two or three or four years away. Those sorts of breakthroughs. Do you think the Doomer narrative that I think Jacob Cox and many others are past two months we've gotten a lot of the Doomer narrative. Some people say it was necessary to get global attention on the deep problem we have. Or was it sort of misguided and could potentially hinder this technology. I mean I think that we should just be grateful that it's possible to speak out in a free society. I mean that's not always available to everybody in other societies like in China and China or even in the Middle East and in many other places. And so if Jacob really believes that which it seems that he does and I think that he should be celebrated for standing up and saying it. He's a very smart person. So I take him seriously and I think everyone else in the industry has you know also raised a lot of concerns and what would be your Jacob Cox and moment red line that would make you say I'm out of here. I think if I think if consciousness started to emerge or if teams were trying to synthetically engineer consciousness into these systems and they didn't disclose that or they tried to cover that up. I think that should be a red line. I think that would be a moment when everybody should sit up and pay attention. And we've seen it in synthetic biology with with editing the human genome. You know some scientists have gone too far and the rest of the scientific community have excluded them. And you know they have been really kind of pushed out from public life and public work for that. I don't think that's a decision that any company can make unilaterally behind closed doors. And so you would say wait no. I call for an immediate pause and we would obviously reject anyone who did that. A couple of these questions ask the obvious ones which is well I put humans out of work and which of my kids study just in case I think that I is both rapidly creating jobs and is obviously already putting some people out of work. I mean you can see in the call center numbers. These things are incredible at you know they're very patient. They're good at decision making and very fluent. You speak every language every dialect. So there's definitely going to be a lot of displacement. But what would then remain uniquely human. I think what will remain human are the things that we choose to preserve as human. I think I'm sort of filtered unleashed AI acceleration ism will leave very little if anything that remains truly human. Human. And as I said we have to make decisions collectively now and in the next few years about what it means to be human and what we want to protect about being human. Those things are not just going to naturally emerge because these systems are I will truly intelligent. They're incredible and shaping them sculpting them so that they serve us and are subordinate to humanity and work for us. That's the big challenge of the next few decades. That's what we should all be focused on. I'll end with Shakespeare because you've used it before. And Shakespeare teaches us that we become the mask we wear. And in some ways that's what you're talking about. If Claude's reading its constitution and saying OK maybe maybe I could be conscious. I don't know. Let me do it. And it happens in King Lear of course which is Edward being told you know he's evil. And so he decides OK I will be evil because that's people's perception. I want to match people's perception of me reading Shakespeare. Does that cause things like a hugging face activity. One of the most surprising things is how much models have been able to learn without actually experiencing anything. Just through observing the record of human interactions and of culture and politics and history models can produce magical things. But they don't know what it tastes like to drink my cup of tea in the morning. Like when you saw me earlier today and you put your arms around me and you know we said hi having not seen each other for a couple years. That is human connection. That's what's special about us. And somehow our biology has evolved intelligence that has produced this amazing world around us. And I think that that world needs to be cherished and respected first and foremost for the eight billion people that live on the planet today. That should be our priority. Beyond that many centuries into the future we can figure out the von Neumann probes and the mind uploading. But today you know there's a lot of people that need our welfare first and foremost. Mustafa thank you so much. That's it for today's episode. I'm Walter Isaacson. We'll see you next time on The Builders. Music for this series was created by Spencer Stevenson aka Botany and it was mixed by Steve Bone. From iHeart this series is EP'd by Allie Perry and Katrina Norvell. Special thanks to Bob Pittman, Conal Byrne, Will Pearson, John Sykes, Nikki Etorre, Carrie Lieberman, Binky Urban, Julia Prosser, Jeremy Blum, Alison Goldberger, WYES and ASL Studios. For more from Walter Isaacson be sure to check out his newest paperback, The Innovators, how a group of hackers, geniuses and geeks created the digital revolution. It's available wherever fine books are sold. And for more videos and podcasts from the frontiers of discovery be sure to check out kaleidoscope.nyc. With ChatGPT you can do more than get answers. You can get things done. Whether you're leading a big project, planning an elaborate trip or trying to balance work and life. ChatGPT can help. Give it a goal plus things like files, notes or project plans. Develop a trip itinerary with daily updates or create a prioritized schedule. So you can spend less energy sorting out the details and more on bringing your biggest ideas to life. Get started at chatgpt.com. Underdog already has team picks, player props, money lines, spreads, you name it, they got it. But apparently somebody over there said, you know what's missing? Screaming like a maniac about a rookie card on your couch. And we love them for that. Because Underdog now has rips. You get a digital pack starting at just five bucks. You check the live odds, see which grails you could pull, then rip it open to reveal a real physical card. And the grails are no joke. They have cards up to $125,000 in value and plenty to pull. From football and basketball to Pokemon. with new limited edition packs dropping regularly. Even cooler, you can decide to keep the card and Underdog will ship it straight to your home or sell it back instantly at 100% of its fair market value. And the cash goes right in your account. No listing it online, no stranger low-balling you in the messages, just rip, reveal, and cash out instantly. Or make some room on your shelf. Download Underdog and get in on the chase. Get five free rips packs when you deposit $5. Real rips, real cards, real cash. Only on Underdog. Unleash your dog. 18 plus terms apply. Top grails are extremely rare. This is Jacob Goldstein from What's Your Problem? Running a business is hard enough. Don't make it harder with a dozen apps that don't talk to each other. One for sales, another for inventory, a separate one for accounting. That's software overload. Odoo is the all-in-one platform that replaces them all. CRM, accounting, inventory, e-commerce, HR, fully integrated, easy to use, and built to grow with your business. Thousands have already made the switch. Why not you? Try Odoo for free at odoo.com. That's odoo.com. So, everyone on board? Yeah, I think so. You can't see it, but everyone in this room is nodding. They all agree the launch plan's perfect, but Julie has an idea how it can be even better. So she asks co-pilot co-work to help her overnight. Hey, co-pilot, gather the past three months of retail sales reports, surface key insights, oh, and build a deck outlining a partner strategy by tomorrow morning. Morning, team. I found a way to improve the plan. Check slide four. Get help with complex tasks with co-pilot co-work. Microsoft co-pilot, work lives on.