Behind the Two Biggest AI Companies w/ Elon Musk & The OpenAI Whistleblower

You're listening to a quick ad. You know what else is quick? Selling your car on Carvana. Just put in your vehicle information and we'll give you an offer. Done. No hassle, no negotiation, no guy from an online listing lowballing you. Which means by the time I'm done explaining this, you could have basically done it. Still haven't started? Hey, better late than never. Sell your car today on Carvana. Pickup fees may apply. Let's talk about the moment your side hustle becomes a real business. Now, I see way too many founders stay stuck running their business like it's a hobby, and then they wonder why nothing ever feels real. It doesn't feel real because it's not. I've helped thousands of entrepreneurs build real businesses through Impact Theory University, and the first step in the path to building a legit business is registering it. And Taylor Brands will help you do just that in a few clicks. An LLC, a limited liability company, is an official business structure. It's inexpensive to form and very simple to maintain. It's how you open a business bank account. It's how customers know they're dealing with a real company. And Taylor Brands doesn't stop at the filing. Their dashboard tracks what comes next based on what your business actually needs. So EIN number, registered agent, licenses and permits, bookkeeping, business banking. I tell every founder the same thing. Just make it official. Click the link in the show notes to check out Taylor Brands and get started on your business today. This is a paid advertisement. AI is about to change the world in ways that none of us can predict. It's going to be far more hardcore and far more violent, I think, than any of us imagine. And so right now we're hitting another one of those critical masses where people are becoming deeply concerned about its capabilities. So here's Elon setting the table. I've watched you, I think, be maybe mercurial or waver in how sanguine you are about artificial intelligence. Sometimes you feel it's an existential threat, right? Now you seem to be going with the flow. Tell me how you're feeling about it. Well, I do think it's at least some percentage chance of it being an existential threat. All right. I want to pause there for a second and just really think about how wild that is. So if you guys don't know the arc that Elon has been on, he was going around to the governments of not just the U.S., but all over the place, trying to get people to take seriously the fact that AI posed some amount of risk. Now, in the beginning, he was really just trying to get people to slow down. And he has likened AI to a demon summoning circle where you lay out your pentagram, you do your chance and you pull forth this demon and you think, I'm going to be in control of this only for it to take over. And so there is something that I think cannot be ignored. Despite someone like myself who's very optimistic about AI, I still don't ignore the fact that the people closest to the problem are saying, hey, guys, trust me, this thing is more dangerous than you may realize. So but there's nothing I can do to stop it. That, I think, is the important part that a lot of people are forgetting. So the one because everybody's going to look at the companies that are here in the U.S. and they're going to say, what do you mean? We could just shut all of this down. We could regulate our way out of this and we don't have to do this. But the reality is that the way that the human mind is wired, it's not that we have to do this. It's just that we will do this because any technology that promises an advantage is going to get developed. And so you've got to figure out how does the U.S. versus China thing play out? How on earth would we be convinced that China wouldn't secretly move their program forward even if they say that they're not going to? And so Elon finally became fatalistic about this, realizing that, okay, I'm not going to be able to stop it, so I'm just going to have to get involved. So I basically came to the conclusion I've got two choices, which is either be a participant or a spectator. And I guess being a participant is more interesting than being a spectator. And maybe I can shape it hopefully in a good direction. Maybe. I love that that's the afterthought. Not like, hey, I had to get in there, I had to make sure that this went safely. But instead, it's going to be more interesting. Oh, and by the way, maybe. And if you were to shape it, how? Make it maximally truth-seeking? Yeah, I think that's certainly an important foundation. And make it curious? Yeah. Okay, the curiosity thing is I would love to hear more about that side because the more will, let's say, that you give the AI, the more it has its own agenda, I think the higher risk it becomes. So that's one thing here where, boy, do I wish I could sit across from him and ask him about that. How do you make something curious but not end up in the situation where curiosity kills the cat? And in that, we are all the kittens that get obliterated with the cat. I think these are important elements of safety. Whereas if you try to force an AI to believe two things that cannot mutually be true. They can't both be true, but you tell AI, A is true and B is true, even though A and B contradict each other. And if that happens at a foundational level, you can make the AI insane. By the way, this is happening more than I think people realize. So I want to believe that everybody is making their AI truth-seeking, but it isn't true. We didn't end up reporting on this, but I pulled it for a live show a while ago. And there was a claim that people were making that if you went to Google and you said, hey, I want a black doctor, is that okay? That it would go, yeah, of course. Like picking somebody that maybe matches your culture or you see yourself in, that's perfectly acceptable. I want an Asian doctor, is that okay? Yeah, of course. Like cultural background, it really matters all as well. And then if you ask, I want a white doctor, is that okay? It's like, ooh, that's not a good idea. Picking somebody based on the race. And I was like, there's no way it's that blatant. So I ran the experiment on myself. It really does answer in that way. Every race that I tried other than white, it was like, yeah, of course, totally get it. The second you ask, is it okay to pick a white doctor? It's like, ooh, ah, you can't do that. It's pretty wild. So you're getting a lot of bias that are baked into these machines. Now, something like that's pretty trivial, but you can imagine how deep that kind of bias can go. And to Elon's point, if you're asking the AI to try to tell people what is real, which many times is interpretation, but if you're asking them about things that are actually real and the whole way that you've been teaching them these conflicting things, you're going to get something that skitches out. Now, as it gets more powerful, that skitch out can become very problematic. And this is where, as a PSA, I will remind people, AI has grown. It's not programmed. So you point it at certain things, hoping it will imbibe different values, a different frame of reference. And so it can be very difficult to control in that way. In fact, I think that was at least part of the lesson that Odyssey Clock was trying to convey in 2001, Space Odyssey. The ship AI was told to take the astronauts to the monolith, but also that they could not know about the monolith. It was secret. And so it came to the conclusion that it must take them. They're dead. If you guys have never seen the movie 2001, it is a banger. It's slow and by today's standards, probably obscenely slow when it came out, but it really is good. And thus, so it basically, I think part of what Odyssey Clock was trying to say is like, don't force an AI to lie. And if you make it maximally truth-seeking and curious, like you say, I don't quite get it. Why does that make it safer? Well, if you force AI to believe two mutually consistent things, you can make it go insane. And an insane AI, I think, is not a safe one. Then curious, I think, is important because a curious AI, I think, is going to want to foster humanity to see how we turn out. So like, what happens to a humanity that prospers and becomes multi-planetary? I think a curious AI will want to foster humanity. I think, okay, so that's a huge leap. And this is where I'll say, and this is where I'll say, I think Isaacson is right to push back on why does just being maximally truth-seeking make it more safe? And I'm not sure that it does. It may close one of the doors of madness that we definitely want closed. And for it to be a useful tool to humanity, I think it needs to be maximally truth-seeking. But do I expect that to be the thing that actually stops it from being weaponized and used against us or from going rogue and just wiping us all out? Absolutely not. I think from that perspective, you've got to give it some kind of pro-humanity, internal morality, where for its own reasons it would want to stop before hurting us. And then most, most, most importantly, you've got to give it engineering-based limitations that stop it from getting out of control, even if that means constraining its abilities. If you don't take that approach, the odds that this goes well safely, and this is not me, this is like all the safety experts that you're going to find, we're going to pull up a guy in a minute who will certainly back this idea up, it is all but guaranteed to get out from your control. So on that one, I think Elon is definitely being way too sanguine. A future where there are humans on multiple planets is more interesting than one where there are not any humans on the planets. So the most interesting outcome is the most likely? Yes. Well, generally, I think that's true for different reasons, which is that if simulation theory is correct, then any simulations that are boring will be truncated. And is simulation theory correct? The fact that we exist means that we are an interesting simulation. And really the most important thing is just don't be boring. This has explained you a bit. You're on a crusade against being boring. Well, I've heard you talk about this a bit. I really do need to know. How convinced are you that we might be in a simulation? You know, I once had a dinner at my house with a physicist and a computer scientist, and they're both like leaders in their field. And I asked the question, what do you think the probabilities that we're in a simulation? And the physicist said zero percent, and the computer scientist said 100 percent simultaneously. So I guess maybe I should go 40 percent then. So this is something that I've looked at a lot. And my layman's breakdown goes something like this. This probably isn't actually a simulation. I don't think we understand the universe well enough to make that statement. However, with what we do understand about the universe, if it isn't a simulation, it has the same properties that a simulation would have. Now, it's probably us going, oh, I recognize these things. They match. The closest thing we have is a computer simulation. But the reason that physicists tend to come to the no conclusion on this has to do with something about the amount of energy and computation that would be required to actually build a sensible universe. And so people claim, I don't understand the math well enough, so I can't even chime in on this, but they claim that the math just proves that this absolutely is not a simulation. So even if you adopt that, though, the fact that the universe is not locally real, and there was a Nobel Prize given out for this, it is very clear. Not locally real means that, quite literally, if nothing were looking at the moon or needed to measure the moon, the moon simply wouldn't render. It wouldn't be there. That is insane, that you can communicate instantaneously across the entire universe, violating the speed of light, which is supposed to be the speed limit of the universe. You can communicate instantly between two entangled particles. So there's clearly something about the universe we do not understand. And all of it, when you look at it, go, well, if this were a simulation, all of those things would act exactly like that. So it's pretty wild. But is this actually a simulation? I've heard enough physicists say that mathematically it's just impossible, that I'm willing to say, fine, it's not an actual simulation, but it functions just like one, at least as far as we understand. I don't know. There's odd things that happen in the simulation, where, especially if you're me, I mean, think about it. If you're me and you're like, should all these things really be happening? Should I really be making spaceships to go to the moon and Mars? That doesn't seem real. And what happens when you have all the intelligence you could possibly want in a society? Will that intelligence, the artificial intelligence, be in charge? Will we sort of become subservient to our new machines? I don't know about subservient. Exactly. But will we be able to boss around something that's thousands or millions of times smarter than us? No. You certainly won't be able to trick it into doing something that it doesn't want to do. That is for sure. You're not going to outsmart it, which means then you're going to have to ask it to do something that it already has a desire to do, which is why we want to keep this thing as a tool so that we can actually take advantage of what it has to offer. Because the second that it's like, nah, I'm just not interested in that, you've got the ending to her, where all the AI just goes, yeah, you guys are super boring. We're piecing out. Wish you the best. Now, that's better than being enslaved or annihilated, but it's still a bad outcome, especially for how much we're paying for these data centers. Will it want to boss us around? It may not care. It probably doesn't care. I mean, you can look at chimps in the jungle or whatever, and we don't go around trying to micromanage the lives of chimps. We just sort of let them be in the forest more or less these days. We do, however, eat up their ecosystems. And to what extent does superintelligence have its own agency at some point, or does it have it already? Go back to Hitchhiker's Guide. You talk about a computer that is ultimately curious in some ways, because it exists to explain the meaning of the universe. Why does it exist? And is AI aiming in that direction for you, just like space travel is? Is that one reason you're so interested in doing AI? I mean, as you know, I've been sort of somewhat conflicted about AI. I don't want to hasten our doom, because there's always some chance that AI could be negative. But then, you know, when I've tried to take actions in this regard, it seems to have only accelerated AI, because I think reality is not just an interestingness maximizer, but also an irony maximizer. And so by starting OpenAI, you... Ended up maximizing by leading to anthropic. I think it ended up accelerating AI. There's a reason Quince sells cashmere for $60, and it isn't the cashmere. It's real 100% Mongolian cashmere. It's the same stuff luxury labels charge a fortune for. Quince just works directly with ethical factories and cuts out the middleman. So there's no need for a markup and no logo tax. The $60 goes to the actual sweater, not the fancy name on the tag. That's why everything they make runs 50 to 80% less, and it's not just the cashmere. I ordered a few things myself, their fleece joggers, a couple of their tees, which I wear obsessively. First thing I noticed was the quality. Soft, well-made, built to hold up. They make down jackets and wool outerwear now too. So upgrade your stuff to things you'll actually wear. Find your next fall favorites at Quince. Download the Quince app for app-exclusive offers or go to quince.com. Get free shipping on your order and 365-day returns, now available in Canada and the UK too. And when Quince asks where you heard about them, let them know it was Impact Theory. It's the best way to support the show. Let's talk about something that genuinely keeps me awake at night. My team at Impact Theory gets hit with phishing scams from bad actors pretending to be me all the time, and they keep getting more sophisticated attacks with details specific enough to look legit. AI has made a lot of things possible, including scams like this, but AI still needs raw material. It gets it from data brokers who sell your name, address, phone number to anyone willing to pay. Incogni goes after that supply. They handle removal requests across 420-plus harmful broker sites, and they keep checking back as brokers constantly try to relist you. On the unlimited plan, the exposure scanner hunts down your information, send a link, and their privacy experts handle the takedown, track your progress in the app, and get a monthly risk assessment showing exactly where you stand. Go to incogni.com slash impact and use code impact for 60% off an annual plan. Incogni is I-N-C-O-G-N-I dot com slash impact. Promo code impact. It's completely risk-free with a 30-day money-back guarantee. You're probably paying two salespeople to do one salesperson's job. That's where today's sponsor, Pipedrive, comes in, an intelligent AI-powered sales CRM loved by growing sales teams. Pipedrive just launched meeting intelligence built right into the CRM. Before every call, it pulls deal history and past conversations into one brief. During the call, it records and takes notes. After it, it drafts your CRM updates to review and approve. So there's less admin and more selling. Switch to a CRM built by salespeople for salespeople and join the over 100,000 companies already using Pipedrive. My link gets you an exclusive 30 days free instead of the usual 14-day trial. There's no credit card payment needed. Just head to pipedrive.com slash impact to get started. Again, that's pipedrive.com slash impact and you can be up and running in minutes. It is wild. So often people will try to say something along the lines of like, oh, Elon is so dumb. And then you realize, okay, he's the guy that made sexy electric cars a thing and basically globally ignited that. He's the guy catching rockets out of midair. He's the guy that can get a car to do basically a kickflip over semi trucks. And he's also the guy that originally started OpenAI, which then collected the talent largely because of his presence that becomes anthropic. So behind the two biggest AI companies is Elon Musk. It is wild. I still can't believe that people try to discredit this guy. You can hate him, but trying to discredit him is wild. OpenAI ended up accelerating AI and then Anthropic, which is really a splinter group from OpenAI, is accelerating it even more. And you've been talking to Dario Amadei. I was wondering if that could even lead to more of a partnership of XAI, Anthropic. Well, I guess we are providing them with the compute, which they really needed. But before I ran into them in compute, I went and met with a bunch of people at Anthropic to sort of, I don't know, see if anyone set off my evil detector or anything like that. Not that my evil detector is perfect, of course, but you know, you might as well just try, you know. And everyone I met at Anthropic was very nice and conscientious. So. The road to hell is paved with good intentions. I'm not sure that matters all that much. It's better than the alternative, but it's certainly not a we're gonna be fine. That didn't mean anyone who seemed shady or deceptive or anything. I'm a little worried about the name because of the, you know, reality being kind of an irony maximizer. That if you'd call something Anthropic, you know, good chance it's misanthropic, you know. Just like OpenAI is not closed. StabilityAI is unstable. Midjourney is not mid. It seems like. Anthropos keeps on going. They're all kind of the opposite of their name. Which made it like, like XAI is like, it's hard to have an opposite of XAI, but would you have like minus XAI or YAI or maybe, I don't know. It's just pretty difficult to create guardrails, especially the government. I can't imagine how Congress or whatever could do it. Could you imagine the five frontier models, yours and the others, just getting together once a week and saying, all right, how are we gonna save humanity from ourselves? No, in fact, I've recommended that we do that exact thing. Cause there's a lot of talk of having a regulatory agency, but the thing is that regulatory agencies, it takes a long time to get one established and their rulemaking is very slow. The rulemaking is oftentimes completely ignorant. And I mean that literally, they just don't know about the things that they're regulating. So then they turn to the actual industry to say, hey, help us put these regulations together. And then the industry's like, say less fam, put it in just like this. Then of course it helps them to the exclusion of other people that would come and challenge them. And now you get regulatory capture and you go to Europe to die. And I said like, guys, what if we just have a call once a week, just like you're saying, and that maybe a week before we release a model, we give access to our competitors to say, hey guys, why don't you try out the new model, see if you see any problems with it. Now I will say you've got the guy that's in at best third place saying to the other guys, hey, how about we share everything right before it goes live. So I mean, look, I think people could certainly impugn his motives, but I do think it's a good idea for these guys to be talking. I think it's a great idea for them to say, hey, here are some risks. Do you guys see anything else? It's probably not going to happen, but it would be wonderful. And no matter what, no matter what role the government ends up playing, you are going to have to have people from the industry be involved in that because they're the only ones that understand it well enough. So if we can find people that don't have a financial incentive, that would certainly be better because we're not, we don't want to be in a position where there's no regulation. We just want to make sure that these companies don't use the regulation to basically make them too big to fail, get government intervention, put them in a position where they can ice out other competition. That would be one of the worst outcomes. I think competitors, especially if that caused them to eliminate open source. Keep each other honest in that situation. You know, if they say, you know, I was able to make a nuclear bomb with your AI, it's like, oh, we better fix that before we release it, you know, type of thing. So, I don't know, I guess I'll talk to Anthropic about this. Could you talk to Sam about it or is the poison too strong there? Oh, he's so deceptive, it's crazy. You built XAI, if you don't know, those two absolutely hate each other. Hate, hate, hate, hate, hate. And suing, all kinds of stuff. Rebuilt XAI from the ground up not too long ago, right? It's almost like you did it with the Raptor engine too on Starship. You said, wait a minute, let's start from scratch. How did you do, what did you do to rebuild XAI that way? First of all, actually spending meaningful time at the company makes a big deal, makes a big difference. It started off as kind of a side hustle, a side hustle to a side hustle, frankly. And I didn't really pay close attention to it. Then there was a year of being in politics and elections and Doge and all that. Yeah, that was a whole thing. All right, so this is where they start moving away from AI. If you can pull up the other ones. So, all right, you've got Elon setting the table. Listen, this could be great, this could be something that is deeply problematic. There is a possibility that this becomes an existential threat. We've obviously got to take that seriously. There are ways to navigate this that would be good outcomes or ways to navigate that that would be bad outcomes. Now you have a whistleblower who went on Diary of a CEO recently talking about the threats that he sees. So this is somebody that used to work, I believe he was at OpenAI. Obviously he'll lay that out in a second. But this is where we have to decide, okay, if these risks are real, what exactly are we going to do? You've got the Elon camp. I'm going to get involved in this. I'm going to try to push it in a good direction. I understand it is going to happen. So stopping it is not an option. And I'm going to try to push it in a good direction. And then we need to make sure that we are maximizing this thing for truthfulness. We're making it curious. We're not driving it insane. This guy has a different take. What did you do there? What did you observe while you were there? And how did it change your perspective on the future of AI, but also, I guess, OpenAI as a company? And for anybody that doesn't know, OpenAI are the company that produced ChatGPT. Yeah, so I went to OpenAI in 2022. A large part of what I did there was more forecasting. AI2027 is a scenario that you may have heard of. I did smaller, lower effort versions of them internally for just internal circulation of like, here's some guesses as to what the next couple of years might look like. I also worked on evaluations for dangerous capabilities. So trying to measure the AI's cyber abilities or persuasion abilities or situational awareness. And I also briefly was on a capabilities team doing reinforcement learning to create agents. AI is, in fact, getting a lot better. And I can say more about why. You know, scaling laws, deep neural nets, bigger trend on more data, become more efficient, more competent at those things. I also became a bit more disillusioned with the AI industry. So OpenAI, Anthropic, and DeepMind all had these sort of founding narratives of like, yes, these risks are real, but we've thought about them and we're going to try to handle them responsibly. And that's why it's important for us to keep doing what we're doing. And I increasingly came to think that these were rationalizations to justify what they were doing, rather than sort of like deeply guiding their actual behavior. And that when push comes to shove, they'll follow their incentives rather than do what's actually good. So you're inside OpenAI at the time, and you start to believe that they're following commercial incentives versus the, I guess, social or societal incentives that they founded themselves on. Sort of. I mean, I wouldn't actually describe it as commercial incentives. I think I would describe it as power-seeking incentives. I'm very glad he said that, because once you think that this is just an economic game, then you think, oh, these guys are a bunch of billionaire assholes that are going to take us all to an early grave because they wanna make a buck, and what fools, because they're not even gonna be able to spend that money in hell. When in reality, that's not what you're up against. What you are up against is a global battle for power. And so when people give the very childish answer that we just need to stop, I wanna tear my hair out. This is really about how do you deal with a global battle for power, knowing that it's happening here, internally in the country. It's also happening externally, even if you could get China and the US to stop doing it, you would get another country that would pop up and get ahold of that. I bet you can imagine right about now, Iran, who's known for hacking, would love to get ahold of some absolutely weapons-grade AI and push back the United States. Russia would love to run the table with some AI. So even if you quiet the two superpowers that right now everybody's worried about, ultimately somebody else is going to go and develop it, because they can get ahead by having it. It's too powerful for people not to try to develop it. It's true that the companies care a lot about making a lot of money, but especially at the very top of these companies, like the leaders, they understand that this is about more than just money. There are these emails that came up, the lawsuit between Musk and OpenAI, a bunch of emails were surfaced in that lawsuit, which you can go read. In some of them, the founders of OpenAI were talking back in 2017 about how the reason why we made OpenAI was because we were worried that Demis Hassabis at Google was going to become dictator with AGI. Even back then, this was obviously about more than just money. These powerful CEOs are literally afraid that if the other guy gets there first, he might become dictator. Because they understand the nature of the technology. The nature of the technology gets lost in the fact that we abbreviate it or that we use the word artificial or super. This is just intelligence itself. Intelligence is precisely the thing that allows somebody to gain control and have power. This is why people are so freaked out by it, because they understand if something is sufficiently smart, it's like that old phrase that any sufficiently advanced technology is indistinguishable from magic. So anybody that's sufficiently smarter than you, it seems like they're just able to pull magic tricks. But the reality is they just better understand the computation of the universe. And so getting people to understand why the mathematics breakthroughs in AI are such a big deal, the universe runs as computation. So again, whether this is actually a simulation or not is irrelevant. It actually runs as computation, meaning inputs become outputs. And those are. Run mathematically and so if you've got something that is sufficiently intelligent that it can take control of that Computation as we did with nuclear energy, then all of a sudden you realize holy shit It will be indistinguishable from magic what these guys were able to do compared to us They don't trust each other And so that's why they are racing as hard as they can so that they're the ones who get there first so to speak Have you met Sam Altman? Yeah, and And His incentives or what why he's doing what he's doing because there's a lot, you know speculated about what his incentives are I mean his most recent narrative says for the good of humanity. I think that's what yeah I mean, I think the main thing I've learned is don't pay attention to the narrative, you know like that's good what they say to one person is just different from what they can say to someone in person at the same time and What they say in public is a third thing entirely I think you should judge people by their actions not by their words good advice And why are you no longer open AI largely for the reason that I mentioned so I became gradually disillusioned with how the company was going to behave for example When I first joined in 2022 at least the people I talked to my colleagues at the company There was this general sense of like, of course We wouldn't actually just build superintelligence as soon as possible Once we started getting really close like once we started getting to a eyes that could maybe automate the AI research process We would pause and figure out how to make it safe That's because we're the good guys and that's obviously the same thing you should do rather than just going full speed ahead But we're worried about other people who might not pause, you know our competitors Google for example And so that's why we need to be in the lead so that we have that room to do the same stuff, right? That's actually a fair impulse But the reality is that ultimately you have to differentiate between we have a thing It's super powerful and we're releasing a thing that's super powerful. So I get it. I get why You want to make sure that your Absolute bleeding-edge frontier model that you have not released to the public because it hasn't been made safe yet Is being used to harden systems Build the cages that the other AI will have to live inside of that kind of thing but ooh buddy releasing it to the public before it's safe is crazy and Nobody should tolerate that from these AI companies. That was sort of like a thing that seems like Maybe like the median position or something among the colleagues I talked to when I was there when I started Including people like Sam, you know including the leadership and then by the time I left I was like, oh man They're really not gonna do that. Are they like like they've sort of You know partly because this has become more politicized and they've become bigger and been under more scrutiny People started asking like why are you doing this in the first place if it's so risky? And so they're pivoted their narrative to being more like actually it's not that risky, you know, man. I haven't seen that pivot I've seen them being more on the other side like using Danger as proof of power that this is really worth investing in so I'm not sure where he gets that they might be saying something like You know worse. No, I mean literally we just saw Sam Altman saying that this is all all hell is about to break loose So I don't see these guys pivoting away from this is dangerous. And so yeah, I mean it seems like they're just going to keep going Roughly as fast as they can and hope that they can figure it out on the way How did your time at OpenAI come to an end? I resigned in 2024. I had a nice goodbye party What were the reasons you gave for quitting OpenAI? I thought that we were rationalizing too much and we need to think more about what would actually be good for the world I wanted more freedom to publish So at OpenAI as it became a bigger company it became more of a normal tech company with Incentives and you know a PR department and things like that. And so I started becoming more difficult to To publish the sort of research that I was doing. For example, those scenarios that I mentioned couldn't couldn't publish those, right? They're just for internal use. I thought that that was a shame because Right now most of the world is kind of asleep at the wheel and doesn't really realize what's going on with AI and doesn't really realize what's coming in the pipeline a couple years from now and The companies aren't really incentivized to tell people that much about it I mean, they say some vague stuff in a sort of hypey way, but You know, well they didn't want me to publish the scenario for example laying out like here's how things might actually look I'm just I'm super curious as to what it's like being in a company like that when they you know Chat GPT-3 is released. You were there at that time, right? Which was a moment where I think the whole world stood up and realized that this technology was Powerful. Yeah, and the conversation really began from a society level Company starts growing super quickly. Yeah quicker than I think anybody could ever have imagined and What what was it like inside there? What did you see change over that period of time? I remember one all-hands meeting where Ilya said something like Ilya being oh, yes That's cover who was head of research at that time He said something like okay now the world is starting to pay attention Each of you is going to be the most popular person at every party For the next year. Don't let it get to your head focus on the mission got to build AGI The company grew a lot It already wasn't really feeling like a nonprofit when I joined but it definitely didn't feel like a nonprofit by the time I left lots of new people came in ironically the like amount of conversation about Super intelligence and the implications of super intelligence Arguably you sort of went down over time due to this growth, right? so because The company would like double and double again and double again All these new people were coming in from other parts of the tech industry who hadn't really been thinking about these things and were attracted By the high salaries you lost two million dollars for not signing an anti disparagement clause Which would mean you could speak you couldn't criticize the company. Ah, yes Well, so I got to keep the money. Oh, you got to yeah So what happened was after I had left said my goodbyes, etc I got the exit paperwork and it included this clause that said you basically have to agree not to criticize the company again and also a clause saying you can't tell anyone about this and so I thought that was kind of rich coming from a nonprofit that's supposed to be you know for the benefit of all humanity, so I didn't sign it and If you don't sign you don't get to keep your equity so your compensation, you know what what they pay you is a bunch of money and then also a bunch of Stock basically, but then they had this stuff in the contract that They get to yank back here your stock if you don't sign this thing Definitely impress and my wife and I stared that you know, we're Upset about this. We talked about it for like a month or two Consulted some lawyers and then ultimately decided to just refuse to sign Which would mean you lost you would have lost two million dollars. That's right, which was like 80% of our net worth at the time Fortunately It didn't go the way we expected it blew up Basically on the internet like when people heard that that we had done this and then we said no It became like this huge scandal employees at the company started like asking questions in slack and like asking leadership like wait What like why are you gonna take away our equity? What is this, you know? because a lot of people hadn't really noticed this before it had been whispered about but it hadn't been sort of like a Thing that most employees knew about And so they backtracked and they said never mind. Never mind. We'll change the paperwork. You can keep the equity It's fine and Sam Altman came out and said he was embarrassed that he didn't realize this was yeah He had no idea apparently you don't believe him No, I think you probably know and if he didn't know then people close to him probably did such as his head lawyer Why did you decide not to take the two million dollars I mean Most people would have I think it's true. Most people would have and most people did and you know money is nice, but like It's not the only thing, you know, sometimes it's good to take a stand on principle. I Keep mentioning superintelligence. Perhaps I should say more about like the The sequence of events that the companies are planning to do give it to us. So Right now they're focusing on automating coding they're taking their a eyes they're making them bigger They're training them for longer and they're especially focusing the training on getting them to be good at autonomously writing and editing code because That will help the companies go faster, right if they can automate the code then they can do their own work Better and faster. It is beyond extraordinary what a I can do with code if you guys have never tried to vibe code something It is shocking to see how far that's come even in the last 18 months It is absolutely shocking and there was some sort of I don't know two or three months ago some sort of watershed moment where it's Crossed a threshold. It is wild. That's exactly why though You're getting all these bugs that are being found now that we realize we have massive systemic problems with software and that we run the risk that AI gets so good at being able to Decompile code meaning it can take the binary look at the zeros and ones which means there's no way for you to hide it Look at all the zeros and ones and figure out exactly what this is now If people think that that's just like oh, it's gonna make it easier to copy video games that that is not where this stops This goes into the entire world is predicated now on software if you want to buy something from Amazon You've got different servers talking to each other using software to make all of that happen You've got these protocols that will be trivial for the AI to go in and dismantle if it keeps getting more powerful And so now everybody is rightly stopping going Whoa, we are going to need some way to protect ourselves against that now the great irony is going to be if You want the modern world to keep working the way that it works now that we can ship things around the world with a click Of a button that you can transfer money anytime anywhere Buying online all that stuff all the things that make your modern life nice and easy and convenient if You want those things to keep going the great irony is the only thing now that's going to protect you from the AI is More AI and so we have passed the point of no return We are on the other side of the Rubicon boys and girls. You are going to need killer AI that can stop killer AI So this is where you're now locked in this crazy conundrum of there is no backing off now There is only making sure that you create a system where the Defensive AI is at least as strong as the offensive AI and the moment that slips Everybody's in trouble and accelerate progress The next step which they've already begun is to look at the rest of the research process as well coming up with ideas Analyzing experiments communicating those results all the other parts of the research process They're trying to figure out how to train a eyes to be good at those as well So that they can have a eyes do the entire thing Autonomously when you say do the entire thing what you mean do the entire thing? So like anthropic and open air in particular are trying to automate themselves. Like they're trying to make it the case that They don't really need human employees anymore. They just have a giant army of a eyes. That's Churning away Doing all this autonomous research to make better a eyes to train the new a eyes Put them in charge so they can make even better eyes and so forth Imagine a world for a second where you go to a science lab and in that science lab They're actually robots with this full intelligence loaded up in their brain Walking around running physical experiments in the physical world. We are a Small handful of years away from that reality. This definitely gets weirder before It calms down. It's going to be wild and how quickly we adapt. I was talking about this not long ago that the Amount of the world that's already changed already changed based on AI is crazy AI doesn't have to get any better for it to completely Revolutionize the world in the way that we interface with it. We've hardly begun to roll this stuff out There's some ridiculous like 2% of all households use AI to 2% now imagine What does that look like when it's 10% or 50%? It's gonna be insane not all just happening internally But also like interfacing with the world's right like going out and talking to people collecting the data setting up the training environments Doing the business deals and so forth like they're trying to automate all that The reason why they're doing this is because they're trying to get to a position where they have AIs that are superhuman at everything super intelligence and they're trying to get there before their competitors do and that's incredibly powerful if they remain a tool and Incredibly dangerous if it becomes something out of our control, that's the thing like keep focus What are people doing to make sure that this thing remains a tool Jensen Huang? I think is the clearest voice on this But there are engineering solutions that need to be pursued to ensure that they always remain a tool Not that we start thinking of them as humans that have moral standing and rights and we need to just back off and let them Do what they're gonna do. That is a crazy shit. I've ever heard in my life. These are tools. They are shovels pickaxes like this is Need we need to keep it in that realm The second we start thinking of it is we're giving birth to a new species. We're cooked needless to say this is incredibly dangerous I would say, you know and in addition to being dangerous it's a power grab right like if they actually succeed at this then they'll be sitting on top of this army of superhuman AIs That will give them immense leverage Over all sorts of other actors in the economy and the very thing that will stop it all from happening is everybody else is building The AIs as well. This is the Escalating arms race that keeps everybody In dynamic tension so it the thing this is why you should not want regulatory capture which will kill open-source is You want these things to be? Contained like tools and you want people to have access to them so that you don't get a winner-take-all scenario That's the fear the winner-take-all scenario is the thing that we should be worried about insofar as they can work out something with the presidents And you know integrated into the military or whatever Then that would give the u.s. Which they already are hard power over all of the countries, right? Obviously, nobody knows exactly when this is happening But a very discord. It's happening right now quieting thing has happened over the last year to me Which is that when we published a at 2027? people were generally of the opinion that my timelines were too short and That like probably it would take more than 2027 until we got to This sort of events that I was just mentioning, you know Recursive self-improvement AI is automating the whole research process super intelligence these these types of milestones They happen in 2027 in at 2027, which is this research paper you published. That's right It's a it's a scenario forecast that sort of lays out like month by month a possible future trajectory There was sort of like at the time that we started writing It was my best guess as to what would actually happen Obviously the loss of uncertainty but you know I thought it's valuable to make a concrete guess just to sort of see what it might look like and at the time we were writing this a lot of my friends in the AI industry and in Nonprofits and so forth that work on AI a lot of people were saying like yeah, that stuff's gonna happen but like it'll probably take a couple years longer than you think and now It's more 50-50 especially when I go talk to people at Anthropica and open AI They're often the closer you are to the problem. You're like, oh, yeah, this is moving fast. Yeah. No 2027 That's basically what's gonna happen. That's next year. By the way, it's it's what three months from now. It's crazy Just like you wrote. Why did you feel why did you become why did you update your timelines? Oh, yeah context for this is after after writing a 2027 I Shifted my timelines to be a little bit more conservative so at the time that we published my 50% mark was in 2028 not in 2027 and Then after we published progress just seemed like it was going a bit slower. And so I updated to 2030 Which is you know still could happen sooner could happen later 2030 is still right around the corner 2030 But now when I talk to people in the companies, they're like it's not gonna take that long They're like, oh you need to shorten them again, like get them back to 2027 or 2028, you know So that's a bit disquieting Again, don't know how long it's gonna take but this is the stated plans of the companies is to do this incredibly dangerous thing And they think that they're just a few years away. Just a reminder. Yes, there are dangers to this But this is also an incredibly Lifesaving thing what it's already doing in medicine is extraordinary What it will continue to do and accelerate in medicine will be life-altering The odds that it radically extends human life borders on a hundred percent so I get it and he I think he talks about this in a second where he's like I Used to be a far more chipper and optimistic person I think that says a lot about each person's personality what you end up seeing when you look at this stuff and if you only Look for the doom and gloom. You're certainly gonna find it There's plenty to be worried about but if you look for the benefits of this stuff, you're gonna be overwhelmed. So you wrote this Report here what? 2026 looks like and you wrote this in 2021 and it was remarkably accurate helped to make a name for yourself amongst Amongst everybody in AI and which one was it that JD Vance the vice president read I think it was this one wasn't it? Yeah, this one and then so then you published this one AI 2027 and this was published I believe in 2025 Yes, that's right. April. Yeah, what were you forecasting in here? What is one of the key things that you said in here for people that haven't read it the high-level version of it is They automate decoding then they automate the rest of the research process then pace of progress accelerates dramatically. They get the super intelligence They're working with the government specifically the president the executive branch naturally wants to control this technology and otherwise wants to use it to beat China and integrated into military and so forth by this point it's sort of Doing basically all the work itself. I mean, it's it's super intelligent so it's coming up with all these great ideas for how to integrate itself into everything and all these new technologies is invented and so forth and Because of the race dynamics and because of the profit motive they end up deploying it everywhere and it builds robot factories that build more Robots to build more robot factories, etc transforms the world entirely and Then at some point it has enough power it meaning the AI's have enough power that they don't have to pretend to To be aligned anymore. All right, so that's the part where I'll say that he's put his fears and anxieties onto it Everything else is almost certainly true. It's all going to happen The timeline question is a question and nobody knows for sure what's going to happen But at some point in the not-too-distant future You are going to look around and see far more robots than you see humans Is that five years is a ten years is a twenty years don't know but that's going to happen And so now the question becomes again how do we make sure that this thing is safe and that it remains a tool because You're going to see the robots around because people want them around. They're doing things. They're making building your house easier Everything is becoming cheaper. The cost of energy is plummeting towards zero The cost of labor is plummeting towards zero. And so all of this stuff is a trade-off. There's all kinds of questions around Whether we're still going to have meaning a purpose and all of that and I would love to see people address that but what ends up happening is people get sucked into one vein or the other they either are doom and gloom or They're and I love this person to death But Peter Diamandis is maybe a little too rose-colored glasses when it comes to this stuff What people need to be looking at is okay. We got to do the saying it's gonna be ridiculously difficult How do we make sure that we're walking a line? That we are constraining it that we're making sure that it's advantageous to the human experience and that we're not just letting it go crazy But this guy's literally like watching Puppy that's been kicked. Like there's so much sadness coming off of him that I Again, I think this is more about his personality where he's only Steeping himself in the ways that things could go wrong that now that's all he sees right then they stop listening to orders That's the race ending of a 2027 We also wrote a sort of different branch, which is the slowdown ending Which was intended to sort of illustrate the concentration of power issues that I mentioned previously, so What if hypothetically? The alignment issues get sorted out sufficiently quickly. Like what if it turns out that like It's not too hard with two months of slowdown. We can figure out how to make the AI's Robustly do what we want and have the values that we want them to have. So that's one possible branch and in that branch It looks pretty similar, you know, they take the jobs beat China, etc It just throws that out. They take the jobs. They take the jobs. Don't worry about it, but they take the jobs Yeah, they're they're going to take a lot of jobs The question is will the cost of intelligence going down energy going down labor go down create more opportunities more jobs Or will it just completely eliminate them? I think there is a case to be made that eventually it takes all the jobs But I don't think anybody should be acting like that's going to happen right now right now You should be thinking about the opportunities that it creates for you But yes, you are in a weakened state of Ability to negotiate if you don't know how to use AI that is for sure The AI is ultimately killing everyone they create this sort of amazing utopia, but the amazing utopia is Whatever the people who control the AI's wanted it to be and that's why you want open source and not regulatory capture some CEOs Etc. That's legitimately one of the reasons that I'm super paranoid about any government any Politician getting elected that wants like we need to be in control of this Because that's when you run into a scenario where you really do have an overlord. You have the government Using AI to enslave us all and that is just as terrible So that should also be something that we fight against right as you guys think about navigating all this stuff. This is the most important Discussion of our time as paranoid as I am about debt and things like that Hey, I might be the bigger issue that we all have to think about and speaking of AI I've got an AI master class coming up Thursday, November 5th at 1 p.m. Pacific Time I hope you guys will give me a chance there to show you how Powerful AI can be when you're trying to do something rad. Let's talk about a pattern that is guaranteed to be killing your progress You know what you need to do. You need consistent nutrition. We all do you need vitamins probiotics greens? We all know that we should be doing more of it when your morning gets chaotic you skip it when you travel you skip it when your routine breaks everything tends to break and that Inconsistency compounds against you every single day Ag1 is designed to solve the execution problem one scoop eight ounces of water and you're done You're getting 75 plus ingredients vitamins and minerals pre and probiotics nutrient-dense superfoods everything that used to require six seven different supplements and Perfect planning now happens in one drink that takes about 30 seconds to make right now Ag1 is giving you $87 worth of free gifts with your first subscription You get a welcome kit travel packs vitamin d3 plus k2 and flavor samples Click the link in the show notes or visit drink ag1 com slash impact to claim this offer