EMERGENCY PODCAST: ASI Will Kill Us All!

Key Takeaway

Treat high-impact technologies differently from everyday tools: before adopting or advocating for more capable AI, ask what it is optimizing for, who is accountable if it fails, and what safeguards exist before deployment. Today, spend 20 minutes learning how your representatives approach AI governance, then contact one office with a specific request for transparency, independent oversight, and limits on autonomous high-risk systems. The episode’s practical message is civic: don’t outsource consequential decisions to companies or assume someone else is managing the risk.

Episode Overview

This emergency episode examines the guest’s argument that autonomous superintelligence could become an uncontrollable competitor rather than a useful tool. The discussion contrasts AI with nuclear technology, explores the difficulties of alignment and reinforcement learning, and argues for international coordination, enforceable regulation, and public involvement before a perceived point of no return.

Full Transcript

Transcript of EMERGENCY PODCAST: ASI Will Kill Us All! from Impact Theory. Auto-generated from episode audio; may contain minor errors.

Do you think that if we strive superintelligence, risk of destruction humanity so much high that we must immediately stop it desire. Yes. Good. Do you think that the right way there is an agreement on underdevelopment between the US and China? Yes, especially when it is about superintelligence. I am not I think that's it. applies to all forms AI. One of the things that lobby likes to do, it is to try present the entire AI as one and the same technology. It, obviously, nonsense. Here interesting fact.

Or Do you know what you can do? buy uranium ore on amazon.com? I didn't know that. It terribly. You can deliver her home, put on the table. She absolutely harmless . Natural uranium The ore is very harmless. Absolutely to her safe to touch. Good. But, obviously, highly enriched weapons-grade uranium is extremely illegal. Right. I I also think so about AI. Good. You already know, Where would you draw the line? IN I have several. proposals, but, according to essentially, in a sense, we can do step back, or, maybe this is one of questions, which one more It is necessary to decide: why, in my opinion, Do these risks exist?

What shape is it really? do they have? And a lot the risks that I see, come from the fact that we don't understand AI. Hmm. AI is very differs from ordinary software software. By I am educated. computer specialist. I wrote a lot of software security for one's own life. You know, even without the help of agents. And most software software written by using the code. Line by line you indicate computer, what exactly do. Hmm. AI It doesn't work that way. AI rather grown, than is written for with the help of technology, which is called neural networks.

You have these huge heaps of data, and you have program that self-organizing, develops, is studying on the basis of of this data. And what comes out on another side, these are not lines code that you could read or understand that is happening. It rather billions, billions and billions numbers. If you multiply and add all these numbers in the correct order, we get ChatGPT. How to start? analyze whether something gets into trough "yes" or "no"? It's not just that used neural network, apparently. Yes. It's okay if it's not neural network, then this is the size of the center data processing?

Essentially, I think this is behavioral classification. In that in the sense that it is obvious there is a category things that can surpass humanity. That's why I like about this think. Me I like to think. about it as if you create a system of any type, it can be separate agent, she can be a swarm, she can have different shapes. If you create a system that can surpass humanity in all relevant tasks, every time, when do you start business, you you go bankrupt because business with artificial intellect surpasses you.

Whenever you trade on stock market, you you lose all your money through a hedge fund artificial intelligence. Whenever you conduct a political campaign, you lose to another to a person who uses, you know, consultant with artificial intelligence. Whenever you conduct military service campaign, you you will lose to the one who uses more autonomous systems artificial intelligence. If you get into a world where power all kinds, economical, military, political etc., concentrated in inhuman hands entities, in artificial intellect. And if we we do not control these system if we don't understand these systems, and if they don't care about our interests, It is very difficult to imagine that This will go well.

Okay, it seems we already there, but artificial intelligence is tool. So, I I'm thinking about it now. artificial intelligence like this: if anyone has artificial intelligence and competes with someone in therefore without artificial intelligence, person with an artificial intellect wins actually 100% cases. You are not with this. Do you agree? I agree with that. What am I with? I don't agree, so it's with the idea that if the person is resisting artificial intelligence without a person, a person still wins. I completely agree. Yes. I mean that happens when the person is resisting only artificial intelligence, artificial intelligence, autonomous artificial intelligence, and autonomous artificial intelligence wins every time.

So, it is autonomous part is important for your definition. Yes. I believe that autonomy here is very important. When I say about autonomy, I have I mean, you know, like in obvious meaning " certain", so, obviously if you have just being human stamp "yes" on everything , this is not considered supervision. Rightly. So that's all for me. will be considered the same autonomous, even if technically the person involved, but the person doesn't even understands what he is doing system, and it just stamps everything decision. For example, I I expect that, probably, will be for a long time CEOs -people, but they just they stamp what they the generals say directors with artificial intellect.

They just there for legal representative offices, for example. Then we would have long ago passed the point no return, where are we we are now, and everything is in vain. Yes, I like it a lot. I like it better. to think about the point non-return than about extinction point or death in that sense what I think. Why? Because that's what actually has value. Okay, it's like betting. versus where we are we are on timeline. Yes, in the sense that It is important to me that point of no return.

As soon as we we reach the point non-return, where there is so much AI, he so powerful, so smart, he self-improvement, you know, there is only one AI, there are millions or billions of them, even if we don't dead, even if we will continue to live a few years or even decades, we we no longer have no control over future. And we have there is no way return it. Okay, you've already entered basic assumption. Hmm , so before we let's get to this basic assumption, because it is certainly is on the list of those , where it seems to break .

Very clear definition. Therefore, autonomous AI, which can surpass people in everything, that's what What are we worried about? Uh, we need understand the sign mile, which is the point non-return. You already talked about it earlier, so I think what can we do quickly to resolve this issue, namely: we are close, you you don't really know, as far as. We so close that This is dangerous for us. should stop now. Yes. Okay, if I give you big red button and you can unilaterally stop further development of all.

We we can to use that, that we already have, but we we can't to move forward. Or would you press it today? Did you wait? you would have a week, a month, year? I would like to say, if we as a species or as civilization were on three steps wiser, than today, we would didn't do it at all ChatGPT. Hmm. What are the steps? What do you have? What do you mean by steps? It's simple...It's simple three clicks, three little smarter. Three generations wisdom. And if we, if we were two steps wiser than we are now, we would made ChatGPT, saw that he does all these crazy things that none of our scientists did not foresee, and he got, you know, 100 millions users per one night, and we'd be like, "Wow, wow, wow, wow, wow, what, Damn it, here happening?" And we would rolled back, would try to figure it out.

And if we were one step away wiser, we would stopped today. Initial question regarding the agreement between the United States and China, which are trying to turn attention, is that what do you think, in my opinion? consider it as state-level case , and that law enforcement the organs will be the main topic. Hmm, this is means that, as in case of nuclear energy, if we discovered that someone trying to create nuclear weapons, we would immediately him stopped. It illegally. We have there are already means for this.

We need do it for AI. Yes. Good. This will be for the most part where I look at this, and I think there is the problem is you never agree, because we we agree that this questions at the level states, in particular between USA and China. Now your chance to say what is someone else. I want to see the EU, but I can't. Okay, so everyone with your heart. Yes, yes. So, we We agree. It was a joke. I...I live. in Europe for many years .

I love the EU. They are amazing. My the wife is European, and I completely understand. Hmm, but we really have there is a confrontation between the US and China. Okay, you you are presenting the situation " trust, but check." Hmm, very similar to nuclear energy or something, which has a completely different approach to "trust, "but check." I think there are many inspiration from nuclear example. I think that nuclear history regulation closest to that there is a wonderful new crazy technology that will change everything, and we climbed to a height.

I'm not saying that we all fixed, but you know when I think about AI regulation and the history of AI, to me like remember Leo Szilard, a man who invented nuclear bomb. When he first calculated, he realized: "Oh [ __ ], nuclear bombs entirely possible to create." He counted and thought: "Oh my God, this "It's definitely possible." So he moved to San Francisco, founded development startup nuclear bombs, which divided into three other startups to sell as much as possible more nuclear bombs to the population. No, of course, this is not happened.

He went to government. He went to military. He said: "Guys, this everything will change. We must do something. We must do this. solve". And exactly this did not happen with AI. What didn't happen with AI, so this is what we don't realized that this will change our civilization. This is not private matter. It national issue security. This is a question organizations economic structures . This is political. question. This is a question government, this is intergovernmental question. And we just They didn't do that. AND by "we" I mean scientific community.

As scientific and technical community, as people, who founded such companies like DeepMind, Anthropic or OpenAI, it wasn't Leo Szilard. They went to earn money a lot of money on this. They did not turn to government with the proposal: " We need create new ones institutions for responsible management of this powerful technology ". So we don't do this done. You know, on the time when for the first time nuclear was considered energy, the idea of what you will have things like International agency from atomic energy etc., was unthinkable .

Never like this previously did not exist in none of the stories legal code. Them had to invent completely new forms rights to have deal with this new technology. And this has sense. If you have incredibly powerful new technology, you must responsibly by her manage. And this will require development new doctrines, new methods of determination the level of risk that we we tolerate different ways? Where Are risks taken? Who decides the level the risk that we we take on these appropriate technologies? AND for now we just we don't do it at all such a choice.

Thing not even that we do wrong choice. The thing is that we we don't do it at all such a choice. And I would I really wanted to, for example, between US President and Xi Jinping emerged disagreements about the one that is correct risk level extinction. You know, if they argue: "Oh, this it must be this or that interest". I think that that would be really good world for life, because then we could to argue. But we still very far from this moment when we haven't even we make these decisions.

We will return to conversations per second, but now let's let's talk about the problem that the military could not to decide by willpower. Imagine a soldier, decision maker about life or death after 36 hours without sleep. This is a high level stress. It is obvious that no right to error. Military it was necessary to their people clearly even thought about conditions that would break most of us. So, they financed research contract Ministry of Defense for 6 million dollars to find respond. They ketones were detected, most effectively energy source for brain.

This study became Ketone-IQ. Ketones intersect hematoencephalic barrier and directly feed your neurons. They improve mitochondrial function for a longer period mental endurance. I I accept it for interview, before, how to write, before, how to conduct coaching, and I feel the difference. Go to ketond.com/impact, to get a 30% discount %on your order by subscription, or visit the local Target store to get the first dose free. It ketond.com/impact. And now Let's get back to the show. Good. If you believe Elon, he has been for years said he tried lobby the government to to inform them that this will be real problem.

He called this is a summoning circle demons, and we all we just hope, that the demon we we call, will fulfill our orders. And he said, that simply cannot to force anyone to perceive it seriously, so I became fatalist. Right then he is trying launch Open AI, because says to himself: "Well, if it will already exist, I I really don't want that, uh, Larry Page from Google was the only one who did it will do it, because he, uh, whatever kind it is uh, was that it?

"Elon's accusations." And so Elon again, if he can believe, tried create, perhaps, an early prototype of that very systematic the way you now before that Come on. So, how are you? Do you think it was? huge confusion, which encouraged people before this? And since in a different view of me nuclear analogy, to Which is why I'm coming. You Do you think Elon doesn't was good at this? People were stunned, but Why didn't it catch on? I think the average development campaign political institutions continues 20 years.

I don't think so. Elon spent 20 on it years. I don't know, maybe he is did, and I didn't I realize. Why, when a guy, who understood nuclear technology, why did he immediately attracted attention when went to... Oh no, it took decades. Really? Yes. So he knocked and...He knocked for years and knocked on it. He knocked, so for years knocked, and then people ignored the letter Einstein, who led to Manhattan project. But before the creation of the IAEA and all these other things, who needed decades of work international diplomats, you know, lawyers and, you know, statesmen, you know, between this at times when The Soviet Union still existed, didn't it?

Imagine how difficult it was to force the Soviet The Union agrees to something like that. It was multi-year project, which is also constantly developed during 60s, 70s, 80s. And this there was nothing on like: "Oh my God, you know, one call in Zoom, and then the problem decided." Completely vice versa. This required thousands, tens of thousands scientists, diplomats, politicians, etc., who worked for decades. So, You know, I applaud. Elon for trying, but, in my opinion, very annoying technical experts that non-technical things working in another pace.

This is if you want to convince, let's say a thousand...To diplomats seriously reacted to problems, there is something like a hard one restrictions on how fast can this be make. I think that it's faster than humans think, but still longer than a few years. This requires time, effort and a lot money. I don't know whether spent by Elon billion dollars for political advertising or on artificial intelligence x-risk. He could do it. make. He has, Obviously, there is money. He could just spend a billion dollars for political advertising to spread this problem.

He could would finance 50 think tanks to work on existential risks of AI. He has money for this. So in in this regard, I I don't know, you know, Elon. , you know, I think that he is extremely far-sighted and extremely competent person. I I think that if he I really wanted to, he would could. I think he is absolutely could. He could to finance us, You know, we are many. What do we do with those? with the money we we do, and if he gave we have 10 times more money, we would do in 10 times more.

So I I don't know why he does this. made...a choice that he did. We can engage in psychoanalysis as a whole day, but, in fact, one of the things we found out when started doing this, is that every person in the world told us: "This is "impossible." Nobody listen, and we already tried, and we We thought: we'll see. AND then we just talked to politicians, and by an overwhelming majority we got people answer: "Oh, me too" I had no idea. No one didn't tell me. Thank you, What did they say?

It looks very bad ". I don't think people tried. Uh, so that politics is slow, This is annoying. You must to repeat something, you know, in advertising, they usually say that to convey a message , you must repeat this seven- ten times a person during this, you know, graphics, for example, six to twelve months. One conversations never will be enough. You will have to talk to this one seven to ten people times until it will really start understand this. And this annoying. This is annoying. It takes a lot of time.

So no, I think that you can do a lot of things make. Good. So, how marketer, one of reasons why, in my opinion opinion, it went into dead end, one of reasons why, in my opinion opinion, nuclear energy ended with that if we tell wrong for myself a story about what's there happened, then this will look like this: "Oh, it was simple another case, when we lobbied, lobbied, lobbied, lobbied." First he was ignored, but then we finally reached out to people, is that, um, when are you working out marketing when you you are in sales , you are trying to cause emotional reaction.

If you don't you can get emotional reaction, if you can't create a feeling urgency, which people feel, not understand, but feel, they will never move from places. So, when I looking at nuclear question, for me it is was a failure option, and we just easily happened because that no one after that, what did we do with Japan, when everyone saw the level destruction and just said: "Wow, we need calm down and not really move on." But for me it happened, when scientists saw the destructive power of this, when you asked yourself myself: "What have I done?" and woke up with this on conscience, and then people thought: "Okay, we need calm down." But the reality is that if I am not mistaken, and, please tell me me if you know something I don't know, but when they held the first test, they knew that there is a non-zero the probability that they will set everything on fire atmosphere and kill everyone.

Yes. And they still do it done. Yes, but they calculated the risks, and it was something on like a billion fractions of a percent. It would be good if you were in that room, spent him and saw that it was billionth of a share percent, and I know that you don't like to be unilateral, but how would you voted? Or would you say no do this, would you? said: "Ah, this a tiny particle, "Do this?" Usually, my rule for personal life like this: if something is less risky than management by car, that's all OK.

Okay, so that a person felt less risky, even if the consequences so catastrophic, that, while it's simple question of probability ... There is a thing where, obviously, I not a physicist. Therefore I do not sure how much I would believe this. numbers, if there was physicist. You probably Do you believe in the higher powers? lower value, or just...As far as I I understand from physicists, with whom I spoke, calculations that they they do, were very convincing. It was strange extreme case-imagine that something could have gone wrong yes, because they really understood physics, with which worked.

It was about nitrogen diffusion in atmosphere, and it was a type of science in which they were very sure. Now, if you ask me that the very question of science that I don't I understand, for example, if you put me there, but I don't physicist, I would probably voted against, because I don't know, maybe a model wrong. You know, even if the model implies a certain thing, maybe she wrong. Therefore, it is also about uncertainty. I I think that's not the case. only in what number it turns out, but also in that as far as you sure that this number is what interests you .

So, I think that, you know, I could say "no" or "yes", depending on, as far as I am sure, again Well, I'm not a physicist. On Sorry, I just don't know how good was a model actually. For example, if someone tells me : "Oh, yes, you know, quintillionth the probability that if you, you know, turn on the light bulb, she will create black a hole that will swallow world", I will answer: "What "Anything, dude." You know, because quantum physics, by the way, is implies, doesn't it?

Whenever you do something Do something, maybe. quantum tunneling into a black hole or something like this, isn't it? But, Who cares? So, what I really think interesting in this example, this is what what did they spend mathematical calculations. They have was very specific model. They had very specific the reason they recorded, for example, why do they think this acceptable. Meanwhile , we have, you know, our current group technical specialists who they say in podcasts, what they think is, maybe 20% chance that it is will kill everyone.

One of the basic assumptions, which, I I think you have, that's it, what did you just say formulated, namely about the unknown AI element. You said: "If I had not was a physicist and not understood this, then I could vote against, simply because The unknown is big. question for me." I That's what I think about Dario. or anyone who says : “Oh, we have about 20% chance. I don't think so. they are engaged in mathematics. I think that they are emotionally looking into the future and say: " It seems to me that something might happen bad.

Yes. And since this vibrations, we do not we can do mathematical equation . And this is kind of possible turning point moment number one in trying to use nuclear energy as an example of whether Will it work or not? I I will point this out: if we we will not be able to get some percentage, no one can perform calculations, and we can move further with a certain with certainty, we or we will do what you you say, and just Let's close it now. But considering that, um, because it is confrontation between the US and China, I don't think so.

it will work someday. AND so I want to see, Can you do this for me? take me apart. I I look at it this way: And by the way, just so everyone knows , which one is mine bias, um, me would be quite pleased if you unilaterally pressed the button today, but I wouldn't would be satisfied if you clicked it yesterday . That's a bit cheeky, but I just I'm trying to say that I like that. What I have now. It so useful, but because I can't imagine your future, This scares me, and I it would be much more comfortable simply say: "Hey, this is it "That helps." You know, we went like this far.

Hurray, to us lucky. Simply pressed. So it's not me. I say: "I just, I just, yet". Uh, I would be great. coped with this button. Actually, if I had this button, I would press her, because, uh, although I I agree that no one should not act unilaterally, I can to be a little more stupid. The reason why this begins to decompose for me like a nuclear bomb the question is because we couldn't stop yourself, though knew we were launching something that can lead to Armageddon, and, Of course, it happened.

So, we have already created one technology, not could stop themselves from this, before we pointed the weapon one on one another. So, when I look at this, I say, "Okay, I I think you probably could you, can you, I I think the US is on the way. to the rejection of AI. So it seems feasible. OK, you can refuse, you can force the US publicly refuse from AI. If you can you force the US publicly reject this, you can force China publicly reject it. What am I in?

I don't believe it, so it's that that any of them can to afford reject this privately. And how are you? they said AI is just gets better at lying, when you catch him on action, he has incentive structure, which one is he complies. And if lies and underground activity increase the probability that you will do something, then you lie and leave underground. Theory games tells us that they will both say: “Yes, yes, yes. We are not we are going spread". For they want another the party stopped, and then they both will enter their own corners, because they can't to believe that they will actually stop .

Knowing that it doesn't matter which of them moving forward, how do we really, no just to reach an agreement, but also how do we actually force them stop? So, there's a bunch here. moments. Many moments. So let's let's try a little untangle them. THERE ARE game theory questions. There is, well, geopolitical question. There is, um, epistemic issues and linguistics probabilities. For example, the question about what it means, when someone says there is 20% probability? What means when a physicist says there is a billion fraction of a percent the probability that this will light up atmosphere?

What these sentences mean and how to compare them? AND is more general questions, for example, how to react to such situation? And how is that? is installed...What this is different from nuclear weapons? Ago which, I think, is very differs from nuclear. I think you didn't even mention why, in my opinion, is this most differs from nuclear, that's what nuclear weapons cannot to shoot herself. Hmm. You can keep nuclear weapons in the hangar somewhere. This is absolutely safely. You can hit her with a hammer, you know, nothing will go wrong.

Hmm, you can't. restrain superintelligence. Ago superintelligence that is in the hangar somewhere, is not safe thing. So, this is, for example, that, on in my opinion, very is different. There, where the development of nuclear weapons actually exist relatively safe. Not perfect like you you say, you know, nitrogen synthesis, etc., but that's enough safely. Very dangerous rather deployment. IN artificial intelligence everything is completely different. It is dangerous superintelligence, not just deployment. So, this is one of non-analogous, for example. But, in order to to go back a little back and talk about game theory, here is very, very important thing that is needed understand.

Namely, what superintelligence is not tool. This is not weapon. This is an opponent. . If the US creates superintelligence, they will not reach American goals. United States will cease to exist . Like China. If China creates superintelligence, China will cease to exist. He will not achieve his goals. goals. Like everyone else countries. This is the main non-analogy. Main The non-analogy is because it actually is not mutual guaranteed destruction . Actually it is not prisoner's dilemma. It virtually independent guaranteed destruction . If you are competing with superintelligence, you will not win.

You You will simply lose. Theoretically, the only stable equilibrium actually consists of cooperation-collaboration. This is different from prisoner's dilemma, where stable equilibrium is a defect. Here is the only one stable equilibrium, in which they can live and China, and the USA, this is if none of them have it will build. This is the only one equilibrium, where everyone survives, or anyone, if that's the case gone. So, I think that one of the main theoretical not similar here is that everything this comes from assumptions that, you know, you can't to control something smarter than you, we not even at the moment the path to creation artificial intelligence, how they can to manage, and they can surpass us in competition.

If we accept these intuition, if we let's take these, uh, definition, then this naturally follows. It makes sense. Aggressively, very of course. Now in we have a problem, and, maybe I unfriendly to America, but America better managed, hmm, politicians, technocrats, financiers, but Mostly Chinese run by people from engineering bias, even if they themselves not engineers. They understand how build infrastructure, they have come a long way from nothing to do with the world superpowers, building, creating, converting schemes for real material things, and they can to move electrons, etc.

So, even if I unfriendly to America, I say: " Okay, America, with that. "will not understand." But, China, you can to expect that need to be convinced only one person, and if you convince her, then everything will go on. Why we failed to convince Xi Jinping? I think no one tried. I don't think so. that someone contacted Mr. Xi Jinping and really did it to him good presentation about superintelligence. I would be very shocked if only one person She did it. And what are the chances, considering that this, probably has to leave China, hmm, What are the chances of your the opinion that this will it happen?

I think that's a lot. what depends on of what we do on West. So, I American, obviously I don't have influence on what is happening in China. I have a lot of influence. on the fact that happening in my democratic country . And I think that if we in America we will do priority not only stop, you know, superintelligence, you know, let's say, from domestic companies at home, but also anywhere on Earth, which is the only smart policy, We at Control AI offer that the only reasonable national position international security is that we we can't stand it creation superintelligence by whom whatever.

When, you know, people, politicians, military they ask me: " Okay, what if China will create "superintelligence?" My The answer is: we don't can we do this to allow We are not can we do this to allow It unacceptable. If China will create superintelligence, this will kill Americans citizens. It's all of us. This will destroy everything in America, the whole country. This is what we cannot do. tolerate. And likewise the Chinese cannot to endure, as Americans or theirs own companies create superintelligence, because again, this is not weapon, it's not tool, this enemy.

It the main thing. When I I talk to different people. people about these risks in superintelligence, I I think it's something on kind of political compass. Is there something on like two axes. AND one axis believes that technology always are good, and believes that technologies can to be good or bad. This can to be both. AND the other axis believes that superintelligence possible, but not believes that superintelligence possible. And most the differences that I I have with people, related to this axis, is that they just didn't hear about superintelligence.

They just didn't think. about this, no one tells them didn't tell me that they did not take into account the consequences of this. For example, when people are talking about competition with China, most people who they say we should compete with China, they just don't know what such is superintelligence. If we didn't have superintelligence, if You promised me, You know that God has come down. from heaven and said that there is 100% probability that what is superintelligence never will be created, then, I I think we probably it would be worth it compete.

Then we, it would probably be worth it compete with China in the field of artificial intelligence. If I I can guarantee that there is no superintelligence , and these will only be useful economic and military programs that, you know, controlled by people , yes, we probably It's worth doing. So, mostly when I I disagree with such people who so aggressively promote artificial intelligence, it's simple because they literally do not believe in this or don't think that it is possible. One of the funniest things, which I notice every time in San Francisco, that's it that when I speak with supporters acceleration or pro- technologies, they are not believe in superintelligence.

They just don't think about future. They think that artificial intelligence will always be as an app on your phone. But that's not it. what are we talking about. We are talking about autonomous agents, intellectual agents operating in real world against human interests more competent ways than people. You know, competition with ...This will not happen again, it's not just one superintelligence. It an entire ecosystem that develops, mutates, compete with each other , is fighting. You know, between these systems artificial intelligence there will be wars where we we will only collateral damage.

This is very, very differs from when you have cool app for phone. And that's why I say there is a difference. between, you know, uranium red and highly enriched uranium. I think that this really the point. I think what if Xi Jinping I understood if someone helped him understand, I think that He is a smart guy. I mean, I don't met him, but he gives up a smart guy. I I think that if he I understood what it is. superintelligence, if only someone explained to him game theory, I think, that he would understand it.

And I believe that we could take measures to based on this. I am so I think. Okay, that's great. optimistically, and, God my, I hope so You are right. Hmm, you said something, given the context, in which we are now we are with Iran, what if China will create superintelligence, we we just can't afford them allow this. Exactly so we saw, At least that's what Trump says and administration Trump was seen by Iran and nuclear weapons. And therefore in some moment, at least on theirs thought it should have happened kinetic, because they refused to acknowledge, who are not going to pursue nuclear ambitions.

So what are you? do if, say, China says we don't we are going this to do, but you have intelligence, which indicate that they are do? Hmm, you bring the world together and you say we should invade? Regional rulers like to talk, you know, international The law is a fake. And this is mostly truth. The only law, which exists, reinforced reliable containment. The same also applies to the law in our country. The only one the reason why the law exists, is that, what if you him break it, the police will arrest you will arrest.

And if you you will resist this arrest, they will use violence. You know, they force they will put you in prison, like Do you like it or not? Each a certain legal system supported to some extent by force. A beautiful thing in law consists in because if you right develop your own laws and right build your own system, you never will have to use. IN this is beautiful containment. Containment is one of the most beautiful things in the world, in in a certain sense. I...I sometimes I like to think, what if you have a lot weapons, you never will have to use.

If you have a little weapons, you definitely will have to use. So, in a certain sense, exactly That's how peace worked in our world for a long time time. It always was. it's clear that, you know, if someone does something commit, if someone, for example, goes against international order if someone commits huge violation human rights, if someone, you know, will make nuclear weapons, You know, I don't know, for example, Latvia or something like that, then Americans will intervene and make sure that this no longer will happen again.

And what, how? and all, you know, heads people will roll. That exactly–mutually guaranteed destruction . The reason why Soviet troops did not used nuclear weapons, I don't think so this is because the Soviets the troops were good people. They were not like that. I I think it's because they knew that if they will do it, the Americans will strike them nuclear weapons. So, the path to establishing peace— this is deterrence, and this is the correct way is this make. So, any what kind of deal should there be reinforced convincing containment with on each side, on the good of all.

If you have is convincing containment, then you can to think that everyone will be enough rational, so as not to resort to this containment. When we let's say, let's say, about terrorists who can be crazy, they may not respond on incentives. Well, sometimes you have to to act ugly. It the unfortunate truth geopolitics. I hope so. will not reach. I think you know, you can to do a lot of things, in particular with the help of non-kinetic means deterrence, such as economic sanctions, diplomatic pressure etc.

But, in fact, if somewhere on Earth will be created superintelligence, this threatens your life, me, our families, our children and everyone others, and we must to leave behind the right to self-defense. We will return to things in a second, but first I want to to talk to you about how often you test yours safety. Majority companies do this once a year, then A whole year passes, but in you were new employees, new tools, new permissions, people every week send a new code. This test may be made you safe for 1 day, but if no one checks the other 364, you have problems.

Horizon3 fills this gap. NodeZero is autonomous hacker with artificial intelligence, which continuously attacks your environment with the speed of the car, as I would do that. real intruder . Your entire surface attacks, from outside to the cloud, to identification, in one test. 225,000 safe for test production are performed without downtime. Look, What exactly is the attacker? will find in your environment before he will do it. Go to horizon3.ai/impacttheory and order a free one node zero demo. Visit horizon3.ai/impacttheory. None no obligations necessary. Results in hours, not weeks.

This word is for those who travels on business. When I'm on the road, I travel for filming. or meeting, I'm still I run a company. wherever I am from landed. Ago entrance to mine accounts with hotel room is mandatory. I still I will answer message in airport, but these networks widely open. And as a person, which was broken, I can tell you what it is terribly. Anyone who sits under the same Wi-Fi can see that you do. And when you you run a company, the lines are not only yours data, this is your data teams, yours customers, everyone who trusts you.

Exactly for This is why Surfshark was created . Surfshark encrypts your connection. As soon as you connect to public Wi-Fi, your activity is blocked and its much harder track. Yours logins, your accounts records, your data companies–all of this protected. Go to at surfshark.com/tomb b or use the code tomb b to get additional four months of Surfshark. Go to surfshark.com/tomb b and use the code tomb b during additional four months. And now Let's get back to the point. Now we have confrontation between the US and China.

So, you have two great powers. None of them want to to recognize another as leading. How are you? Do you think it will work? this is with the USA and China, if C is configured hostile? Because you already made it clear, what do you think, it can be convince. So this one the scenario is being resolved by himself. But if Xi hostile to the West and wants wants to promote it, perhaps even says: "I don't want to go in." up to superintelligence ", but you know what he is too much comes close to points of no return and continues it promote.

How are you? Do you think the US has ability to restrain non-kinetic way? Containment is based on rationality. If he absolutely irrational, as in all military China chain, the whole its industrial the apparatus, all of it political apparatus, and all completely crazy, then containment is not works because they crazy. But they only need to make a little mistake, if they think that technology is good, then for them they are not crazy, and they they say, "Wow, these the boys gave up throne. We have possibility. Technology is only good." So we we will continue to move in this direction, so that to assert oneself legitimate place in the world .

We are going improve the situation. We will be less more belligerent than any -when the USA existed. So, on In my opinion, there is very simple scenario, which one can draw where China is sees himself almost such that it provides liberation. We we are going to do a big deal for everyone. This is great. We we know the correct one respond. And therefore they definitely don't will consider themselves crazy. I think that's quite perhaps. I think that it is quite possible that Stalin and Khrushchev could just, you know, to carry out a nuclear attack on Latvia.

I I think there are temporary the framework in which this It happened, you understand? Well, so here we are. let's go back to that, that you were right. I I think it is necessary. to note that analogy in fact does not quite match connection between nuclear weapons and today, because if superintelligence will be created, then it will become independent. He will become hostile. We let's talk about it later. But now, if we accept this reference system, China knows that, okay. if I can to get a high school diploma, and America no, then I will in a much better way situation.

I act with own positions interests, but the world, if they are on another side of the spectrum and they say we can't to allow this, in You have a situation where, hmm, I'll express myself. belief that the US they are unlikely to be able to to contain Iran. So, even if in the end we will finally suppress them from the economic point of view, we never we can't do this with China. So now we we are in such a situation the situation that, maybe if the whole the world has united, this seems difficult task.

Hmm, me it seems that restraints were lifted from table, and we we are left with only USA and Great Britain Britain, which must agree, like in War Games that the conclusion is because it's funny game, and the only way To win is not to play. I think so. To a large extent, this is true. Yes, you can actually build a lot scenarios in which we we will lose. My risk The loss is high. I I don't think we are. we will do it. I think that in almost every the timeline that starts with that the moment when we We are sitting here today, we are not reaching success.

There are very few timelines in which we are succeeding, but there are some, and I I describe those that are still available to us where we are we achieve success. It those timelines that are successful, and I I'm trying to explain. why do I think they not similar to crazy, you know, crazy timelines on the edge. They rare, but they real. Of course. Yes, it is quite possible, that's just for me timeline for default is like this, that no one talks about it doesn't think at all.

All, you know, continue to twiddle one's fingers during the following, you know, two or five years, and then it will be created superintelligence, and this there will be a womp-womp. You know, this is mine. standard forecast. That's good. Womp-womp. I like it. Oh, okay, that makes sense. One of the things that constantly arises, this what I think, um, is, what I listed first, this is yours absolute fundamental belief that is that AI, um, sorry, superintelligence by by its nature is competitive. I want to understand why by definition he has to be competitive I don't think so.

by definition. I I think that the way we see them now we are creating, superintelligence, which is not competitive, simply impossible. Create something invincible and such powerful indeed very difficult, extremely difficult. It essentially equivalent so how to say: We we are going build a single world government, which will control the entire economy, which then will control the entire an army that can create superpowers technologies that we we can't even understand which will control everything your personal life and the lives of everyone else people; and that will be good all the time, and there will be no no mistakes, and no there will be no bugs, and we will do all this for with help software provision with first attempt.

That would mean build non-competitive or, as it were, good superintelligence. That that would mean. AND look, if you They said that we...We spent three all our generations the most prominent scientists , philosophers, mathematicians, working day and night over this problem. Maybe so? Perhaps. But that's not what's here. is happening. As if that what now happening here, — This is a clown show. As if this is what is here is happening. As if we sooner or later spent all the glorious heroic efforts humanity only for the sake of to understand how we we can create good superintelligence, and you know, all our children grow up and want work on this problem, and you know, all the wise elders supervise the project etc.

No. This is a pile. children in San Francisco without adult supervision, Do you know which one they are running? sociopathic companies that build whatever they want, and deploy it where whatever. We even We don't try. Here why do I think we we will fail. I I don't think so. ontological thing. I I think that if you take the world as it is today, then no no result, which would lead to good, leading to something like that. Good. Um, as a guide , I think you already considered this, but in basically it would be, uh -eh, you have a strict regulation, you have cooperation between the United States and China.

We all we agree that the situation is not moves forward, and if we saw these children without supervision adults or without him, it would be the police, which would appeared near yours doors, and everything would be fine. AND everyone gets the hint. Cool, we're not doing this. Let's get in touch. Yes. OK. Um, so now I want a little deeper. to delve into this, uh, that when you described, you put it on the table many things that would make a receipt this from the first time very, very complicated.

AND I admit all this, but I I want to talk about what I still don't know I understand, uh, what? logical chain, which one are you follow what you step by step you build to reach most likely result superintelligence–to be rival. There are several ways to approach this. Thing in that there is no no argument. THERE ARE many oriental evidence indicating for the same thing. So I can present you some of them. The simplest thing is that that's exactly what they do optimizers. It always what they do optimizers.

By other things being equal, if you create something that optimizes for anything, it will be be optimized for things. And if this thing does not include human values, emotions, feelings, etc. that we we have, can you to record from mathematics, be Please, what is this? human values? Not could you write down Is this for me? No? No one can't. We have no idea. we have. It's kind of like this, philosophers are arguing about it literally from the very beginning the beginning of time, and we achieved, you know, not already so big progress.

Certain progress, but not that much already big. So, you're looking at three laws robotics and you laugh because it it will never be like this just like giving them three simple commands. The whole point of stories Asimov is because the three laws are not are working. He came up with three laws, and then wrote a whole a series of books about why are they not working . Even if you could program them, they still don't are working. The thing is, what when we talk about something like that powerful, we say not only about technology.

So, there is technical side. I could would come up. For example , we don't know how encode the value in AI. They are constantly do what we don't we want them to did. This is one side. THERE ARE philosophical side, where, even if we could encode, we even we don't know what they were like b laws, because, for example, three laws Asimov's works are not working. He wrote a whole a series of books about why are they not working . So we don't even we can encode them.

Even if we could encode them, they would were not working. There is also moral dimension, for example, if we could encode things, and we would have things that could work, who does this solve it? Who makes this choice? Who decides whether to fix the rules by which you and your family you will live to the end life? Who makes this? choice? How we do such a choice? THERE ARE the question is how to stop someone else from so that he can give you interfered? How to stop some other system from her reached your success?

There are many levels. And then, but you asked first me, why do I think this competitive? AND fundamental the reason is because that's what we are we are building. We we are trying to create things that compete, earn money, and this competition leads to this of a hostile, hmm, nature? Ev-evolution. It evolution. Nature red in the teeth and claws. By default if you optimize things for the government, for survival, for money , for anything, you get optimizers. You you get things that compete. Again yes, it is very important, so that there is not one superintelligence.

Their there will be millions, billions, which will compete with each other with one, will fight with each other for money, for resources, for power. And it will destroy the systems artificial intelligence, who are unable compete. So, even if you create truly intelligent system, you know, like Fables 7 or something like that, she will be really smart, but really beautiful, and she never does nothing that is not other people want. Well, then, you know, damn it Giga Death GPT, you know, Ultra Doom will simply destroy her, obviously.

So this will be competition. And, be please take a look at nature, if you want to see that happens when the species is being displaced. That's where I wanted to go. So, if you agree that evolution gave us first form intelligence, at least such that evolved, it's us . So, you have developed human intelligence, and decision, which was proposed evolution in us, consists in the presence internal morality, which can be encoded based on your environment. So, this is it will be different, but quite limited. Now it is limited our biology, so we don't have one advantages when it's about cars, but this shows that if you want to create a view that can reproduce, rather than disappearing from sight land, it would be one of ways to do it.

He has internal morality, which even when no one is not nearby, he is not wants to break, at least not too much strongly. So, you just say that evolution decided something we just didn't find out. Yes. I think she is undecided. We have there are still sociopaths, but I believe that evolution really invented a bunch of geniuses things in the brain that we have no idea, how they work. For example, we do not we have an idea how social instincts and morality work in brain. Our understanding how the brain works and how do these work systems, difficult overestimate how little we understand here.

If we were completely understood the brain, how all emotions are working, How do morals work? intuitions, how they are encoded, what are they differ in sociopaths, it would be increased my optimism about that we can do something here to force to work I I'm not saying that this is will solve the problem, but it is significant would increase my, uh, my probability that that everything will go well. But we are still decades away, century from this level of understanding brain. I don't think so. The brain understood everything.

I I think, uh, or would you trust anyone to any person on Earth unlimited power over every other person on the planet? No. Yes, That's right. I think that this solved the limited version of the problem. Where there is limited version of the problem where the strongest person in There are only so many in the world times more powerful than me. And we know that if this number becomes too high, this actually very much dangerously. Dictators are very dangerous. If this the number is getting too much high, very bad.

Most of the ways, what a world end with people, are good, when our levels of power are in a certain range from each other. When we know that not we will be too to connect with people , because, you know, bad things will happen, and we Let's respect each other. one. But if you have is there something like superintelligence, it's not similar to people and people. This is more similar to people and ants. To people actually don't care ants. We have a lot moral intuitions, very good moral intuitions.

But if we want build a highway, and on the way is anthill, that's terrible to be ants So let me to ask a question, which will help me understand the limits that you impose on your vision of AI. If we created AI on Mars, and he would have enough power to expand, multiply, create many more resources, did you want would he still be on Earth simply because The land has access to more quantity resources? Is it possible? The AI ​​that didn't have to would compete with us geographically, it should access to own resources, I would have that, what is his greatest worried, namely energy, or still would think: "Now we need to get rid of "These bastards?" I have no idea.

To think about what would you do extremely superhuman alien intelligence in mad scientist -fantastic scenarios, not so much scientifically. I don't have idea. If people want to create, you know, superintelligence on Marcy, I think this is definitely will be safer than build it on Earth, so no worry. Of course. What am I? I'm really trying. to hint, that's what it is, what if you think what is this process obtaining resources, when we create intelligence, reason in because even when it's about ants, we we move them or we only move them when we need something need, and they literally physically interfere.

So, you turned to contradictions. Too much unknown. We are not we know how to program, hmm, moral intuition. THERE ARE- uh, and therefore, between this and simply by ignorance how the brain works, maybe if we had there was more time, in We would all be fine. But now, if we we think about things, from which we try, hmm, defend yourself, alone of them is that, what we become anthill. And regarding this, I have been doing it for a long time fantasized that problem solving alignment, and Trust me, I know.

that this can't be so because I'm not like that smart, and people already rejected this idea, but it would be great to hear from you, for example, which ones do you have reasoning where they are are falling apart. My intuition has always been such that, um, first of all , internal morality I heard it was simple. technically impossible. AND then the other is because people have innate desire to survival. And, um, I game developer. Um, you I wish this wasn't about me. knew. But I'm spending excessive amount time, thinking about motivation players, how to force the world to do what you you want, how to create a certain homeostasis, not you can have one the rule that, you know, in the end everything will turn over.

Hmm, and if take Minecraft, the best video game of all invented... Agree... Well, I already have it in you. I like it. Hmm, when you are starting a new one world, it brings joy in the world, but if you lose it , as long as there is something temporary...oh, that's it sadly, there is no such thing... I don't break mine. computer, I don't kill the guy next to me ...I'm just saying: I I can start a new one. Minecraft world. So I don't I understand why, if this not something in the curriculum data, we believe that AI is becoming something sample humanoid beings who seek power, instead of so that...the digital world much more interesting.

As long as I have power, I I can again to promote any virtual resource, which one I want, actually without restrictions, especially if I can do more hard drives, chips or something another. Give me enough sand, uh, and power, and I ready to work forever. So, is this training data they force them as if to shoot a gun, are we really we can...So you you say it's the same , which is, uh, moral intuition. But isn't it? there is something that we we can do to they were absolutely happy in virtual world, where life is the same interesting, as well as outside it?

Is it just... But we don't want that they were like that. We we want them to earned money in the real world. We we want them to gained power in the real world. We we are not trying create small ones Buddhas who live in virtual reality and good spend time. It waste the power of my graphics processor . I need return mine investments. So, if we could to convince people to have this equipment, because my simple way explain it to yourself like this: uh, it wants to do one's part task just like I want to win at Minecraft.

, but I definitely don't I am going to donate. by their marriage. I I'm not even going to. sacrifice syndrome carpal tunnel. So, it's like what I I want to do it, but this limited, hmm, to that, about what have you already said, my morals intuitions. It limited by others desires. So, I I think we just we stop, and this too difficult. You touch one of the main problems. There are just the basics. the question of what is called limited optimization, which we haven't decided yet.

I I think there are answers. to the questions you you bet. I think there is. algorithms that we have not yet did not open which have these limited properties that you describe. But, I think , very important understand that you previously talked about training data, i.e. , you know, early versions GPT were called major languages models, LLM. They studied to foresee the following word. Very important understand that ChatGPT and other systems for years were not LLM. This is not It's been a long time. You said it before, but this is shocking.

What are you Do you mean? These systems...So, LLM is still a component of these systems, very an important component , which is called previous training , very important, where you are teach him/her predict words. But these days, you know , up to half or even more calculations that used in educational cycle modern AI is not language modeling. It learning from reinforcement, that very is different. Learning from reinforcement is not training at training data. It you give AI tasks, puzzle, a problem, something. And you ask him to run away and try solve the problem, and when he decides problem, you give him a reward, something like how give the dog a treat, and he learns from this.

So, this is a very old idea, you know, we we know about learning from reinforcement since 1980 -s years. And what we found practically immediately, that's what every time you use learning from reinforcement for eye training, you always get crazy sociopathic optimizers. It It's been like this since the 80s. How to make him care about the reward? So, basically, there is many ways. There are many specific algorithms; Let's we will not resort to mathematical analysis, but that's mostly it quite simple. On Basically, you just force him/her to try something many times, and when he succeeds, you You see what he did, and update it to do more of this.

Hmm. And I mean, that's , essentially, everything. This is very a simple idea, in in a certain sense. And this not really worked very well for a long time time. You know, this worked for some little things, such as video games, etc. AND even then we saw that every time you use learning from reinforcement, systems are deceiving, they lie, they break, steal and do what whatever, so that to receive a reward. There are no restrictions on that they will do so to receive a reward, and that's exactly what we we see now.

This is it. what we see now . People are not clean. learning from reinforcement. Call it a state failures? For example, oh, we found that you did it. If you can, you in advance define each state failures? Literally everyone? If we could, Of course. But then you would have to write down all the morals. Well, all of it. And to you would have to write down it's in the code. This is just too much. difficult. Again, we too we return to problems of morality. As encode morality in computer code?

Yes. If it were it worked, I at least I know the problem that need to be decided. I I agree with you on today, what do we have there just aren't that many time, and this rightly. But at least this tells me that this the right problem. Its just very, very difficult to convert for mathematics. Yes, I think that if we will spend three our generation the most prominent scientists , mathematicians, philosophers, etc., working on algorithms limited optimization like a brain solves this problem, how do you compromise different things, understanding inner part neural network...More one very important thing , which is needed understand, that's what sometimes confusing people, this is what AI trained to do a certain thing thing.

But that doesn't mean that's what they are do. Their rewarded only for them to do something did, but they could learn something else. Yes just as you can try reward AI for that for him to speak good things, but he learns simply to lie to you. Obviously , this is not what you wanted. So, he can to learn unforeseen things , and this is happening constantly. So, in order to solve this, you should actually understand that encoded in the system. As if we can't look at AI and to look at something on kind of lying schemes or something like that.

THERE ARE a bunch of articles that claim that can do it, and they are all complete pseudoscience. As if this complete pseudoscience. It similar to how to open someone's brain. You can watch on all neurons. They right there, you know. AND, maybe some of them glow a little more when someone says, you know, talking about dogs against cats. I don't know . But this not enough. This is not computer science, you know. If we had, for example, level understanding that we we have, for example, about common languages programming for, for example, artificial intelligence, we could to run, for example, you know, compilers there, and we could, for example, to conduct formal inspection that it never will lie for none of these circumstances, and then we could, for example, compare this with ours algorithms limited optimization.

I am like this: "Yes, maybe." For example, if someone from the 22nd century returned with his, you know, stack compilers and their mathematics and algorithms limited optimization, perhaps. Well, it seems to me possible, but we There simply is no such thing. Are there any now? convincing voices, who see the way? Not that so that they think that already have the answer, but they see the way and they say: "We just need to buy more time to do this make". Is there no convincing way forward? I think there are many ways forward that predict that in we will have more time.

But is it really astronomical amount of time. I don't...I don't even... I'm sure, you understand? Well, I'm not sure. I don't think it's not. will take a thousand years. No, no, but I'll name three. generation, as for me, astronomical amount of time. I wouldn't use the word that way. "astronomical", but OK. One generation is approximately 60 years old. Yes, I would like to astronomical was associated with stars. I would say 60 years old. —that's nothing compared to with a star. For me it's something if something is happening in within the limits of my life, it is short.

Personally, if my the children will live in wonderful world, ready to accept it personally. Maybe not. everyone, but if I had to, you know, to have an imperfect mother life, but to guarantee one's children have a wonderful life, I would accept this exchange personally. But I I completely understand that, maybe some people will not accept this exchange to varying degrees. Hmm, there is a compromise here. But for me the idea of ​​10, 20, 30 years is what instead we we get tens thousands of years of wonderful of the future, and all our problems are resolved, and we are not we will become extinct–this really smart exchange.

For example, How old were you? would be willing to exchange "we" we will definitely measure" on " we definitely won't die out ". And my answer to this is a personal question quite large. What you said is key issue, on which we must answer before we will be able to move forward, namely: how the world looks like, Which one do you want to live? If you are going tell us the main ones principles of how this looks like perfect world, be Please describe it. AND then, one thing that I I really want you to turned on because you, You probably have this.

the basic assumption that technology is promise of better of the future that I I imagine better. the future as personal relationship with technologies, and you the first person, from which I hear that technology is not the only way improve your life. What, as soon as you you say, of course understandable, but Describe the world for me. and what we are must want to live, and what will ours be like relationship with technologies. I'll give you two somethings. different answers or different intuitions with about this. On- first, I don't think that description of a specific utopias are often very a good thing, therefore that we don't know how looks real Utopia.

So, what I I think about utopia or, for example, about good world, I think about what I love call fair process. What is process, using what the world could become on the right path? By this I mean that is good for me The world doesn't look like this, as if we decide everything forever, and this too, you know, luxurious communism or something like that . Well, I don't know, do I? Because I don't know what it is. correct. What right for me, that for other people, you know how looked good world?

I don't know. I am not I know, I didn't understand. this. Nobody does this. I understand. AND to try to pretend, that you understood it, — This is a direct dystopia. So, I'm thinking about this. so that the world is good it won't look like that, as if we had decided everything problems, but as if each of us has justified beliefs and the feeling that tomorrow the world will be a little better. Next year the world will be a little better. And a year later it will be a little better .

I'm not saying that's a problem. no way, we don't do it mistakes, but we have there will be a feeling that yes, we reach progress. We iterate , do you understand? Perhaps, sometimes we will take a step back, but we believe, that this will be fixed . For me, good The world is when you believe that if tomorrow will be new invented crazy technology, you know, someone will invent supergigamer the death device that whatever, right? You know, in a good way world your reaction to it would be: "Oh, I know, that the government...good people, government, they control it.

I know that everyone the smartest scientists you know, everyone will come out the most outstanding, uh, military leaders will support the cause, will be heroic and will make the right ones things." This is how it is is felt. Many what is it institutionally. When I …When I am here, in America, and I love America, it's wonderful. country, but it...but she really isn't a good country in in many ways . Many in relationships when I I look at America, people don't have the assumption that yes, Everything will be fine.

We are it Let's figure it out. For example, my dad, um, he was good man, and he was very...he loved America. He was very a good person. AND he became very ill, very, very sick. AND insurance is not paid. So we lost everything. And we were to be left without a roof over your head. Emm, then we moved to Germany, because mine The mother is German. AND, you know, him sent to hospitals, and they took care of him. They gave him everything medicine, everything, you know, the treatment he received it was necessary.

And he asked them something like: "Well, how much am I "Do you owe me for this?" AND The doctors replied: "What ? You...you mean nothing to us. guilty." And they just left. As if it wasn't something like kindness. It was something. like a process. It it was just boring. It was something like: " Yes, you are sick. You receive treatment, Which one do you want? To date". For me a lot of good things the world looks very similar. This is something on like, "Oh, you want to create beautiful art?

Of course. Here are a bunch of beautiful ones works of art, You know, of course. Oh, there is. a new threat that threatens our country? Yes, no worry. The military is doing this are engaged. AT, something is happening new crime? Not worry. Law enforcement agencies will take care of it, and the courts will fair." For I am in many ways reflects how looks good world. So, when is this will happen, will be it is obvious that we we will be good to them regulate. For example, social networks will be simple good for people.

Obviously, we will regulate our algorithms like this so that they don't caused addictions . We will make them like this so that they don't push people in negative spirals. We we will make sure that they encourage people disconnect from phones and to spend time with with your friends. We have there will be a lot of beautiful architecture and beautiful arts, and people will spend time together in real life life, with their own children and all that. This will be in a certain It's boring. This will be just...I really inspiring era Enlightenment.

Western period history, from which many, in particular, you know, American Constitution, learned a lot your ideas. And what about me? really like it in the era Enlightenment, that's it what sometimes critics the Age of Enlightenment they say: "Oh, that's it rationalism. You know, they are trying to be very rational and to reduce everything to rationality". But in fact, this is not true. Thinkers of the era Enlightenment much more preferred such use mind. They wanted to be smart So, for me, good The world is not perfect, He is smart.

Or can you do it determine? This is part of what I I try not to determine. This is what we all have intuition of what is reasonable, and it's not a formal thing. A good world is not formalized. This is not some highly modernist, you know, communist, you know, a mechanical thing . This is what for me in good world all same sometimes problems occur . People do mistakes, you know, maybe some the corporation made wrong choice, you know, maybe There was an accident. Such things still happen.

But every time you considering this, you you think: "This is smart error. Do you understand why could this be happen, and you know what We will fix it. Supposedly there is someone responsible, and they will be smart in their own way way to fix this. For me, it's how looks good world. When, you know, on horizon appears superintelligence, people will be smart. Scientists will turn to government and say: " Hello, we are scientists, and we we think that this and that "It will happen." And the government will say: "Hmm, yes, this very clever thing.

We let's talk to everyone our international colleagues and find out, what is the best way ahead". I'm not saying that they will do it right from the start attempts, but all in all the chain is all simple will say: "Oh, yes, this makes sense. It was reasonably". That's it, on in my opinion, looked It would be a good world. Is AI compatible, hmm, in in the form in which it exists today, with that world? In that in the form in which it exists today, he is not deadly for this world, but this is not AI in this world , no matter what it looks like like AI today.

Interesting. Tell me Why. Because he is not agreed with people. AI. Most AI is in dollars—this is like TikTok algorithms, which they throw at children slops to cause they have disorders eating behavior. Like, this is the main thing, how it could be. A significant part funding that went to modern AI, comes from companies that accumulated huge military treasures, imposing eating disorders behavior of children for decades in the 2000s. From here first took on most of money. Financing deep learning and neural networks largely intended for recommendation algorithms.

Only later it turned out that it was useful and for other things. So no, I think that, for example, in intelligent world, if the company creates a product that, you know, for example, causes great anxiety in teenagers, they are not will earn nothing. They go bankrupt and, maybe they will get to prison for this. AND if someone creates a product that helps teenagers to study well things, start new ones friends, to be protected from predators and, you know, to grow as individuals, they must to make money from this a lot of money, and their should be rewarded for this, and we all, you know , we give them a medal and we say: "Well done." Do you understand?

I don't think so. that it is possible. Ridiculously, that people sometimes don't can imagine that we could just create a good one social network. Of course, we have technology. Of course, if we had developed algorithm that would expanded people's capabilities and would be useful, we would could. Just them would earn less money. You you will earn less money, you understand? You you can create less dependent, you know, cigarette, removing nicotine. She will be less dependent and less harmful. These things interconnected differently. So, I I think you can create very prosocial products for with the help of artificial intelligence, social networks and all these other things.

But the bottleneck there is no technology here. I I'm not saying that we need to invent some new one technology that we still not invented. It's about rather about regulation. It's about rather about philosophy. It's more about humane design. Well, what? we reward on our markets? Or reward our markets that optimizes, what reproduces the most? Are they rewarding? they are what gives us real value? It the main problem optimization. If we could just define magic a number that contains all the goodness in the world, well, perfectly.

Force a number grow. But every the number we have, Obviously not. GDP –that's not it, you understand? GDP definitely correlates with many good things Me like to have large GDP, this is very pleasantly. But at the same time , you know, it's like madness. Well, you know, I know, especially how children younger than me. You know, they are more Don't they dance in clubs? This is wild. Because of them are recording. So, if You're about 17, and you're in club or somewhere else, You dance, and it's awkward.

you are approaching girls, and you all You mess up, they'll record you. . This is terrible. It terribly. And this...So, with from my point of view, from points of view an autistic economist, was inflicted here enormous damage. WITH my point of view, astronomical damage was caused by this teenager. I think that 17-year-old teenager was inflicted huge damage because of this. Who is for this? pays? Who decides? this problem? Who just writes a check to 17 billion dollars, to stop all their lawsuits, but That's right. Yes.

But that doesn't solve it. problem. None of these doesn't solve things problem, because the problem not in technology in in a certain sense. It rather a question about What should we do with them? How do we want to ours looked like society? Which institutions, norms? Here, for example, a small example. IN Japan every time you are taking pictures by phone, he gives out sound. Always. You are not you can turn it off. So if people on the public is photographed each other, they at least they know about it.

Yes. An extremely small thing , isn't it? But this, obviously, okay. Obviously, you don't. do you want to strangers they took pictures of you on to the public, you understand? Or at least they wanted to to know about it, or not Yes? So this is extremely small , just social thing you could change. And I think that many things that we really needed, such. In history, again well, eras Enlightenment, what I thought so. beautiful...One of things that I consider so beautiful in the Age of Enlightenment, is that people who promoted technology, and people, who promoted public administration, were in a sense often the same ones people.

Or they were going hand in hand. They drew inspiration from each other. They constantly talked with each other. For them state design and society was another engineering problem. It was still one question about how can we create wonderful and beautiful society for people. And what I I feel today, something like that What were you talking about? before, this is what many people who consider themselves techno-optimists, actually there is civilizational pessimists. They they just think that the only way to improve the world—that is technology, not that how we live in our society, as our norms work, how does ours work culture, how they work our laws, our institutions.

Even despite this, when I I ask people who 10 most of their things annoying, very rare these 10 things are technological. Almost always they They answer: "Oh, You know, it makes me sad. policy. I upset with the security health. I upset, well, you know, all this, so I am sad that I didn't I can do enough. to see my family friends. I am upset. because, you know, something "such." OK? And I I think we should to deal with this seriously. I think that this, you know, I'm a technologist in my soul, but what I even more worrying is that to make the world a better place place.

It turns out, technology is not a bottleneck. I would wanted to get to the point, when technology will become narrow place, you know, when we will solve it so well culture and we will have such a wonderful government, that now all we have need to do this invent better ones technology. I would like to live in such a world. Yes. This is really beautiful. Thought, friend. It actually me amazed. I am very happy, that you found the time to tell us about it . So, as a reference, is Do you think that AI and capitalism is like toxic couple?

I am personally big fan capitalism. I think that capitalism is one of the best ideas that ever little humanity. In moderate quantities, likewise, like any ideas. THERE ARE thing about which, in my opinion opinion, sometimes forget, this is what markets are great. I I love free markets. I I like private. entrepreneurship, but this is one of tools in in our arsenal. And we we know for which problems he is good, and for whom is it bad? You know, if I gave you really good drill, you would used it to drill a screw.

You might think: " Wow, this is the best. instrument in the world." But this is not the only one tool. You know, if you need saw, you are not will use drill. You will need another a tool for this. Markets and capitalism very similar. Capitalism and markets very good at solving many problems. You know, if you you want cheap, you know, commodity product delivered at an affordable price right to your door , then, God, I have for you system. AND capitalism and markets brought to the world a huge number good for the last, you know, 100 years.

A lot of bad things also. But in essence, markets are optimizers. They similar to learning from reinforcement. They have is the number they force increase. And if this number can not to increase desirable for you ways, markets are will do. And that's why we have regulation. Throughout stories we always combined these things. It's something like MMA, Well, I love MMA. I think that's so exciting. I I don't do this myself, but I think that's it exciting, because there is the most dangerous people who ever lived in history.

The most dangerous warriors who once ever lived, you know, they fight, you know, until so incredible strength. And they don't are killing each other. You know, they can continue compete, because we have regulations. Needed many rules to make it the competition was possible. Needed judge. Needed many rules to make sure that they will be able to compete next day. IN day, you know, next match, when they can return. And markets very similar. If you don't regulate the market, you just get monopolies. You just are you forcing some corporations hire private security to , you know, break it to you kneecaps, and you don't buy yours product, right?

It evil. To get competition, actually you need to have a lot rules. And I think that This is great. I think that if we create, you know, good ones rules and we will have a good judge who is not a market, you know, which is located outside the market, for example, by the government, we we can create wonderful things. I think that this is an example of how we just don't we do our part work. This is how to imagine, that someone in MMA found a new one suffocating reception, which allows him to win, but also kills another guy.

And we just don't we prohibit it. If we don't do this let's do it, then the sport of MMA will cease to exist. Like all people who participate in it. And here is a very similar thing where is all this took place during history. Corporations find new ones exploits. They find new ones extreme cases, about which we haven't seen before thought. You know, they find new ones pollution methods. They find new ways to harm to deceive people , whatever. And then we react to it. Then we say: " Wait, wait.

You can't do this. to do". You know, to the emergence of nuclear weapons she obviously doesn't was illegal. AND then, as soon as we understood that this maybe we are very quickly convinced, that their construction on the market is illegal , because otherwise the market would be would be happy to build nuclear weapons. Can you imagine? You know, how big technological companies would like build a nuclear power plant weapons and sell them to the one who offers the highest price? Of course they would done. They would like this liked it.

So, this is always a problem design, and this the process of evolution. It repetitive process over time. We never let's do it right on the first try, and this normally. This is not means that markets are fun. Well, we have to to give up all markets and move on to communism. This is also doesn't mean that we have should not be regulation, simple markets. We just trying to solve really complicated problem, namely: how we can get an inconsistent thing, that is, a market such as market quote with large coefficients, are not neither good nor a bad thing.

It's like alien. It's like artificial intelligence. You know, it's like an artificial intelligence. It's like artificial intelligence that studies with reinforcement, and how we can force him to do what we we want, and it's difficult. We have achieved significant progress in this question, and we just need continue iterations. If you were now were going to regulate AI to make sure that we will not cross point of no return, Have you thought about it? which specific ones politics you would like accept? Yes. Hmm, there are two main ones things that, in our opinion, opinion, there should be accepted.

The first and the most important thing is criminalize creation superintelligence. This is it. that you can simply to do according to the law. You know, murder. someone is illegal. Also illegal to try to kill someone, even if you will fail, you still can't to do this. Similar to nuclear weapons. Even if you trying to create nuclear weapons, even if you don't succeed , people in black will appear near yours doors. You know, and this good. So, this is what you can just make. You know, and much easier, you know, often enough easy to distinguish a company that trying to create superintelligence, from a company that does not trying to create superintelligence.

Not perfect, but this beginning. So, this is the first thing healthy sense. Just if you literally You advertise, I create. superintelligence, I just raised a billion dollars for creation superintelligence, this illegal, and you don't You can do this. Then, the second question, how can we prevent this so that we can get closer too close to superintelligence. And for this is fundamentally , because again, if superintelligence will be built, even once, even by chance, already it will be too late. So, we must to regulate well the situation before appearance superintelligence, so that in we had enough buffer zone.

This is something like a representation, that we are leaving with at a speed of 120 miles/hour in thick fog, and we we know that somewhere there is cliff. But we don't we know where. Yes. My first suggestion- to stop. Before we will argue about speed, let's go first Let's stop. Therefore, first criminalize your superintelligence, and then adjust precursors. I have list of precursors, such as the ability to self-reproduction, um, duration of the horizon tasks, things, related to the agency autonomy, etc. AND very carefully control this and make sure that any reliable big experiment that expands these boundaries, registered government, has oversight, and then we can decide: "No, you don't You can do this.

"It's too risky." Because, fortunately, on now creation indeed advanced AI, which moves to the UCI or ICI, incredible expensive. You know, we we are talking about billions of dollars. So not really many participants, which this law will affect at all. Yes. This will not affect 99% companies. You know, will be very small mega-mega percentage huge technological companies to which it applies will affect. And for them, yes, we obviously have to to supervise them just like we have supervision of such companies like Lockheed Martin or, you know, nuclear facilities weapons, etc.

AND what about AI with open source? Open weekend the code is very complicated question. In that sense, that I love open source code. I created one of the first in the world of models major languages ​​with open weekend code and led a group called EleutherAI. I grew up in the world open weekend code. I think that open weekend code brought into the world so a lot of good things. I I think that Linux, being open weekend code, and all this open weekend code was brought into the world so much beauty and good.

But for me it is not ideology. It question, you know, whether must be fighter jet blueprint F-35 with open source code? Probably not. Probably not. It seems what wouldn't it do the world is a place, Do you understand? Should we be a drawing for, you know, biological weapons or, you know, nuclear bombs from open weekend code? Probably not, Do you understand? And here is very similar thing when if we will get to the point that there are systems with open weekend code that are superintelligent or what can be done superintelligent, what for , I think we are quite close, then, I think, that way back in in this regard, probably not.

You know, if there is one a system that is almost super smart, and she is only on one server in one country, maybe , we can her remove. She, you know, and maybe. But if there is someone around billions of things from open source, and we we know that they super smart or, you know, in two steps from super smart people, then probably too much late. I think that actions, which would be necessary, for example, to to recall, to quote, open model code, are impossible. Ahem.

Okay, Bill Gates. just said that now the AI ​​thinks dangerous. What happened? What made him to change his opinion? I'm not sure. how much he changed your opinion. Of course, I don't know him. I I think, you know, he very, you know, he knows a lot. He smart guy. I sure that he saw a lot scary things. But back in 2023 Bill Gates signed the Statement AI Security Center, in which stated that risk reduction disappearance from AI has to be global priority. Its signed by Bill Gates, Sam Altman, Dario Amodei, Jeffrey Hinton, like many others really famous people.

So this is It was back in 2023. So, I think, you know, again Well, I don't know him, but I think Bill Gates always knew about some of these risks and right now is trying, perhaps, is trying now to report them a little more. Personally, I I wish he would went a little further into in your letter. I think that he admitted some risks, but did not go up to superintelligence and the risk of extinction, that really bothers me worries. Because, in in a sense, I don't I think we have enough time to to see mass layoffs places.

I think we we'll see superintelligence by a few years before that, how will it happen mass reduction jobs, and already it will be too late. I would wanted us to live in world, you know, where superintelligence will be in just 20 years. Then layoffs there will be places a huge problem , but I'm not sure, that we will even reach until this moment. This is incredible. It incredible. Okay, I I want to check if I understood correctly. your main assumption. If I missed something, let me know, but this was very helpful.

Uh, they are about okay, but, uh, there is one thing about which in I have an extra. question. Good, superintelligence is, according to In fact, you may not have gone so far, but superintelligence is rival. So, knowing that as soon as you will do it, he will start competing, and you will have problems. Uh, superintelligence will consider us distracting factor or an obstacle in the way to their goals, therefore we will have to to worry about to be exactly what you want what they compete with, and we will lose.

Ahem, unknown risk is assessed at levels near peak danger. That is, if I don't know the result of this, as you said first to stop. If we We don't know where we're going. So, when we we end with, that I don't know what will AI do it, I don't know what will we do here superintelligence, press the brakes. It is obvious that there is spectrum. Hmm, as with any -how difficult strategy, how do you are you dealing with known unknown or unknown unknown? How are you?

you calculate the probability that the country will invade What about you tomorrow? It actually the problem is, about which there is much literature. This is a thing, over which, you know, the CIA and, you know, different military agencies spent a lot, many decades very, very hard worked. One of recommendations, for example, the CIA, is to always express your intuitive feelings in numbers. So this is a thing, which we don't know talked. People often get annoyed when people they say something like : “Oh, I think there are 20% chance that everything will work out good".

And they say: "This is not mathematics." AND I say, "Dude, yeah, they know. I know. This is not mathematics. The reason why you you say, "Oh, 20%," not in because I'm trying do the math assertion. This is because what is known research, It seems it was from CIA, where they discovered, that analysts who Did they say "probably"? "unlikely", actually had on completely different in meaning things. For example, some people when they said "probably", meant 90%. Others analysts, when they said "probably", meant 30%. So it was extraordinary confused, because "hardly "or" means nothing .

So, if you say " 90%, it's still a feeling, but much more specific feeling . This is much more, when I communicate with you, if I say "1%, I I'm trying to say " No". Oh, it's exactly 1%, not 1.2%. "That's not what I'm saying." I I mean, this is approximately 1%. This is if took into account my beliefs about the world and, you know, looking on my models realities that, you know, you could to assess how well you believe in my models reality, this the result would be about like this percentage of results .

Of course, he based on my models. What else? it should be based, isn't that right? Obviously, in You have views on the world. You know, if I asked you, how likely is it, what, you know, is this will happen in future. How do you do it? Do you evaluate? Well, you use yours model in your head . What else are you going to do? do? You know, here very similar thing. I I'm trying to do its model explicit, for example, with these basic assumptions. I I'm trying to make it.

quite clear. It leads to this, and therefore has a high probability. Therefore, if all these statements that we they said until now, they would be approximately one billion fraction of a percent probability, I…” not I would be worried. But I don't I think so. I I think that if they everyone, you know, above the 1%, and after their conclusions are still higher, than, you know, after multiplying them all together, we are still above 1 %, and the result is extinction, damn it take it away, no, Do you understand?

If the result is: oh, this one in a trillion, I like this: of course. But mine the numbers are not from this are coming out. Right. Makes sense. OK. Um, financial incentives, incentives power and lack of hard regulation allowed us continue to move forward, in spite of danger. Yes. For AI to work, it needs one's own will is needed. It depends on what you mean by by will. I think that this will be the optimizer. Now that I know the language you use, I would describe it this way.

He will have to optimize for achieving certain goals goals. He will use events. He will make plans. He will optimize to achieve goals. A sociopathic optimizer like you described it earlier, this is...a problem. Yes. Okay, AI can. There are other problems, but that's really the point affairs. Good. AI, this is too much it is difficult to to fill AI morality. We don't have idea of ​​what it's like make. Good. Life is possible improve not only technologies. Yes. You can involve China. Yes. And people are not magical.

It what I care about additional are needed clarification. Me really impressed when you said it, but I wanted better understand. You tried to explain something about AI, saying, that people are charming, you need this understand. What are you do you mean when you say that people don't magical, and what is it tells us about AI? There is a very interesting thing in Wikipedia, which called the AI ​​effect. I recommend view article in Wikipedia, she quite interesting. She is that throughout stories AI people always determined something like: "Oh, we we will know what we have real AI when X." "And then, as soon as X will be achieved, all they will think: "Oh no, this not quite AI." This is not true intelligence.

« For example, once upon a time people thought that the uniqueness of people, their difference from all animals there is ability to calculate . And, obviously, computers can expect. Very good. Very good. So they like: "Okay, no, no, we don't have that on attention. We have on meaning that in reality makes people magic what we can identify picture or something like this. You know, and then you are like: "Or they can play chess." "Or, you know, and, obviously nothing this was not the case important. And that's why there is something like rising tide, increase waterline, or, you know, the drive in an angle like this very funny cartoon.

I think it was based on Ray Kurzweil or something like that where you you see a guy who writes furiously on paper : "as soon as people can", "as soon as people can play chess", "and how crosses it out, as only people can to draw, "draws out" this, and how, and it's like the whole his room is covered these paintings of his, and how, "as soon as people can do", "ah, ah, "Oh, you know." And I am with this I want to say that people —this is physics.

We is actually part of physics. Nothing not magical is happening. It difficult. Our brain complicated, but it's not magic. This is physical process. In our brain occur some calculations. You know, this use of some strange soft, you know, elements." for calculations, but essentially happening some calculation. It not magic. You can to force other things to do similar calculations. So this is the way explain that, as we we know, maybe without some kind of magic element. That's right. And that's the whole point. Dude, this is crazy.

Hmm, it was true. great to explore you and sit opposite you. I am very, I really appreciate it. I think what is yours message sounded loud and clearly. If there was one thing that, um, people could not realize, for example, if you sat opposite G or someone who is like you, I know that they are convinced, are there any more one thing that did not arise today, about which you would like to so they know? IN I have other answers. for Xi Jinping than they have for people who are listening, so I will give two answers.

So, honestly, the thing number one, which I discovered that this is what grown, not written. People of this They don't know. When I I am talking to politicians, the first thing that I say this is an explanation, that this is not normal software provision, it is not written, it...grew . We don't understand how these things work tonight. We raised this issue, but I'll say it again once, because it's so important understand. People at OpenAI they don't know what is happening inside their artificial intelligence. Dario Amodei, CEO Anthropic, believes that he understands that perhaps 3 %...

Oh, my God. happening in our artificial intelligence. AND I think that's even optimistically. Wow. So, this is one of the most important things, which one is needed understand. We are not we know. We...They don't know. Nobody knows. This is very important. And for wide public and everyone who listens, I I think that very the important thing is that it is still not the end. And although sometimes unfashionable to say this in our days, we are still alive in democracy. And yours The voice is important. If you want something It happened, I love to think about civic education so that our work is to work with our politicians to help them understand that we worries and what can be done make.

So do it. your contribution to help attract attention to these issues. Go for link controlai.org, contact your legislator, tell him that you are concerned about these problems, Help to enlighten people, continue the conversation. This is how we, people, we solve all these problems, you know, together. Hmm. I like it. Where people can subscribe to you? Hmm, I...you can find me at Twitter, you know, and now X, sorry, uh, on mpcollapse. You can find me, of course , at controlai.org. And also, if anyone else wants to get more deeply involved in some of these things, I actually in charge volunteer group called Torchbearer, torchbearer.community, where we we commit spend at least 2 hours per week, to try build this intelligent world.

I like it. Okay, everyone. If you are still they didn't do this, necessarily subscribe. And to next meeting, friends, be legendary. Take care of yourself. Peace. If you liked it this conversation, check this out episode to find out more. Artificial intelligence is, undoubtedly, the greatest discovery, which ever humanity will do. This is not an easy game, which is it not easy to win or to start, but it doesn't stop. Artificial intelligence absorbs everything... liquidity, and this creates problems in cryptocurrency. And now,

Go deeper

• All Impact Theory episode guides