Why AI’s Next Breakthroughs Could Come from Outside the Big Labs

Key Takeaway

Audit one AI-enabled workflow today as if it were a new employee with unlimited speed and a credit card. Map what it can access, which APIs it can call, and which actions require explicit approval; then reduce permissions to the smallest useful scope. The episode’s practical lesson is that agent swarms turn ordinary internal access into a security concern, so granular authentication, logging, and least-privilege design must become default operating hygiene.

Episode Overview

The panel debates whether AI safety policy should focus on vague existential-risk rhetoric or concrete, observable engineering risks. It argues that cybersecurity—especially authentication, incident reporting, and agent access controls—offers a productive starting point for AI governance. The conversation closes by exploring how innovations outside major AI labs may make models cheaper, more probabilistic, and easier to embed in traditional software.

Full Transcript

Transcript of Why AI’s Next Breakthroughs Could Come from Outside the Big Labs from A16Z. Auto-generated from episode audio; may contain minor errors.

If you adjust AI too early, you're not really solving anything. You ultimately still face the same risks. You are implementing it , but you haven't figured out how to manage it. The problem now is that there is a gap between the labs and the security community, which constantly comes to two conclusions. Negligence and incompleteness in reporting what happened. Workers have about a 10% chance of extinction. Swarms of agents change that completely. They're just wandering drones, but 10,000 times more powerful, and they easily confuse a good task with a bad one.

So now we need an inner layer that tracks a lot more authentications and API calls. About 15 years ago, the US stopped leading in antitrust regulation of technology. The problem is that Europe will lead this process because they have nothing to lose . This could fundamentally change the nature of software. The center of innovation has simply moved. This is a signal that, guys, welcome back to the podcast. Glad to be here. Thank you. Your beard. I did n't think we would ever do that again .

I can't believe it. This is great. I mean, Martin just builds companies worth hundreds of billions of dollars. Too busy for this podcast. So, at least he takes credit for it, like venture capitalists do. That's right. We have a lot to discuss today, but first, Aaron, why don't we start with you? On the edge of the possible. How did you react and what do you think about what happened and the discussion that arose after it? Oh my God. I think we should start with Martin on this issue.

You fought many important battles. Maybe I'll say one thing that we can all probably agree on, and then we'll see where our opinions diverge. I think we can agree that any AI lab at the cutting edge needs to be built in the safest way possible, with the highest level of governance and protection, and with any definition of consistency, this is an incredibly important area of ​​research. This is an extremely important area for the spread of AI. You won't get AI adoption without extremely high-quality products that businesses can trust and that don't constantly hack systems.

So when I at least read Dario's post, I actually disagreed with almost nothing. Well, because it all came down to how to provide better, you know, security of these systems, isolation, better testing. There will be some discussion about bringing in testers, do you agree with who they are and does the entire industry agree with that? But I think that actually all the main points were relevant and valid. The only question that remains is whether this will be used for something we may not agree with, such as significantly slowing down the development of AI through regulatory control, which would prevent competition with leading laboratories.

Or will politicians adopt this message , and the results could be even worse, for example, they will use it to close data centers much faster, etc.? Therefore, I believe that the essence of this topic is incredibly important for the development of AI in general. And then the question arises, what to do about it, especially from a regulatory perspective? This is probably where industry opinion will be divided, but Martin made a good case that we should not allow this to be used for regulatory market capture. I also agree with this, but I think ideas about the pace of development are important.

It's a bit of a strange concept, because maybe it's not so much about pace as it is just about proper hygiene and quality engineering. With good engineering, of course, there is a certain slowdown, but it allows you to accelerate implementation, because you would n't be able to spread AI if no one dared to use it. So, the post is very valid, but the overall atmosphere is not quite right. For example, an employee says: this will stop the possibility of a species becoming extinct, and you know what Dario says.

I agree with him more than I disagree, right? On television. On television, the same day he released these things, and to some extent it's impossible to have these conversations in isolation, because if he agrees on the extinction of the species, this post looks like a weak surrender, completely inadequate to the current situation. So, I think the atmosphere is completely ruined. And many of my comments were about this very atmosphere. And I have one more remark, it worries me a lot, because I feel obstacles, namely: I think that "pace" is the wrong word to describe this process.

First of all, yes , this has nothing to do with security. Mhm. Exactly. Well, you can build nuclear weapons very slowly, and that won't reassure anyone, regardless of the speed. So this is the first thing. Secondly, it looks like capitulation to those who demand a pause, without any real solution to the problem. You say, “Well, we won’t pause. " We will adjust the pace to satisfy them, but at the same time somehow please the regulators." And I think that makes both of them unhappy. Because the proponents of the pause say, “Well, this isn’t a pause.

It's just a matter of adjusting the pace." And the regulators say, " You keep doing the same thing anyway," right? So, it seems to me that the labs are actually trying to do things right. And I congratulate them for that. I consider this a pragmatic proposal and I congratulate them for it. I think the message is wrong because they are trying to find a compromise between the internal fringe group, those "doomers", and the regulators on the other side. And the problem is that they make both of them unhappy.

And I think what they need to do is speak out . They need to directly answer questions about existential risks . They have to say: no, we don't think what we're doing will cause extinction . And then, in my opinion, this would be a very smart approach. What would you do, just to get on the other side for a moment? What are you doing? Do you create space for that same intersection of interests, where a researcher in a lab is simultaneously very scared but continues to develop AI because they believe it's so important to do it right that they want to do it .

Of course, the terminology is very problematic now, but such a person exists —this is a real type of personality in our industry who says: we need to be at the forefront of AI development. I'm also very afraid of this, and that's why I'm working on it. So let me answer this straight. I once worked at Lawrence Livermore National Laboratory on the nuclear weapons program. I know what it's like to work on things that exist. You were the first to set the pace. We were part of the first studies.

Yes. So if the group in the labs, the most knowledgeable people, believe that this carries an existential risk. Yes. The answer is to nationalize it and implement controls that we know work now. If they don't really think so, and in private conversations, most conscious people don't say so. Only a small fraction does this. This is an HR problem. Is n't that right? So for me, an HR problem is a company problem. If they're worried that they can't hire people for sure or they can't retain people, and I think a lot of that is just a concern, almost like a kind of research currency, if that's the case, I think that's the wrong reason to impose a national lockdown on a very promising technology.

So listen, again I think this post is actually very valid. I think there are real security issues; we've had many computing eras with real security issues. I do n't think it's possible to reconcile discussions about existential risk with this proposal. These two things are simply not compatible, and that has always been my main argument. Yes. Yes. OK. To release. OK. Hold it back. So, okay. First of all, you can't... Is there a timeline that they've released that shows when all this disaster will happen ? They didn't do it.

So you can't set the pace because no one knew when it was even going to end. This also looks insincere. This is a complete claim, as if the press is reporting that Apple's new iPhone is late. A phone that no one knows exists, and that they have n't told anyone about. Yes. My Apple car is very late. Yes. I don't understand. To slow something down, you need to know the speed at which it was moving from the beginning. So this is all just complete nonsense, and there's no getting away from it.

And then by the way, that's another problem, like again , my little remark about PR: no one believes it anyway, right ? And if that's the case, this pace, right? What part are they going to slow down? That they are going to slow down. They ran at full speed. They raised more money than ever before. They were developing faster than ever before. There are no signs that they are slowing down. So if that's what you want to emphasize, I got it . Of course. Yes. Well, the internal problem is obviously that the models that everyone has inside are vastly superior to anything that others have access to.

So it's really just a matter of the pace of external releases. But again, going back to your point, pace compared to what? Yes? We don't know if the model that scares everyone doesn't work when writing legal documents. She could ruin everything, so who knows, right? That's the thing. And it's just disingenuous to claim that you're worried. Secondly, why do they have to announce all this and ask the government to tell them what to do? This is where it starts to seem pretty creepy. If you are most afraid of the consequences of your product, just stop.

Don't do this. You know, I worked at a missile factory in college, we had nuclear missiles, and I walked around the shop floor. What's your business with missiles? I didn't, I just did software. In college, you were one of two types of people: either you protested their presence on campus, or you built them. That was the choice. Nuns from the Catholic Church would come to us and pour blood on our missiles , and I would just carry computers on carts and say, "Here's a secure PC." And I was scared to death.

I didn't understand what was happening at all. I thought : "This is a nuclear missile." And then it turns out that this is what stopped the Cold War. It was a Pershing missile, and that's what worked. And you also said something very interesting: pace is such a vague, meaningless word between action and inaction. Yes. The problem is that you are absolutely right—no one will be satisfied with the golden mean. And one of the things I find almost as interesting to watch as sports. They think that all these people who tell the government to do or not do something will get what they want.

This is a complete, 100% misunderstanding of how government works, because when you go to the authorities, they actually know how everything works. They just listen to you, like everyone else. And no one will get what they want, because everyone knows: to achieve anything in our system, compromise is needed. So no matter how many opinions get to the government, the result never makes everyone happy, right? In 100% of cases, either nothing changes or it goes too far. Yes. Therefore, you cannot expect that you will talk to the government and persuade them to make the decision you need.

So the conclusion is: if they ask for pace, they will get the wrong speed. Well, my favorite story, how was it? David Sachs , it seems, or someone from the government said: you are asking us to regulate you. No. Facebook. The answer was "no." Well, the thing is, it was probably Trump, I think, but what they know now, and what you just understand from experience, is that once the regulatory flywheel is set in motion, there's no stopping it. And now it has become an election issue for every party in every jurisdiction, at every level of government.

So now there is a whole basket of approaches to regulation. Well, the next election will 100% be a referendum on AI. This should happen, 2028 will be an "AI election," and in principle, one could run with this, but it is unclear who will stand for AI, because this story is too vague. So the only thing left to do is vary the level of regulation or at least try to avoid this topic. Yes. But it's unfortunate that we, as a country, have found ourselves in a situation where the arguments for AI sound too complicated, it requires too many words, you know, it's very nuanced.

This is an excuse. Yes. This is an excuse. This is a defensive position, and we don't own any term, right? So the whole discussion boils down to "pause." To " swarm attacks". To " get out of control ". Every word was chosen by those who do not want AI to develop. Yes. This means that the first thing to do is to come up with new words and prove that their words are false, and that will take so many words that Yes. We need to talk about unionized jobs, we need to talk about cancer, we need a whole different cloud of words to emerge.

I don't, I don't understand why the labs haven't taken a stand on existential risks. Because, apart from this, I see no other way except for strict regulation. Yes. Well, it would be simply negligence on the part of the government to say that there is a 10% chance of a species becoming extinct. The CEO of a leading company says he agrees with this . How can the government not do this? Have you implemented regulation? Well, but who...Who really, knowing this ecosystem, would be able to get someone to sign up for this?

No, I would say Dario said that... No, I mean you can't force anyone to say a lower number. Well, actually he does the worst thing he can do, he agrees with people who say they believe in a certain percentage of extinction, but he specifically tries to say, "I'm not going to name a percentage." Which, to me, is the strangest thing. No, but, in fairness , in fairness . No. Good. No, no one can be fair. Uh, to be fair, it's not 100% hypocritical or anything, he probably doesn't agree that it's exactly 10%.

Or maybe I agree, and it's just too scary to voice. Let's just play binary search. So, is it more or less? But this whole concept is made up, we created it ourselves. No one can, it is impossible to quantify this. Well, that's the gist of Martin's position. But you just had an official position. The only official position that could be obtained is the PR version, and the only thing that would be intellectually honest is: AI carries real risks . At the same time, there are incredibly positive benefits.

We are working to mitigate risks to minimize them as much as possible. I don't think that's, I don't think that's true. Listen, what do you think? We have gone through several eras of technological development. We went through computers. We went through the internet. We went through the network. We went through social media. We discussed risks, not to mention existential ones. True? For example, we can say that we do not consider the marginal risk of extinction of a species to be anything other than that. What if they think he is superior?

Well, then we should Okay. Okay. Yes. I think the answer is that they see it as something bigger than the internet. One of two options. You believe we're going to die out and we're shutting the hell down. We kind of close it or they believe that this is just a chance that we have to say. Dario thinks less, but these cold-blooded warriors. Nobody like the Pentagon, you know, they've done a lot of simulations on the probability of nuclear war. The wonderful film " War Games" is about all of this and more.

But the thing is, it was, it was non-zero. Yes. And so once you said that this is Can they say, can they say " non-zero" then? Is this allowed? I think as soon as you think it is non-zero. The problem is, if you say " non-zero," it could mean, like, damn, he thinks 93%. Wait until we all have a clear conversation. Let's talk about marginal risk. We're not talking about absolute risk, right ? Okay. I just think that once you say it's non-zero, the only answer, like catastrophic, the only answer—you have to nationalize it.

Yes. And so what he's trying to do, what labs in general that have this view are trying to do, is to get non-zero value as a license for a certain set of actions without the burden of becoming national. Well, do you have...I don't really know all the precedents. You know, I hope, but, but, there have to be some things that are non-zero, uh, let's say, existential risks , that are not particularly nationalized. But...but the regulatory environment around them is so heavy that they could be considered nationalized because of KYC requirements, like, for example, I'm sure you have to go to a certain lab to develop anthrax.

Well, there are BSL4 labs, but they are nationalized. They are national. Okay. But ethics are usually nationalized. Yes. As a normal human safety is not very good. Right. As industry oversight, which eventually becomes federal regulation. Right. And the progression that I think is very important to this discussion is that since the post-World War II era, most of the industries that are critical to infrastructure, energy, banking, and healthcare. The trend is to essentially be nationalized. Yes. Just because of your KYC examples and everything else. The banks are effectively nationalized, as is the financial crisis.

Okay. But you said " actually", right? They're not literally nationalized, are they ? So maybe the idea of ​​labs is to be like JP Morgan or Verizon. They said we are critical infrastructure. We are strictly regulated. This is not in the interest of open source software, at least cutting-edge software , but it is a very likely scenario for the industry—the question is whether it is good for innovation. I think that's fine, just don't use the threat of species extinction as an argument. It's literally the difference between things stuck in the lab...

let me just say: I actually think labs are moving in the right direction . I actually really liked the statement itself. You know, in my discussions with executives and leaders, I see that they understand this tension and are going to resolve it. So I'm pretty optimistic that labs are doing the right thing and trying to do the right thing. I just think it's all happened too quickly, and they're just trying to figure out how it works . And I think Sinoski hit the mark. The political process is a separate story.

And I do n't know if it's naivety or pride , but they just don't understand how to navigate this. Well, I think the tech industry has been learning the same lesson with every wave of technology for over 100 years: we don't understand the regulatory climate and we don't know how to operate in it . Even companies like AT&T and IBM , which were de facto state monopolies from the very beginning, never learned this. Both received antitrust lawsuits. Both have undergone significant structural changes, although in the 1960s they had hundreds of lawyers to handle these cases, and then Microsoft came along , and we're like, "What?" Yes.

We had no idea what was going on, and Bill Gates was playing golf with Bill Clinton, and the Clinton administration filed an antitrust lawsuit against us. Bill said, "I played golf, here's a picture," but it didn't help. And he's like, "Shouldn't I just play golf?" ". This is a shortened version of the whole story. But I think, and I think it's just I always give this example: Hollywood rallied during the " Red Scare" and censorship, when they were worried that the government would ban movies because of sexual content or adult themes.

They all got together, and although they would never have won in court if they had tried , they banded together because of the threat and formed the Motion Picture Association. Yes. And movie ratings and all that— they started to control themselves. So, how do you guys like it? Do you like FINRA's proposal? No, because FINRA is...well , it's as close as possible to MPA, only with more powers. No, it is not, because FINRA becomes a legislative act that provides direct supervision. Yes, I think the MPAA may not have that much influence on the functioning of society.

First Amendment. Yes. Ahem. Did I win this argument? I...no...first of all , that's fantastic, but I still don't know if you won. I agree that freedom of speech is very important, but... you know, I just think ... There was no public risk there. I just think that what's in our movies, it's...we'll survive, like being in another continuum... All history is always relative, and at one time being a communist was really bad, and 30% of Hollywood was fired because of it, well, you know, that was all.

Yes. So there's always a risk of mentioning something like that because it sounds so silly. Nobody thinks about movie ratings, and that's because, in fact, it was decided that they cannot be constitutionally imposed. So they couldn't regulate them on cable TV, and we grew up with HBO and everything else. But the problem with FINRA is that it is a perfect example of, essentially, the nationalization of risk. Yes. Because even though banks pay money , and that's how it works, all of this is mandatory. OK. Wait, sorry.

Do you think we will end up with a better situation than FINRA ? FINRA seems like the best option, but that's only a best-case scenario. Although not even now . I really think that this technology is so powerful in the long run compared to what it can provide that Congress will pay attention to it sooner or later anyway . It's simply impossible otherwise. Question. If it eventually happens, at what point? If it ends up making every recommendation in your treatment process, and it's inside your medical device like an open-scale model, and it 's in every high-frequency trading system , and it's on every airplane, then there's no way the government won't say, we need something like FINRA, which is probably the best-case scenario for what it looks like.

Yes. Aha. And this is the most important moment now , because all the people who are in the Senate now were there when the Communications Act was passed , and the responsibility was not shifted to internet providers and social networks. And they sat and discussed it at the time, they literally said to us, "The Internet is so huge, how could we not have anything to do with it ?" And that's why Al Gore gets a lot of criticism for saying he created it, right? He was actually trying to get ahead of the curve and say, "No, the government played an important role." And it all turned against him because it made him look like a madman.

But it is absolutely true that they felt they had lost their chance to control the internet. When Al Gore was on the side of supporting innovation. It was their support. So who is Al Gore then? There are none. There are some, well, it seems like people in the defense sector kind of want it to be private, but not really, which is exactly what they thought about the internet, right? And that's why it's very interesting. A lot of it is just like Senator Elizabeth Warren's tweets that boil down to, we missed it with social media , and it's like...

There's a lot of pent-up energy against technology that could just flow into AI right now. This is exactly what is happening. Here's what's happening. Yes. I think there's another problem: we all try to predict what's going to be bad, and that's not how we usually do things. For example, by that time on the Internet we had already caused tens of billions of dollars in economic damage. We had viruses that took out 10% of the infrastructure. Yes. Internet infrastructure on which critical infrastructure operated. Our hospitals were failing.

We really should have blocked the internet sometime in '97. How about us...Yes. Yes. Good. We...I remember the days when you bought a Windows 95 CD, and by the time it was installed, you could have already caught a virus. Well done, Stephen. No, no. This was the reality for PCs until 2001: you couldn't connect your computer to the network and not get infected. Viruses were everywhere, and there were two rounds of congressional hearings. There were all these, it's actually normal. OK. I must admit, this is a very interesting moment.

Such was the spirit of the times in '94—we probably wouldn't have had the internet. Listen, in the automotive or aviation industry, there have always been these formative periods where you learn from the dangers and the technology, just like with the internet. We were at the origins of all this . That is, everything was constantly breaking down . And there were constant economic losses. New viruses appeared. New worms appeared. They were everywhere. Y2K. Y2K was supposed to ruin everything. But here's what's interesting. I just want to say that when we created policies —and we created a lot of them—we relied on concrete data to understand what we were doing to make sure the policies reflected the real facts.

In this case, even when you were talking about "what ifs," it's very difficult to plan policy in advance when we don't know what we're talking about, except for one area: There seem to be new risks emerging in cybersecurity. That's great, and what's nice about the discussion that's starting to form around this is you hear lab experts saying, " Emerging cybersecurity risks are an engineering problem." So, the more specifically we talk about the real risks identified, the easier it is for everyone to reach agreement, but so far the discussions have not been like that; This is a very recent phenomenon.

An apt observation, because if you look at 1986, that's when the Computer Crime and Fraud Act was signed. This was the result of a very specific situation where two groups of hackers hacked GTE Telenet. One of them lived in my dorm. Just imagine. And of course. No, it wasn't you. Oh, he worked at Cisco later. And, um, and the problem was that it was 1983, and there was no crime. and it took two and a half years for the bill to pass . which made it very specific, and it's a law that says you can't access unauthorized computer systems.

So I think we can all agree that we probably already have 90% of the laws at the application level, like you can't hack systems and so on. Do you think that in terms of responsibility there should be something on the level of models? What, of course, then how do you make the technology legal? No, all my reading of Hugging Face, OpenAI—they're all absolutely blatant computer crimes, except they later amended it so that if you're a " white hat hacker," it's no longer illegal. And that's because people constantly made mistakes .

and they didn't want to arrest everyone who was genuinely trying to improve the system because they screwed something up. So the Justice Department wrote a memorandum that said, "We will not prosecute for this." And we also won't prosecute if you're just violating the terms of use, right, of the system, and not trying to hack it. And so I believe that the legislation is sufficient for this scenario. And all this was written, this GT Telemet was used by NASA, Livermore and all the laboratories. That's why it caught Washington's attention, because it was the federal systems that were being hacked.

And the problem we have now is the disconnect between the labs and the security community, which keeps looking at all their incident reports and coming to two conclusions. Carelessly. and you are not telling us the full story of what happened. And so, of course, the CVE process emerged in the 1980s. a computer virus and vulnerability reporting system with CMU, and for years they've been working on very structured reporting with commitments, and for some reason they're not using any of that for that reporting. So there are very basic things that I look at and say: until they do that, they really should stop talking, they shouldn't do a hack report that looks like it was written by an intern, and it's not an incident report, it's selective memory, it seems.

It's the type of report you do when you hire outside lawyers to investigate something random and only give them certain information. because you don't have to provide the hired lawyers with all the information about what happened. For example, where are all the messages in Slack? Where are the actual details of what happened? Can I make one annoying remark? So, this is, well, you know, this is very much on topic, this is something that, uh, I've been in the security field for a long time , actually, in the cybersecurity community, and this, you know, has always been one of those things where they just don't like practical solutions.

So even if you build, say, a secure system, the question is: what if someone shows up, you know, like ... you know, can Russell Crowe break the encryption or something? This is "Mission Impossible." Some kind of team, there's always some kind of stuff like that. What I liked the most was something that happened recently when Noam Brown was on a podcast— we all talk nonsense on podcasts. I already said oh, you didn't like it. No, it was great. I thought it was fun. No, it was fantastic.

No, I'm preparing the ground. Yes. So, Noam Brown said: listen, you don't know what superintelligence is capable of. Maybe it could use the CPU heat to exfiltrate to another computer, which brought me back to mine. I am very comfortable now; I could have endless, pointless discussions on this topic. But what's interesting is that you have people who are doing existential risk, saying something they think is plausible, and then you have security professionals who are having this endless discussion, so I think that on some level... these communities are coming together now, and I actually think that was a perfectly reasonable assumption on Noam Brown's part; Sure, you can pick on him, but we all say weird things on podcasts.

I actually believe that the risk of exfiltration is real. I mean, I worked in highly classified, secure computer environments where the covert data channels were incredible. You actually, I mean, these are very real comments, but we're actually having a real discourse, and this was the first time I saw real systems experts having a fairly, I would say, constructive... calculating bitrates and Yeah, it was great, it was a constructive discussion. And, you know, there were typical people there, taking existential risks, they were participating. Of course, it was Twitter, so there were a lot of insults and this and that , but I felt like this was the first time there was a real discussion.

So I hope we see more of this . I hope we see more. You know, I think the lab should talk more about cybersecurity. I think it will engage the community, and once that happens, we can actually do that, Greg has been talking a lot more about this topic lately. Hmm, what was that, what was that good. But I think the main takeaway is that Nam Brown should brainstorm more often on podcasts. I think it just opened people's eyes to the fact that there are actually a lot of security risks that most people don't understand.

Yes. And that means there are more things that you can't create a baseline risk level for, like how does your authentication level work? You can't just brush it off and say it's not important and then mention that, like, " aliens might attack," and that confused me a little. But what, what do you mean, it was like, "well, you know, there's also this risk," and that's how I took it with the heat thing. But at least now we're in a realm where we're comfortable, I can talk about entropy, we can have a concrete discussion, not just "oh, that's super powerful," at least we've reduced it to the laws of physics and the laws of systems.

and most people, I would say, had no idea that such a risk actually existed. I mean, when I was working on Pershing missiles, I had to test video cards and PCs because the Department of Defense requirement was that the screen memory not be retained when the power is turned off, and that was instantaneous. That is, the moment the power went out, the image in memory would disappear. And if there was a delay of at least three seconds. Yes. Oh, you, you could have read that. Oh, you see, we had people come into our offices and literally measure the distance between monitors through TEMPEST attacks, which is a 100% surefire way to use electromagnetic radiation to leak information.

I saw the same thing with the BIOS's spectrum expansion. I saw it from, I saw it from the audio speakers, we had to remove them because it's a very powerful channel. Our building, our building, these are all very real things. Our building just had music speakers aimed at the windows, just to create interference . You should go work with security in one of these labs. I've never seen you so engrossed as you are in a warm, you know, connection. Can I tell you about the craziest hidden channel I've ever seen?

So, remember the old CRTs ... I know, it's like one of mine. So I'm glad someone is older than me. Actually, no, but I act like it. So, remember the old CRT monitors? It turns out that if, say, it's nighttime and you're using a CRT terminal in your room, the brightest thing in the room is actually the pixel that the raster beam is currently on. Most people think it's the monitor's glow, but it's actually that particular pixel. Mhm. So, someone figured out that if you're, say, in a hotel room and you're working on a computer.

If you have a device that can read color through a window, you can reproduce the screen. Oh, this is crazy, right? You just do it at the same frequency that the raster beam moves. Then someone else figured out that if you can swap three pixels, you can use them, because they just look like dead pixels. You can use them to actually send messages. You could literally sit anywhere, read the messages, and it's a relatively high-speed, one-way connection. So what Noah Brown was talking about, maybe warm—it's not the way, but that level of complexity really exists, it's a real thing.

When I was at the rocket factory, I had to lock the keyboard. I...that's actually a rude word, but that's what we called some...I had to lock the keyboard at night because they didn't want the cleaning people who didn't have clearance to walk by and notice which keys were dirtier or cleaner. And one day I left it, and I found a note from security, you know, demanding that I report to security and pick up my keyboard. The security guard walking the hallways simply took the keyboard off my computer that night.

And this despite the fact that I didn't have permission. I was just such a sensitive element, you know? I was nobody. I was an intern. The big problem with this conversation is that now all of this will end up in the training data of every AI model in the future. So but this is also a threat model. Well, that's the threat model for these things: there's always a trusted party and an untrusted party. NIST has 500-page manuals where for every untrusted party you essentially assume there's an oracle that can know and do everything, and then the question is, can you get information from the trusted party, which is what Noam was talking about, by the way, which is again quite a lot.

Which I think raises an extremely interesting question, which I'll get to: humans can't get the gist and they use confusing terminology, and in fact AI can test all of these things in a very short amount of time. Yes. Aha. It doesn't get tired, it doesn't get bored, and you know, the interesting thing is that now there's a whole layer of security where you have to check every API, every internal service in your network , because nobody thinks that an internal GitHub, Slack, or financial tool is vulnerable to DoS attacks, but swarms are different.

they literally look like a DDoS attack. Yes. So now we need a whole internal layer that will track much more: what authentications are being performed, what API calls are being made. Yes. And this will become a basic requirement, and all the old security experts write to me: "Why are we even explaining this to everyone ?" It's so basic because no one has done this inside companies. Yes. And well, you didn't have to worry about your people, and that's the whole point, but now your people are just software.

Yes. And the type is unlimited, and has a credit card. Yes. I mean, we've survived in information security to some extent because most people do the right thing 95-99% of the time . Oh, wait. The number of employees who are perpetrators is one in 10,000. Yes. Yes. Yes. Yes. So all of these systems are essentially open to anyone who wants access, or with a tap on the shoulder—and access is gained, and swarms of agents completely change that, because they're just like roaming drones. Yes. And that's multiplied by 10,000, and they easily confuse a good task with a bad one and vice versa.

So, data security in our systems is a moment for a huge upgrade. I, I actually, Marthe, think we're going to need a different access and security model. Yes. In the future. In which direction will we move? Because the model we have is not detailed and productive enough to handle this. So, I want to take a step back. I want to express a meta-opinion: I believe that this is how these conversations should happen . We have identified a new risk, like cybersecurity, for which we have evidence, and now we are discussing a solution.

I think the whole discussion around AI can look like this, and the biggest mistake is that it was n't. I think the whole industry and the community is very excited to interact in this way, and I have something to say about exactly what you're asking, but I think that's where the conversation should be. So we are known for having healthy discussions that everyone should learn from, and that is what we do. So, here's the thing. I don't think there is a technical limitation here. These threat models are well studied and have long been described in the literature.

I mean, operating systems research, multi-layered security (MLS) looked at these things from an academic perspective. The reason why this was not implemented was usually due to usability issues . It was just really hard to maintain, and you weren't obligated to do it. So it can be argued that AI solves the usability problem because AI is the one using it. Perhaps now is the time for a renaissance in operating systems , networks, and computer languages, and we should go back to old research and start rebuilding systems that are secure by design.

And by the way, if we don't think these things are safe to release, we don't release them until we build these systems. And by the way, AI is very smart, so it can help us build it. And so I think again, computers are always evolving; Do you know how much of the stack we had to change for the internet? everything is accurate, tell me about it And you know how vulnerable everything was and how vulnerable it wasn't. We might find ourselves in one of those moments where we're like, " Oh my god, we need to rethink everything," and that's okay; we've done this before.

But, but I think we should discuss this, so I agree that it's time to think about the development of these things. Well, look, you mentioned earlier about booting up a PC and catching a virus in 30 seconds or something. If you look at how you take an iPhone out of the box, which many people are doing this week, you know, what happens is the entire network is effectively blocked except for getting the latest version of the OS, because even if it was packaged 6 weeks ago, you know, some zero-day vulnerability could have been discovered since then.

So in effect, the entire first-time setup process now involves an update , during which the device can't do anything else until it's updated. These are the things, completely innocent, that we turned off in all that software from a bygone era that was once considered useful. For example, macros in Word to create automatic quotes were super cool until they became a virus. We had the ability to insert a CD and it would automatically run a program, and then someone could make a copy of that disk, replace the program with a virus, and it would look like a regular program that collects data and does damage.

Yes. Then we turned it off. And here's what's happening now: there are a whole bunch of things that are happening on your corporate network that actually need to change as standard procedure . Two-factor authentication wasn't standard in most places five years ago. This whole SaaS world, I remember back in 2015, when you talk to a new company about enterprise pricing, they realized that the first step had to be to integrate Okta or Google Auth, because they couldn't have their own catalog to manage, and that became the norm.

And now there is no SaaS application that does not start immediately with managed authentication, right? So there are so many things that have to happen before you can even start working with the software now. Yes. Well, yeah, we...I mean, I guess every level of the stack has to evolve a little bit on this issue. For example, even the lack of granularity, you know, these modes where the agent either asks you every time if you want to give it permission to do something , or the exact opposite—it can just wipe your entire computer, right?

And our OS was probably not built for the level of granularity of tools you want to provide to an agent. We've done a lot of work in this area, because of course, do you want to give an agent access to, say, your entire file system? Probably not. Maybe in some cases yes, but often you want to have detailed control: say, you can read and write in this folder , and read only in that one . And how to make all this intuitive for the user? This is very complicated, so what you just said is a very profound observation.

I know, for sure. Yes. Which is effectively the conclusion of 40 years... You can't do this. But maybe with artificial intelligence you can, maybe there's some... If you had asked me when I went in what I wrote down in my notes as my biggest fear. Yes. This is what Europe will decide that GDPR was the best invention in the world. Ugh. And they will simply apply GDPR to AI. Yes. And everything will be fine with AI itself . He will have one. It will have one query when the text is generated: " This provider is producing text, and it is probably erroneous, yes or no?" This will happen because you will not be able, at least in North America, to send your speech.

In Europe they will still do it, and they will have filters, keywords and blacklists. But then, with any verb, every time an agent—be it backend or frontend— touches a third-party product , I'm really afraid that they'll just say, "We need a GDPR request for this," and that's it. Oh, back to the user agent. And every record, or every operation that's not a regular search... yeah. It becomes like a safety warning, like the airbag in your car . And regulators love it because it's a division of responsibility, and so it has a kind of legal precedent.

and I'm really worried that this is the very compromise we're going to come to, and unfortunately, because the US stopped being a leader in antitrust regulation of technology about 15 years ago. The problem is that Europe will lead in this because they have nothing to lose . Yes. And I don't want to get upset about it, but I just can't get the thought out of my head that they love requests. I mean, look, I had to implement that browser choice. No, they...they...because they're the same thing.

Look, get in the new car. Yes. Which I haven't done in years, but , you know, you peel off stickers, you have...well, you know, a bunch of stuff like that. And some people think it's a success. Yes. And it's like...has anyone ever read what it's like if you have a child in this chair ? And it's like, well, it's relevant to some people sometimes, but it's a whole fabric. It's attached to the seat, and they love it—it's a very specific thing that they just love. So I would say if I were an AI right now, the one thing I would try to avoid, and look, we added that.

Okay, FINRA for AI, we're going to create this standard. I mean, look, we had to implement this...when the internet was new, the main thing was to download a program and run it, and of course, if your machine is running as administrator, it was: download a virus and lose all your files forever. And so with Windows XP, which came out in 2000, we added this thing that would prompt you and stop you, literally your machine would stop— User Account Control—and it was a real nuisance. We also did this in Word; a stupid little macro to help you write your diploma also came with a warning every time you opened the diploma saying it contained mine.

everyone just pressed. and everyone...so you find yourself in a world where, like with GDPR, everyone just doesn't care. Yes. and then they say, well , this has to be bigger. although Mac, I mean , Mac to some extent does. nobody downloads software, that's the thing, on Mac it's a completely different usage model. so I would like to. I don't have, well, I don't even have 10 apps on my Mac, but hey, it's 10 and I'm done. Yes. And this...But a lot of people. still doing all this.

Imagine if this happened every time you visited a new website. Yes. What are you doing now, because that would be bad. Yes. I mean, to start from here, but to bring it back to the macro level, I would like us to discuss this very thing , which is, I think we've already dealt with a lot of these issues. I feel that when we talk about philosophical existential risks , we are not addressing these very pragmatic issues. I actually think this is a constructive conversation. Perhaps assistive devices would help.

I don't know. And I think part of the problem is that people don't remember what unrestricted access was like, how bad it was , and how relatively safe it ultimately turned out to be. For example, I even remember how at Stanford during our graduate school the oscilloscope worked somehow strangely. I'm like: what's going on with this oscilloscope? He, you know, slows down a bit . I measured network traffic and it was higher than expected. Um, and I'm like: why is there a network here? I didn't know there was a TCP stack in this thing.

Someone hacked it because it had an old version of Windows CE on it, and started a porn server, you know, and that's it. I noticed that previously there was nothing for the protocol. It used to be that wherever you looked— Yes. Yes. Someone broke in somewhere, and was that the worst-case malicious scenario? But in reality it is very rare, although there was a possibility. So if we could somehow tone down the rhetoric and put it in context, these are still computer systems, and yes, there are very serious things, people have definitely died because of network outages, there have been real problems, but can we just argue about GDPR?

That would be great. But the problem is that this is not what the discussion is about. This is n't about GDPR and prompts; it's about the extinction of a species and. philosophy and philosophy, and undeniable things, and it's simple. very soon, I think that's a great argument, and I think you hear people saying now that we're regulating airplanes and cars, and you forget that the first cars came out at the turn of the century and "Dangerous at Any Speed" came out in the mid- 1960s, completely.

And, you know, people were selling pharmaceuticals during the gold rush, and then 50 or 75 years later there was thalidomide, and not even in the US, and then the FDA came along. And, you know, the first pilots flew, of course, in the early 20th century, and it wasn't until the 1920s that you had to get a pilot's license, and you literally flew in your own plane and got a certificate , if you had one. It was like getting, it was literally no different than today's driving courses, and then for another 20 years no one checked the airworthiness or inspected your plane, it was minimal.

And it wasn't until much later after World War I that they started doing what you would consider the modern FAA. So we're talking about 40 years, right. innovations. And they didn't move slowly. I mean, if you saw that black and white video where all sorts of planes were crashing, and it was all happening 20 years after the Wright brothers. and it's incredibly efficient, and and and yeah, it's incredibly, it's the safest mode of transportation, you know, and so I really think that this process is also the slowest and most difficult form of innovation.

And so if you started with an F, if you founded the FAA in 1910, you would never, never, never...Well, I was listening to Nick Bostrom's podcast just a few days ago. No, no, I... that was, that was interesting. That was interesting, and but no, but he actually substantiates his point. If you regulate AI too early, you are effectively solving nothing and the risks ultimately remain the same, but you don't understand the situation well enough yet. You are bringing something to life , but you haven't yet learned to control it.

Well, we still have to figure out what it really is, new things are coming out all the time, and we perceive AI very differently than we thought even six or nine months ago. And I think all of these innovations that are going to happen at the application level are going to make things move in and out of the model in a very different way. And we thought, up until last week, I think text requests and text responses would be the best way to interact with... I know, then Jeff and then Jeb Chef are so good too, and then so let's talk about this , I think why you find this so remarkable.

Yes. Well, okay. So, the way I think about it: LLMs worked on the principle of "text in —text out," right? They generate text, and they originated from chat rooms, right? It was for communicating with a person. And we've spent the last few years trying to take this text-producing thing and squeeze it into a traditional program, right? But traditional programs don't really " speak" text , do they? And so you end up doing something clumsy, like writing in the query: " Here's the schema." Here is the diagram.

But the thing is, the model generates text and often ignores it, so everything works quite crookedly. And what Jeff actually said is: listen, text generation—it's very expensive, and it's more complicated than you need it to be. So why do n't we do this: we'll read the text using all this knowledge, but instead of generating the text, which is very expensive, we'll just, if you give us a set of options, choose the best one. We can do this incredibly fast, incredibly cheaply, and also much more accurately because we can train a model specifically for this.

And for all usage scenarios where it's not about a chatbot but about integration into traditional software, this is a great option. And this was probably the fastest implementation of an AI model after ChatGPT. It's just amazing because we were all ready for this. I just want to add to that because I can't express how much I love seeing this form of innovation. Because it does something that has annoyed me from the beginning: there has been no user research that proves that interacting with a computer using full natural language is effective.

Mm. This is literally always the least effective way, and it's very simple: ask yourself how many people actually know how to ask questions, and it immediately becomes clear that less than half of people can ask a good question in a meeting. Yes. And how often do you look at the answer and feel annoyed before it's even finished, but you have to pay this money to watch these seven paragraphs come out and then apologize for it being just a little bit—so having a different model—is cool. And of course, my favorite is the result designed for probabilistic programming.

Yes. Instead of saying "if this is a support question, then direct it to support, otherwise to the general help desk", it looks like "well, this is 80% a support question ". And this is exactly modeling. It turns out there are about 50 years of research in computer science devoted to literally probabilistic if constructs. And suddenly the most interesting area in computer science becomes probabilistic programming, which was the whole field in the 60s and 70s. This was a major undertaking, because all computers started with mathematics, and it was all modeling.

For example: let's launch a rocket and hit the target, but the wind is blowing. But the wind is not constant. So let's simulate the wind and decide where to point the engines to calculate the trajectory. And most of the programming up until 1970, before accounting came along, was probably... And then we screwed it up. No, but there's no probability in accounting , right? However, most programming was just such simulation. So most programming language developers have struggled to figure out how to implement probability into if-conditions or while loops.

For example, do this... until something happens, maybe in most cases. In my first year of CS, the second assignment was a simulation of a line in a store, although I did n't realize it at the time. I learned all this when I read about Jeb, the professor wrote a book called Modeling Theory back in 1960, and I didn't even know it, because by the 80s it was already extinct, replaced by hypertext. And this concept of probabilistic... That presentation you sent about the future and what's special about language models was very apt.

Oh, Halpern is phenomenal, Thomas is great, but I was missing the thought: " Wait, this isn't anything new." The entire computer world was built on these probabilistic things. It will be very interesting to " shake the dust" off these developments, because that is exactly what is happening now. Now if—it's not always, but with a certain probability of X%. Jeb works like its own programming language: here's a prompt, return a percentage. Yes. And then you put that in an if statement. But the most important thing is that we finally have a way to integrate these language models into traditional software, and it's kind of funny because it begs the question: why have n't modelers done this before?

This is not an accusation, but rather a reflection of how they think. They are trying to create creatures, and the creatures talk. If you're trying to create God, God speaks in natural languages ​​or something, whereas this is something for traditional software, and that's not the direction they've taken at all. But one of the reasons for such a sharp rise is that many of us software developers have tried to integrate these models into software. But nothing worked. So even before you get to the probabilistic aspects, for example, if I want a language model to handle an "if" statement, that's very difficult today, and with this model it becomes much, much, much easier.

And of course, this could fundamentally change the nature of software, making it more stochastic. I'm sure it would, and the cool thing is that it's happening outside of the models themselves, because that's exactly what I think should happen - the center of innovation has simply shifted. Yes . And now people need, it turns out, I mean, outside of the labs, the big platform providers — 100% you know, they're reaching a tipping point where innovation stops happening at the platform level. And then, you know, Apple, there's a famous expression in the Apple community called " Sherlocking," where Apple looks around and extraneous things become their features.

People complain, and that's a real [], but that's how innovation works. Because when you become a platform, you are overwhelmed; no matter how many people you add, you are constantly busy maintaining stability, compatibility, etc. So I think this is a signal that people have understood where innovation needs to be implemented right now . This is great. For the model, but beyond it . Yes. Yes. Guys, thank you for coming. It was wonderful. Okay, great.

Go deeper

• All A16Z episode guides