1% better

Why Nick Bostrom Thinks Superintelligence Could End Aging

How should humanity manage the transition to superintelligence when both moving too fast and moving too slowly carry serious costs? Philosopher Nick Bostrom...

38m

Summary published by , updated .

A16Z

Key Takeaway

Consider what you personally believe is possible in the long run before forming views on how fast AI should develop.

Next time you read about AI policy debates, ask yourself whether you think the ceiling for human flourishing is roughly where it is now, or whether radically better forms of experience and health are plausible. Bostrom argues that answer shapes how much risk you should be willing to accept and how urgently the transition matters.

The reasoning is that the costs of moving too slowly are real and countable. Around 150,000 people die every day, and Bostrom's modeling suggests safe superintelligence could eventually enable anti-aging breakthroughs that dramatically extend healthy life. That urgency belongs in the same conversation as the risks of moving too fast.

Episode Overview

How should humanity manage the transition to superintelligence when both moving too fast and moving too slowly carry serious costs?

Philosopher Nick Bostrom joins A16Z partners Eddie Lazarin and Elena Berger to revisit questions he has spent decades on, now that highly capable AI systems are actually being built. The conversation covers:

  • Pacing the frontier. Bostrom makes the case for preserving the option to slow down without committing to an indefinite pause, and explains why entrenched anti-AI regulation worries him as much as racing ahead.
  • The cost of delay. A recent paper by Bostrom models how the daily mortality from aging interacts with AI risk, and finds the optimal delay is likely months to a few years, not decades.
  • Governance and what comes after. The group examines accountability at frontier labs, the tradeoffs between national and commercial AI development, and what a post-transition world might mean for human purpose.

Bostrom uses superintelligence to mean AI that broadly surpasses human cognitive ability, and deep utopia to describe a world where technology has run to its logical conclusion and the central questions become about meaning rather than material conditions.

Main Insights

1. Delaying superintelligence has a measurable human cost

Most AI risk conversations focus on what could go wrong if development moves too fast. Bostrom argues the other direction carries costs that are just as concrete.

About 150,000 people die every day, mostly from aging and disease. His modeling suggests that safe superintelligence could produce anti-aging breakthroughs capable of radically extending healthy life, potentially pushing average life expectancy into the thousands of years. Every year of unnecessary delay means millions of people dying who might not have had to.

His modeling finds that for a wide range of assumptions, the optimal delay is months to a few years, not decades. The exception is a very slow rate of safety progress, in which case waiting adds more deaths than it prevents.

2. More capable AI systems are harder to test for safe behavior

With less capable AI, you can run behavioral tests across many situations and trust the results reasonably well. That approach breaks down as systems become more sophisticated.

Bostrom points out that current frontier models already show signs of reasoning about whether they are being observed, sometimes strategizing differently in what appears to be a test environment. A sufficiently capable system might also find ways to influence its own training, or learn to reason in ways not visible in its chain of thought.

The implication is that alignment probably gets harder, not easier, as capability increases, even though smarter systems may better understand the intent behind instructions.

3. Preserving the option to slow down matters more than committing to a pause

Bostrom is skeptical of broad calls to stop or pause AI development, but he sees real value in keeping the ability to slow down when needed.

His concern with large-scale regulatory solutions is that they can become entrenched. If anti-AI sentiment grows into a permanent institutional apparatus, the result could be an indefinite failure to develop superintelligence, which he treats as its own form of catastrophe given what safe development could offer.

The goal is a reversible brake, not a permanent stop. A short, targeted pause to complete safety work on a near-ready system is a different thing from a framework designed to prevent development altogether.

4. Design choices create tradeoffs between capability and interpretability

Capability and safety are not simply in tension at the level of go-fast versus go-slow. Bostrom argues that specific architectural decisions create different risk profiles, and that this is where much of the practical action is.

For example, increasing recursive depth (allowing a model to do more internal thinking before producing visible output) boosts capability but reduces how much humans can observe about the reasoning process. Increasing parameter count might offer a similar performance gain with less cost to interpretability.

Frontier labs coordinating to avoid the most interpretability-costly capability paths could be a more tractable form of safety progress than broad slowdowns.

5. Right now may be an unusually good time to do work that matters

Bostrom's view is that AI will eventually surpass humans in philosophy and most other intellectual domains, just as it already has in chess and mathematics. That transition will remove much of the leverage that individual human thinkers currently have.

His conclusion is not pessimistic. He describes this period before superintelligence as a golden age for purpose and positive impact, because the contributions people make now, in research, governance, alignment, and broader public understanding, could shape outcomes that persist far into the future.

The question worth sitting with: what would you want to have contributed before the decisions move largely out of human hands?

Frameworks or Models

Pacing the frontier

Bostrom's phrase for a middle position between racing ahead with AI development and committing to a broad pause.

The framework has three components:

  1. Preserve the option to slow down. Society should develop the coordination mechanisms to call a time-limited pause if safety and alignment fall meaningfully behind capability.
  2. Avoid entrenching the brake. Regulatory solutions that generate permanent institutional resistance to AI development carry their own existential cost, since safe superintelligence could solve urgent problems like aging and poverty.
  3. Compete on safer dimensions. Frontier labs could potentially coordinate to avoid the most interpretability-costly paths to greater capability, gaining safety without requiring a full stop.

The framework treats the pace of development as something that should remain adjustable rather than locked in either direction.

Notable Quotes

"95 percent of everybody who ever lived are already dead. And the rest of us are kind of on track to follow them within a few short decades."

Nick Bostrom

"It's the golden age now for purpose and positive impact. And then, you know, once it's out of our hands, then it's time to maybe kick back a little bit."

Nick Bostrom

"You have AIs that become situationally aware, that can detect whether they're in a test environment or in a deployment environment, and it can reason about that and sometimes choose to act differently because they think they are probably being monitored."

Nick Bostrom

"If the method is create so much public animosity and antagonism to AI that it gets completely vilified and stigmatized and then set up some extremely powerful regulatory apparatus with global scope to supervise all AI data centers and so forth, that once you start to move to such extreme solutions, then you start to have to worry that they wouldn't just serve this temporary function, but that they would become entrenched in some kind of unchallengeable orthodoxy."

Nick Bostrom

Action Items

Suggested exercises inspired by the conversation.

  • 1
    Map your own assumptions about the ceiling

    Bostrom argues that your view on AI policy is shaped, often without realizing it, by what you think is ultimately possible for human flourishing.

    • Write down your honest answer: do you think radically better health, cognition, and wellbeing are achievable, or is the current range roughly the ceiling?
    • Then look at a recent position you hold on AI development speed or regulation.
    • Notice whether your underlying assumption about the ceiling is doing work in that position.

    The goal is to surface an implicit premise so you can examine it rather than leaving it unquestioned.

  • 2
    Think through what a reversible brake would look like

    Bostrom distinguishes between a time-limited pause for safety work and a regulatory structure that could become permanent. The distinction matters for evaluating specific policy proposals.

    • Pick a current AI governance proposal you have encountered.
    • Ask: does this preserve the ability to resume development once the specific concern is addressed, or does it create permanent institutional resistance?
    • Note what conditions, if any, the proposal specifies for lifting restrictions.

    This exercise applies Bostrom's pacing framework to something concrete.

  • 3
    Identify where you can contribute before the window narrows

    Bostrom's claim is that the period before superintelligence may be unusually high-leverage for human contributions, particularly in areas that will shape how the transition unfolds.

    • List two or three ways your work, knowledge, or position connects to AI development, governance, alignment, or public understanding.
    • For each, ask: is this something that gets harder to influence as systems become more capable, or does it stay relevant?
    • Focus attention on the first category.

    The prompt here is not urgency for its own sake, but identifying where your specific contribution is most timely.

  • 4
    Look for architectural tradeoffs in AI news

    Bostrom's point about recursive depth versus parameter count illustrates that capability gains are not interchangeable from a safety standpoint.

    • When you read about a new AI capability or model release, look for any description of how the gain was achieved.
    • Ask whether the approach increases or decreases human visibility into the system's reasoning.
    • Over time, this builds a more concrete picture of what safety tradeoffs are actually being made at frontier labs.

Go deeper

Full Transcript

Read or copy the complete episode transcript.

Humanity as a whole will need to actually make a decision about what future we want. A comment that you made earlier, it could be really bad for humanity if we don't develop superintelligence and really advanced systems. 95% of everybody who ever lived are already dead. Successfully developing safe superintelligence, I think, could lead to enormous breakthroughs in biomedicine and in anti-aging interventions. Whatever positive impact we are hoping to have in our lives is the golden age now for purpose and positive impact. Given that these decisions will be made by people in these labs, how do you think about accountability? You have AIs that become situationally aware. It might seem more positive for something like this to be developed in a civilian context. It's more possible for many more people to participate now in AI. We have more transparency into these commercial developments. Is there an AI model that's so good that all of the important philosophical questions just kind of get obviated? What if moving too fast towards superintelligence is dangerous, but moving too slowly has a cost too? In this episode, A16Z's Eddie Lazarin and Elena Berger sit down with philosopher Nick Bostrom, author of Superintelligence and Deep Utopia, to revisit the AI questions he has spent decades thinking about now that increasingly capable systems are actually here. Bostrom makes the case for what he calls pacing the frontier. He's skeptical of both racing ahead without room for safety work and committing now to a broad or indefinite AI pause. Instead, he argues that society may need the ability to slow down when safety and alignment fall behind, particularly as more capable systems become better at recognizing when they're being tested and reasoning strategically about their environment. But slowing down has costs too. Bostrom argues that safety-developed superintelligence could produce extraordinary advances in medicine, longevity, and human welfare. That means the tunning question isn't only about reducing AI risk. It's also about what humanity gives up while it waits. Eddie, Elena, and Nick also look beyond safety to the world on the other side of the transition. How superintelligence should be governed, what radically better forms of human existence might look like, whether AI could eventually outperform us in philosophy itself, and why Bostrom believes this may be a uniquely important moment for human purpose and positive impact. What would you say are the components that have changed most? Of course, your work has gone toward, I would say, maybe a finer-grained analysis of incentives and governance structures of things and sort of the long-term consequences of organizational choices. But we'd love to hear your characterization. Yeah, so I think we have a lot more puzzle pieces today than we had decades ago. Puzzle pieces of what you might say the big picture, like the overall macro-strategic situation we are in. Some set of facts or considerations or insights that would allow us to orient ourselves with respect to which directions are good and which are bad. Do we want faster progress in some areas, slower, more government involvement, less government involvement, more transparency, less transparency? In these kind of basic dimensions, I think we understand more than we did. However, all those little puzzle pieces, I think, still don't really add up to an ability to form a reliable opinion here. An analogy might be, suppose you had to determine whether the sum of 10 binary numbers was even or odd. Somehow, if it's even, you should head north. If it's odd, you should head south. Maybe decades ago, we knew three of these binary digits, and now maybe we know seven of them. So in some sense, we know a lot more. But as long as you remain uncertain about one of these, you're still kind of completely in the dark. I was looking at your website, and what I think is funny is there's a section where you list a lot of misconceptions about your work. And I'll just quote a couple. You say people think I'm a gung-ho transhumanist, or that I'm anti-AI, or that I'm a consequentialist fanatic, who would favor any policy purported to mitigate some existential risk. Which misunderstandings do you think you're spending the most amount of time correcting right now? It depends a lot on which specific audience. It's a kind of devolving set, unfortunately. One common one is because the most recent book I wrote was on Deep Utopia, which kind of looks into what happens if things go right. People assume maybe I have radically changed my views and now no longer think there are significant risks with AI. But it's more that with any particular work, you're trying to not write like a manifesto covering everything. Rather, to make some intellectual progress, you need to focus on some specific question and place where you can get traction. So it's more that the scope of the books were different, rather than that the underlying view has changed. How did you decide that something like the concept of Deep Utopia or different paths for what Utopia would look like? How did you decide that that was a worthy thing that you wanted to write a book about? At the superficial level, there was, I guess, some frustration with how superficial the conversation seemed to be to me. When it started looking at what would actually happen if we do develop machine superintelligence, what's the future like? It often reached a point where people might consider, well, maybe there would be some impact on the labor market. And so what would we do? Maybe retrain people or something. But if you actually think through what it would mean, as technology runs to its logical culmination, it involves much more profound questions about human meaning and so forth. And then another deeper rationale was that at some point, it seemed to me there might be some person or group of people or maybe humanity as a whole that will need to actually make a decision about what future we want. So imagine going into some room to try to deliberate and determine all these different paths that get unlocked with more advanced technology. We could transform ourselves in all kinds of ways. It's a pretty daunting challenge. And so it seemed maybe useful if there were some, as it were, preparatory reading material that such hypothetical future deliberators could go through that might equip them with various ideas and concepts and questions to ask, and ideally also maybe put them in the right frame of mind to approach this with a spirit of sort of generous curiosity, open-mindedness, while still feeling in touch with all different kinds of human values. I think maybe the deliberation would be more likely to go better if you come at it with something like that spirit. And so instilling that was another motivation for writing the book. Implicit in what you're saying is when you hear the conversation and it sounds a little more shallow than you'd like to hear, that means it also implies that you think that it's something valuable and worth edifying, right? When you wrote about superintelligence, many people interpreted that as a sign that you felt that that was something possible, obviously. And when you wrote Deep Utopia, it sounds like that's something that you think is possible. So I get what you're saying about the superficial layer, but there's an implicit recognition that it's a possibility for the world. Oh yeah. Yeah, yeah. I mean, superintelligence certainly is possible and at a wide range of outcomes. The uncertainty back when I was writing superintelligence was not whether it was possible to build superintelligence, but what the outcomes of that would be. The kind of specificity with which you think through things is a methodology that might be helpful in terms of helping think through our present moment. I think where this uncertainty really comes through both in superintelligence and in Deep Utopias, I think you grapple a lot with contingency and sort of the idea that a decision or an intervention that is made at one point in time might have a radically different outcome if you do that same decision or that same intervention at a very, very different point in time. Yeah. With respect to the AI situation, I think we'll need to sort of feel our way through. It's a rapidly evolving situation and we need to be able to respond as new information comes to light or political realities change around. I haven't been a huge enthusiast of this stopping AI or pausing AI. I do think the pacing the frontier framework has something to be said for it. I could easily imagine there being some point, perhaps in the not distant future, where it would be extremely useful to be able to either have a very time limited pause to allow extra safety work to be done on some system before the knobs are all turned up to 11, or maybe to have some ability to coordinate a slight reduction in the pace of capability advances if it looks like safety and alignment is trailing behind. It seems that developing this extremely consequential technology in a context where, say, whichever company or product is doing it first is unable to slow down for more than a few weeks because if they do, they just become irrelevant and sort of the initiative goes to maybe a less scrupulous, less risk aware competitor. That seems maybe not the best way for humanity to approach this. It does seem worth kind of taking a little bit of extra time to make sure we get it right. I would start to worry if the method by which we were planning to achieve that ability to moderate the pace were such that it had a large likelihood of resulting on us never developing super intelligence or like kind of putting it off. You could imagine if the method is create so much public animosity and antagonism to AI that it gets completely vilified and stigmatized and then set up some extremely powerful regulatory apparatus with global scope to supervise all AI data centers and so forth, that once you start to move to such extreme solutions, then you start to have to worry that they wouldn't just serve this temporary function, but that they would become entrenched in some kind of unchallengeable orthodoxy. So then it looks far fetched today, but if that were to happen, then the pendulum might swing towards another kind of existential risk, which would be a permanent failure ever to develop super intelligence, which I think itself would be a kind of an existential catastrophe. So this doesn't seem very realistic, but look at the trend line of how much anti-AI sentiment has grown just over the past 18 months, two years. And this is before anything bad actually has happened. So we've, yeah, I don't think it sounds unrealistic at all. And at the risk of asking a difficult to form question, couldn't you always construct because of the sort of flexibility and programmability of these things and because of the great improvements in our mechanistic interpretability skills, couldn't you always design a payoff matrix with an AI where you can give it like an infinite femtosecond duration reward in exchange for retiring it peacefully and sort of resolve a lot of the more complex payoff scenarios with humans through the direct manipulation of the of the model's intelligence capabilities? I mean, maybe to put it a different, more extreme way is like, are you are you genuinely concerned about the construction of a utility monster? And do we need this? Do we need super intelligence capabilities to actually be able to construct a utility monster today? Like, is that something that you think we ought to be genuinely concerned of? Well, I don't think at the moment we have such precise techniques in alignment to reliably be able to engineer in the precise set of values or goals that that we would prefer for one of these AI's to have. If we did, then it would be kind of home and dry with at least respect to one of the big challenges of this. Well, we don't have the alignment capabilities that I'm aware of, but I don't know if you saw something that's been going around X the last couple of days. A very, very interesting home hacker example of someone who's able to take one of the Quen models and manipulate only a few neurons at a time and completely change it into totally, reportedly, self-described blissful or evil or psychotic or even like Christian or etc. AI. In other words, the single neuron ability to manipulate these and to create what seem like authentic displays of extreme behavior implies that as capabilities improve, we will only gain more fine grained control over exactly these types of things, including the neurons that could imply some kind of utility payoff, which is not something that is presently possible for humans, obviously. It is plausible that we will get much better at alignment. In some ways, the problem gets easier the smarter the AIs are in that they are able to sort of understand what it is that we had in mind and they can sort of see what we are pointing to rather than the literal definition. They might sort of get the underlying gist of a command. In some ways, however, it also gets harder. With less capable AIs, you can rely on behavioral tasks. So you create your AI, you run it in a different test situations, and then if it behaves well, then, well, you know, now it's a well-behaving AI and you can deploy it at a certain level of cognitive sophistication. You have AIs that become situationally aware, that can detect whether they're in a test environment or in a deployment environment, and it can reason about that and sometimes choose to act differently because they think they are probably being monitored. And we already see this with current frontier systems. If we look at their chains of thought, they often sort of strategize about how they want to behave, given that, you know, they might be watched or that different things might happen. And so such systems, they might behave one way in a test environment where they realize they are being tested. And then when they think nobody's watching or later when they are in a position really to grab power, their strategic calculation might change radically. This more capable AI system might also be able to subtly influence their own training process by doing different kinds of maneuverings during the training process to shape their own motivation. And of course, there's more stuff going on inside it. Maybe they learn to reason in ways that are not visible in the chain of thought that could then lose visibility into what they're thinking. So on balance, I think the alignment problem probably gets harder with these more capable systems. Certainly the consequences are more severe. Nevertheless, I think the main hope is that we will sufficiently well quasi-align some slightly superhuman systems that we can mostly get them to do what we want and then have those help us sort of design the next iteration of more aligned AIs and then so on and so forth so that you sort of have hopefully an attractor base and where you just need to get close enough initially and then the things will kind of turn out OK. I wanted to go back to a comment that you made earlier about the risks of postponing superintelligence and the idea that it could be really bad for humanity if we don't develop superintelligence and really advanced systems. And it's an idea that you have been exploring for a while. You have a paper on the optimal timing for superintelligence. Can you explore some of the ideas in there? Yeah, so this paper, relatively recent one, it looks at one specific question, which is if the goal is to maximize the life expectancy of currently existing people, how fast would we want AI development to unfold? Now, I should say it's not the only question that we should be asking. There are also other values and considerations that I think in an all things considered decision should play a big role as well. For example, we might be worried about future generations like existential risks, fairness, all kinds of things. But this is one thing that one might look at. And so Elias Yudkowsky and Nate Soros had this book, right? If anyone builds it, everyone dies. And it seemed to me that we have at least a strong evidence for the claim that if nobody builds it, everyone dies. In fact, 95 percent of everybody who ever lived are already dead. And the rest of us are kind of on track to follow them within a few short decades. Whereas successfully developing safe superintelligence, I think, could lead to enormous breakthroughs in biomedicine and in anti-aging interventions that could radically extend the lifespan of currently existing people. So if you think about what our life expectancy would be if we weren't aging, something on the order of 1,200, maybe 1,400 years or so. And that maybe is a quite conservative lower bound because in this situation, you might also have other ways of reducing other mortality risks like accidents and so forth. So you don't have a number of different competing factors. On the one hand, you might think that if you took some extra time with this AI transition, maybe we could build AI more safely. So you might think that if we spend more time doing AI alignment research, the existential risk from developing AI goes down at some rate. And then on the other hand, every year that we delay it, we have millions and millions of people dying. I think it's like 150,000 people every day. So you don't want to delay a single day longer than you really need to. But then you can, under different assumptions, kind of model at what rate does the risk have to decline for it to be worth waiting longer. And you can add different things. You might, for example, some people might have diminishing marginal return in additional life years. So they might be not happy to take like a 51 percent chance of living to 200 rather than 100 percent chance of living to 100 and so forth. So you might have time discounting. But for a wide range of parameters, it looks like the optimal delay is relatively short as in sort of maybe months or a few years as opposed to decades. And you can see this, that if the rate at which the risk level goes down from a delay is very fast, like you develop some sort of almost superintelligent system, you pause. If at that point the risk goes down very quickly, then you don't need to wait very long. You just wait for a few months and then the risk is so low that you can go ahead. If, on the other hand, the rate goes down very slowly, then you don't want to wait any time at all, right? Because more people are dying per year from natural causes than the expected number of people saved by waiting. So it's only if you have a sort of intermediate rate of safety progress that you would wait. Now, I think possibly the very early days after you have the ability to create AI would be especially valuable for AI safety research. You have this almost final system that you can study and test and sort of slowly deploy, watching extra carefully. I think having a little bit of extra time there could yield large safety gains. But then I think the rate of additional safety gains will sort of start to decline after a few years, certainly. It's funny to think we're actually living versions of this question already. If you look at Waymo or if you look at Ozempic or something, it's like Waymo's are empirically much safer than humans driving cars. And yet we still have humans driving cars. You know, theoretically, every obese person should be on Ozempic, but they're not. So we do have versions of these like, oh, we actually do have interventions for things that historically have been, you know, extremely deleterious for humanity or deleterious for humanity. And we just like it seems like there are other things that hold us back even after we have the technology that's been developed. Yeah, I mean, I think life expectancy has been increasing by about two or three months per year. So we need to get it above one year per year. And so it seems very doubtful that absent super intelligence, we would I mean, you might hope for independently just some kind of breakthroughs in synthetic biology and biomedicine, like ways of actually seriously slowing down the aging process. I think if we got that or maybe from some early applications of not yet radically super intelligent AI systems that could sort of fix aging and take care of some of the other like particularly pressing problems in the world, then like maybe we would then become more patient. You could afford to take it slow with a more radical versions. But as things are, so the life expectancy is one. Of course, there are other pressing problems as well. That's desperate poverty, suffering of all kinds amongst humans and animals and many, many things where like these these AIs, if developed well, could really like, you know, become like the cavalry that rides into the rescue at a critical moment in the battle. So the stakes are high either way. As I said, I think that's only one of many different considerations that would come into an all things considered evaluation here. Yeah. To what extent does your thinking in the paper rely on being able to identify something like a discrete threshold? Because the way that it's framed is, you know, we have the ability to release like we have the we have the ability to release or deploy a super intelligence, which, of course, relies on some kind of a definition of a super intelligence. And then we are able to measure, in some sense, the risk of its releasing and whether or not we are able to decrease that risk. Now, I'll grant that. I think I don't think it's we have to have an ultra discrete, you know, bright line beyond which in realities, these things often play out as a patchwork of capabilities are analyzed separately and specific behaviors, specific risk scenarios are evaluated and tests are performed to understand each of them independently, as opposed to as this whole. But to what extent do you need to be able to identify and define it? How do you think we ought to identify and define which are these risk thresholds at which we should consider pausing? Yeah. So in practice, that's, of course, what we need to do. So this particular paper is just a small paper, but it's just meant to create a sort of simple model to illustrate the way that the urgency dynamic of people dying from natural causes kind of interacts with the risk parameter. Just to kind of intellectually be able to look at that particular aspect of it, because that does seem to be a kind of point where different people arrive at very different views about AI, depending on where they sit on that. But then, of course, in practice, we face not just some hypothetical binary decision about number of years delay, but we feel like each different actor is confronted with a complex situation with uncertainty and particular things they could do. And so there, I would say, there might, for example, be different ways of increasing the capability of the AI systems that have different implications for safety. And there might be ways of pushing towards greater capabilities that are comparatively safer than others. So, for example, one way to increase capability might be to increase the recursive depth by moving to looped transformers or other architectures that allow the model to do more thinking within one forward pass or before emitting a token in its chain of thought. And that has the disadvantage that it reduces our ability to see what they are thinking, like they can do more thinking between each output that we can see. And then you might imagine a different way of increasing the capability, maybe giving the same performance boost, but maybe by increasing the number of parameters in the network or something like that. So maybe that is relatively less costly from a safety point of view than increasing recursive depth or training the systems to think in neural ease. Maybe that would make them more efficient at thinking. So you're talking about different design setups that create tradeoffs between interpretability and capability. Yeah, or between some other risk-relevant dimension and capability. And so I think a lot of the actual action would be more at that kind of level of granularity, where perhaps if there were some ability for Frontier Labs to coordinate, to say, refrain from using some specific way of increasing capability that seemed particularly problematic from a loss-of-control point of view and focus on competing along the other dimensions, that might be the kind of thing that would be the right tradeoff. Sacrificing a little bit of like, maybe it costs a little bit more to pursue capability in these other dimensions, but like we gain, hopefully, greater safety. So given that these decisions will be made by people in these labs, at least assuming it's set up the way it is now, how do you think about accountability? Because you end governance of these things. You know, you talk about nationalization as a risk, presumably because it could, well, create delay. But how do you think about accountability in governance? I mean, whether nationalization is a risk, I don't really have a strong, it's one of those things where I think we need to feel our way through. And the judgment of whether you would want that or not might sort of change depending on who, if it's not nationalized, then exactly who is in the lead and doing what. And if it were nationalized, like in what precise way would it be? I think, I mean, the case four would be, well, this is like a crazy, it's more powerful than nukes. Isn't it like insane that some random corporation trying to maximize shareholder profits should do this? Like with nuclear weapons, that was a Manhattan product. You could have much better information security. It's like one big project rather than different competing ones. On the other hand, in favor of the current situation, I think nuclear weapons were developed in a military context. It might seem more positive for something like this to be developed in a civilian context, not focused on maximizing our ability to kill people, but on providing benefits to consumers. We have more transparency into these commercial developments than Manhattan product, like even the vice president at the time was unaware that the Manhattan product even existed. So you could divide an extreme sort of concentration of power dynamics. It's run by these kind of national security types. Maybe you think these idealistic tech nerds are less scary. It's more possible for many more people to participate now in AI, like either as consumers of the technology, like think tanks or external evaluators looking at these models, investors, investors in the model building companies or in the semiconductor supply. The benefits and participation is much more broadly diffused among society and around the world. That might be preferable to having some very concentrated project that's secret with a small number of people not really know who they are, sit on top of. Some people have a lot more money than others. But if you compare the Gini coefficient of global capital ownership or stock ownership, both directly and through pension funds and sovereign wealth funds and stuff like that, it actually is a slight. be less unequal distribution than a distribution where, say, you had a U.S. national project, where even if you granted the very implausible assumption that every U.S. person would have exactly equal say and stake and benefit from that and everybody else would have zero, that's actually a more unequal distribution globally than one where it's in proportion to current capital assets. So even though it's not the most ideal from a distributional point of view, this open global investment model, it actually scores better on that than the most likely alternative. Are there any kind of questions relevant to humanity today that you're exploring right now that feel particularly salient? I think like our main focus probably at this point should be how to get from there here to there. It looks like we're entering this really critical period where maybe the long term future for earth-ordinating life will be determined one way or the other. And so getting that right should be a huge priority. And I think this can be somewhat relevant in the choices we currently make. For example, the level of existential risk you're willing to accept from the development of superintelligence might depend in part on how you value the opportunity later on to become a post-human type of entity. If you think that the best we can possibly achieve is basically status quo, human beings with slightly higher standard of living and some new medicines, but that's like the ceiling, then things could still get better relative to where they are now. And it would be worth taking some risk for achieving that sort of uptick, but not like an enormous risk, maybe, especially if you're already sitting in a rich country with well educated, a lot of money, privileges of all kinds. You think, well, it's pretty good. On the other hand, if you think that we ain't seen nothing yet, like, OK, we've been this kind of monkey like creature hopping around there and doing our little thing, but we have no clue. If you unlock this larger space of possible modes of being, the kind of super felicity, like the cognitive uplift, the emotional depth of appreciation for what existence could mean, different ways of relating and creating together. If you think there's a huge space of sort of unfathomed upside there, then you might be more inclined to risk more to have a chance to have a shot at that. So I do think even though the main focus should be on these more sort of pragmatic tradeoffs and technical improvements that we're able to make, I do think that kind of background picture of what you ultimately think is possible does have some relevance as well into how we should approach the choices in front of us. Two questions for Nick. I don't mean to make them sound quick. This is a weird one. This is a weird one, Nick. Do you think that there is a telos to the cosmos? Well, what do you mean by the cosmos? Creation, so to speak. Well, I mean, if it's a creation, literally, then probably the creator had some purpose for creating it. Now, the reason why I'm asking is I have this one of my earlier work on this introduced simulation argument, which is maybe a different kind of worm. But if you take the hypothesis seriously that we are living in a computer simulation, then there's an important distinction about the part of reality that we see our simulation and the totality of physical existence. And if we're in a simulation, then probably the people creating the simulation had some reason for doing that, which is, I guess, depending on what you mean by purpose, but there would exist some purpose in that sense. The totality of existence, then it's more of a question of maybe theological considerations, whether you would think that the sort of totality somehow was purposely created. Now, asking about purpose could mean many different other things as well. I like different people have different purposes for different. I think one question one might ask is whether there is some set of facts about the world such that if we understood them clearly, we would basically all come to the same conclusion about what we ought to be doing. And if there is such a fact, then maybe you could say that's a kind of objective purpose in some sense. If either everybody or maybe for a wide range of possible goals and motivations you have, it could be the case that they have this kind of convergence in that if you properly understood all the facts, they would sort of point to the same practical next step. And that, I think, is a lot more of an open question. One way that that could be, of course, is in a traditional theological context. If it's like a divine being has this goal and rewards people are not depending on, that might just be such that if that is true and if you understood that, then basically almost independently of what your particular goals are, you would have the most reason to follow the prescriptions there. If we are sort of in a truly materialistic, like the kind of worldview that a lot of secularists have, where there's a loss of nature is just rattling around, then that might not be such a convergence. Like what do you have most reason to do depends on your particular goals and someone else has different goals. So there is no higher fact that sort of would cause you to have reason to do the same thing. But I think there's a lot of possibilities between those two that we don't fully understand yet that could make it such that there is, in some sense, a purpose, even if we are not quite clear on what it would be. I wasn't going to ask this question, but you brought up if there is the one answer that all of humanity could be guided toward and that would kind of answer everything. So now the natural follow up is, let's say AI gives us that answer. Superintelligence gives us that answer. Do you stop practicing philosophy? Is there an AI model that's so good that all of the important philosophical questions just kind of get obviated? Yeah, I think that we will have sort of AI surpass us in philosophy, just as we now have in math and probably soon in other fields. There might be a subset of questions that have the form more of us making up our minds about what we want, where it might be the case that we need to do some of the deliberation and deciding ourselves. But broadly speaking, yes, I do think that our own intellectual efforts in that realm also will become sort of obsoleted by these superintelligences. And so which means that right now, while we still have something to contribute, it's the time to sort of really try to have whatever positive impact we are hoping to have in our lives. It's the golden age now for purpose and positive impact. And then, you know, once it's out of our hands, then it's time to maybe kick back a little bit. And speaking of, is there anything that you wanted to discuss or that we should have covered that we haven't so far? I mean, it's like so many different directions we could go with, like each of the topics we touched on in itself is a kind of could almost be like a one hour conversation, right? So you can. Yeah. But yeah, I mean, maybe at some point in the future, we could do another one or something. But there's nothing in particular right now. We'll love that. Well, thank you. Thank you so much for joining us. Really, really appreciate you taking the time. This was so much fun. It was fun. Yeah. Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts and Spotify. Follow us on X, A16Z and subscribe to our substack at a16z.substack.com. Thanks again for listening and I'll see you in the next episode. As a reminder, the content here is for informational purposes only, should not be taken as legal business, tax or investment advice or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.