AI Crisis in Math: Navier-Stokes, Riemann Hypothesis & Complexity Theory | Lex Fridman Podcast #503

Alright, so this is a kind of emergency episode. I think it's fair to say about the state of mathematics and artificial intelligence A kind of crisis moment in the history of mathematics over the last few weeks Machines claimed to have solved some of the hardest problems in the history of math Navier-Stokes the Quasi-Riemann hypothesis the unique games conjecture and hundreds of other results so Here with me we have Stephen Brunton who's a mathematician fluid dynamics expert at University of Washington Who can speak to Navier-Stokes turbulence all things fluid and many other topics? also, here's Ankur Moitra mathematician and theoretical computer scientist at MIT visiting Institute for Advanced Studies in Princeton this year who can speak to many things unique games Conjecture and complexity theory in general and the mathematics of how the AI learns and all things theoretical computer science and Alex a mathematician number theorist the Rutgers who can speak to also very many topics including Riemann hypothesis and Lean the automated proof and automated verifier and checking and so on and so many other topics in mathematics All three are brilliant all are very kind to show up this early morning and talk to me assemble this kind of Multi-agent mini swarm of human brains to try to interpret what happened in this very moment a lot of mathematicians brilliant mathematicians including Terence Tao and Others are freaking out some mix of awe terror dread and few inklings of light hope also And now a quick few second mention of our sponsors check them out in the description or at Lex Friedman comm slash sponsors It is the best way to support this podcast. We got whisper flow for voice to text writing Finn for customer service AI agents element for electrolytes blitzy for code generation in large codebases Shopify for selling stuff online Net suites for business management software, and of course perplexity for curiosity driven knowledge exploration Choosewise of my friends and now on to the full address I try to make them interesting, but if you skip, please still check out our sponsors. I enjoy their stuff Maybe you will too to get in touch with me for whatever reason go to Lex Friedman comm slash contact All right, let's go This episode is brought to you by whisper flow the AI voice dictation app that lets you write by speaking and at this point I Speak to AI Maybe an order of magnitude more than I type to AI which is a crazy thing to say me who's anxious about social interaction I Never thought I would find comfort in voice as the interface to AI systems But I do It's fascinating. I love the keyboard. I've loved the keyboard for many decades But we must say goodbye to the past and say hello to the future and one of those Tough goodbyes has been the kinesis keyboard for me and the keyboard in general I'm typing less and less and less these days, but playing more guitar and playing more piano The different kind of keyboard Anyway, if you want to try to use voice as the input layer for AI tools This is the way to go go to whisper flow to AI slash Lex to get one month of whisper flow Pro free That's w I SPR flow dot AI slash Lex This episode is also brought to you by Finn the number one agent for customer service 65% average resolution rate it's trusted by over 6,000 customer service leaders and top companies Including many of the top AI companies, you know when an AI company trust another AI company to do a thing That means they're legit Finn has found a niche an incredibly valuable niche and they do it extremely well There's some wisdom in that about how to succeed in this insane old world of AI systems everywhere find a niche that brings a lot of value to a large number of people and And Get really damn good at that niche Until frankly the niche swallows the world because we live in a time where it's just a bunch of niches and you realize there's so many people on the globe That the niches are quite large. They can fill stadiums. They can fill nations anyway, go to Finn dot AI slash Lex to learn more about transforming your customer service and Scaling your support team. That's Finn dot AI slash Lex This episode is also brought to you by element my daily zero sugar and delicious electrolyte mix I'm sipping in it right now watermelon salt Delicious all the crazy stuff I do. I've been basically exclusively eating only once a day working insane hours going through all kinds of Crazy whirlwind rollercoaster journeys inside the mind Trying to comprehend everything that's happening in the world What a difficult episode this is on mathematics Mathematics has been at the core of many of the things I admire about the limits What the human mind is capable of and here we are breaking through those limits at a rate that Nobody could have anticipated and it it's just a lot. It's so much to deal with. It's so much to deal with for mathematicians It's so much to deal with for other people for thousands for maybe millions of people Scared about the future of the world. I'm with you All right, get a free 8-pound sample pack with any purchase. Try it a drink element at comm slash Lex Speaking of a rapidly changing world. This episode is brought to you by Blitzy an AI powered autonomous software development platform. It's build for large complex code bases if You're one of them vibe coders who Open clause Peter says is a slur by coding a lot of the people that use AI agents to program are really working on small Code bases and often actually code bases that don't interact with customers with real people and if you're working on Code bases that serve a large number of people and those code bases are large. There's a whole set of tooling and systems and concerns you need to be aware of and that's what Blitzy helps you with The future of autonomous software development is here learn more or speak to a member of the team at Blitzy Blitzy.com slash Lex. That's Blitzy.com slash Lex this episode is brought to you by Shopify a Platform designed for anyone to sell anywhere. Why does it fill me with so much joy when I mention Shopify? I think about Shopify engineering. It fills me with joy. What am I gonna do? Anyway, it's a Great way to build a great-looking online store Toby is the man He's the CEO of Shopify Brilliant guy brilliant engineer brilliant leader of engineers and it's always nice when a guy like that Who's at the cutting edge who pushes the company to be at the cutting edge is running a company as impactful Shopify and it is so if you want to Be connected with such an impactful company sign up for a $1 per month trial period at Shopify.com slash Lex That's all lowercase Go to shopify.com slash Lex to take your business to the next level today This episode is also brought to you by NetSuite all in one AI powered business management system There are NetSuite next system automatically services insights throughout your day Their AI agents work alongside you to complete various tasks It's a great integrated system that looks at all the different modules within a company You know Managing of financials all the HR stuff inventory supply e-commerce all of that kind of stuff all the mess the beautiful beautiful mess It makes of a company all those are different modules hierarchical modules all of them need to be connected and managed and organized and the Utilization of AI within that whole system is essential and that's we does a great job of that for the first time ever You can try NetSuite for free go to NetSuite.ai slash Lex That's NetSuite.ai slash Lex This is the Lex Friedman podcast and now dear friends back to this fascinating conversation about mathematics You So I apologize to start with a question about feeling but what have you been feeling about this historic Moment, maybe Alex. Do you wanna go first? Yeah, I don't know if I would call it a crisis There's a lot of opportunity in that crisis. It's a it's a chance to rethink What it is we're trying to accomplish and it's the chaos of opportunity that some people are Reacting negatively to some people are reacting positively to and sometimes those reactions are happening in the same person and so I think we're entering if we're not already in a period of some kind of golden age of mathematical knowledge after millennia of Having a candle that we are walking around the woods and this is as far as we can see now We have a flashlight. I think we should comment on the magnitude of this moment for people who are not aware so maybe to quote you something like Something like holy shit. If a human did this would be an instant Fields Medal. Maybe you didn't say holy shit, but basically saying This is an incredible result It's a lot of people wouldn't have expected this amount of progress in this amount of time. So this is truly shocking So you're now speaking as a person who is Dealing with the feelings you felt over the last couple of days having found this out. So maybe also it'd be useful to go back to The feeling you were having when you first saw the report that in this case open AI dropped And the two of you happen to be on camera Loosely on the same day Maybe a few drinks and because it was a conference or a get-together Maybe I'll call you. Can you speak to what you were feeling? It was an interesting night when the results dropped now It was not entirely unexpected because for weeks there had been rumors about the two biggest problems in my areas that were in that That collection of results. So the unique games conjecture and RL equals L. There were even rumors about matrix multiplication so really my first reaction was Wow, it's all true Thank goodness It's finally out there because now people can actually go through it and process what's happened and figure out what to do next Now as to your earlier question, you know, what are we feeling? Well for me it changes on a daily basis But I think the biggest most overarching feeling is really feeling like you're getting pulled in two very different directions my work has always been about asking the right questions and To that extent these tools are amazing for me because you know, they Increase my ability to learn about other fields It's like having a world expert who's there to teach you about things in your own language your ability to make connections is just magnified But you know one of the parts of my job that I really love the most is the mentorship aspect Not just training grad students and postdocs and seeing them do great things, but having them come with me on these journeys So the part that pulls me in the other direction is the recognition that you know the playbook and The blueprint for success that that we had for so long is now thrown out the window So it becomes a lot harder to advise students and postdocs about you know What are things they should be working on that are going to make an impact are going to make a name for themselves So I feel very excited but maybe more intellectually lonely than I felt before In part because you know a lot of times great results come from asking very humble questions at least for me and now there's so much anxiety and dread and feeling that Wow, you would have to do something really momentous to stand out in the sea of AI results that it feels like we've lost Some of that patience in terms of planting the seeds from which the great oaks trees grow Terence Tao Fields Medal winner wrote I am horrified by the many personal years of ongoing patient and deliberately slow research efforts particularly by graduate students and postdocs Toward many motivating problems in mathematics being casually disrupted or destroyed by such a release Much as one cannot unhear a movie spoiler or a crossword clue One cannot explore a problem as profitably and richly once one is aware of an existing solution and I Mourn the path not taken and the opportunities lost in the frantic race to develop the technology so I mean if we're being realistic, it's more dramatic for a PhD student and a postdoc then That may be the way you've articulated Let me you're saying it quite calmly, but if I was a PhD student right now, I'd be in a state of like shock Like my whole life I was going through some of this I know a lot of my friends are programmers They're going through the state of shock and last December when basically most of the programming world switched to AI writing most of the programs So I would approximate it to a feeling of kind of a loss of meaning, kind of terror, like dark stuff. We'll keep exploring that. Steve, what do you think? What are you feeling? I would say most of my feelings are pretty hopeful. I think I'm an engineer. I like solving, you know, problems that make technology more realistic for humans, artificial hearts and airplanes and our ability to solve real applied problems is going to get much, much better. Our scope and reach is going to increase. And so for that, I'm extremely hopeful. But there is a lot of uncertainty. And I think, you know, some of this fear of meaning, of losing control, of, you know, just being kind of cut out of the process that we all love being a part of, it is a little scary. And I don't think any of us would be surprised at either of the kind of two paths that might take place over the next five or 10 years, where we really are just dwarfed by, you know, other intelligence, or it kind of levels off. And this becomes an incredible tool that we use for our curiosity, for our human problem solving. And that uncertainty is massive. Those are two very different worlds. And so uncertainty is difficult for humans. I think it's also difficult for those of us who have chosen math. We chose math, you know, probably because we liked rules and formalism and, you know, uncertainty is uncomfortable. I think the humility aspect is really important, though. This is a really good opportunity for us to remember why we're doing this. Like, what is human purpose? It's not to be famous. It's not to have a Fields Medal. You know, solving interesting problems, helping other people, being curious, just wanting to understand the world. Those are really good things to kind of get back. Like, what do we need to be satisfied humans? And I think that's a really interesting moment for us to think about. Like, I've been thinking about it a lot lately. It is more difficult for young people, right? Like, figuring out how to find their way. But I actually have more faith in them than us, in a sense. Do you think there's something to what Terry says about it's harder to be motivated when you know a problem has been solved? To explore that space when a problem has been solved like this by an AI system? Yes and no, in the sense that I do very much agree that the path is as important as the destination. Like, if you go on a road trip, the road trip is usually just as fun as getting there. At the same time, if we found out that another country had invented teleportation or time travel, I guarantee we would reinvent it in the next five years. Like, if you knew it was possible and you could reverse engineer a little bit, we would be curious, we would dive into it, and we would figure it out. So, knowing there's a solution also is an incredible motivation to get there much faster. Let me ask the question this way. If this moment marks the moment of the trajectory where AI basically, quote-unquote, solves math or kills math, can you make the case that that is, in fact, a trajectory? And then can you make the case that that is not a trajectory? Can you steelman both paths? That this is a path towards which, like, mathematics, as we know it, is done completely. So, that's going to be a thing done by machines. And then what humans do is something fundamentally different. Or there still will be something that looks similar to our current mathematics. Can you make a steelman in both cases? Alex? Yeah. So, you know, you said two things, and I'd like to separate them. One is solve math, and the other is destroy math. So, let's make that precise. What does solve math mean? If you think back to when chess engines first came out, and people were like, oh, this is going to be amazing. We'll play centaur games. So, this is a human being plus a computer system against another human being and a computer system. And that lasted like 10 seconds. And then the computers got so good that if you did anything but choose the top move that the computer recommended, you would lose. And so, the human being's role in that game became, yes, accept the first move. Yes, accept the first move. That's no longer fun. And we don't see many games of centaur chess anymore. Right? And so, in fact, I just had a conversation with a colleague named Thomas Coberta, where we axiomatized our argument for what the future might look like. And one of the axioms that I'm willing to even accept for the sake of argument is what I call the solved axiom. And the solved axiom says, let's assume at some future point, we're nowhere near there now, and I don't expect us to be there necessarily. We'll see where the capabilities taper off if they do. But let's assume the solved axiom, which says any mathematical problem that can be solved with human interacting with AI could also have been solved by AI alone. That's the solved axiom. So, that would mean that really there's nothing new for a human being. You can say to an AI system, okay, I have problem X. I want to think about problem X. But the only way that I will make progress, any progress that I'll make on problem X will just be press a button. The AI thinks about it. It doesn't necessarily solve it. It doesn't mean that it'll solve every problem. The Riemann hypothesis is not solved. Okay? That's what I really want the solution to. I just didn't think in my lifetime I would see any kind of progress towards it that we saw a few days ago. So, it's not that AI systems are solving every problem. They're solving every problem that could be solved by humans interacting with AI. That's what that axiom would be. So, I'm saying if we are ever in that world, does that necessarily destroy mathematics? And my strong argument is no. I don't think it destroys mathematics. If that axiom is true, what's the role of humans? So, I don't think that mathematics, pure mathematics, not engineering mathematics, not things that will cure cancer and solve, you know, everybody has a jetpack, whatever other technological advancement you might imagine, pure mathematics. You know, we know that we're proving theorems just because we're curious about them. I don't think the solution to theorems living in ones and zeros and data centers is valuable to human beings. I think the value of pure mathematics is that it lives in the heads of human beings. So, just to, you know, add to this, I mean, I'm glad you came right out with the question that's on everyone's mind. So, thank you for that. I would caution against overfitting to one objective function for this, right? Because, you know, we used to think of ourselves as just the smartest things around, you know, human mathematicians can solve, you know, reach the highest treetops. But now we're faced with the fact that many of the problems that we thought were, you know, quite a ways out have been solved by AI. But maybe one of the consolations we can take is actually the fact that we're very computationally limited compared to the way that AI is solving it. You know, given the same finish line of proving the same theorem, I can't really hold that many papers or ideas in my head at one time. So, in order for me to reach that same finish line, maybe I have to develop more generalizable and abstractable ways of getting there that then help you not just solve that one problem, but, you know, explain connections or other directions you want to get into, right? So, it's always amazing when you look back on papers you've written and you can see these threads, invisible sometimes, that connect different areas of work because developing systems of ways of understanding problems helps you for surrounding problems, not just to get a lean, formalized, you know, one theorem down. But the interesting thing here is those threads, what you're saying is sometimes invisible, are quite visible to the AI systems now. It seems like that's perhaps the actual very reason why they're able to make so much progress is they can extract the threads and pull on them because they have the view of the so many papers. I think it depends on what you mean by threads. You know, for me, the reason you prove results is so that you better understand the world around you. Mathematics, to me, is a vehicle for abstraction and understanding things. And, you know, it's when you have a major result, what's important is not just the truth of the statement, but what you've learned en route to doing that. Usually people's first question is, okay, you proved something fantastic, but how did you do it? And then people, once they digest that new idea, that new definition, that new way of looking at things, there are lots of other problems nearby that fall. So somehow you're building structures that help you attack not just one problem, but a whole sea of problems. And, you know, it's not that we need fully formalized, you know, results to do that. Sometimes we reason in very sloppy metaphors that, you know, help us conceptualize the world around us. And how much of this is something Alex mentioned offline, attention is all you need? How much of this is the human ability to shine a light on particular areas that quote-unquote matter? So the human ability to ask the right questions, the important questions. That's always been the case. So within this, before AI, there's a, you can think of like, here's the axioms, and just abstractly from the axioms, you can get to logical deductions, right? There are theorems that we could prove from the axioms. And there's a much, much, much larger space of theorems that we could prove than the space that human beings ever actually care about. And so, you know, so by almost by definition, it's like a tautology that the theorems that we'll get excited about are the theorems that we care about as humans. So maybe AI can convince us that there's some interesting structure. Maybe you can say, hey, look over here, let me give you this new definition or something. It hasn't done that yet. And sometimes the mechanism by which we select the things that we care about, it could have an application. So Navier-Stokes is connected to fluids and all natural systems. But it could also be a thing that some kind of mixture of beauty and difficulty that just pulls at us, maybe there's an intuition. You know, the prime numbers are full of these really complicated problems that maybe don't have an obvious application in the immediate sense, but it pulls at us somehow. And maybe as a symptom of the pursuit of the hard thing, great applied mathematics and engineering and solutions to real world problems comes out. So just, you know, as an example, right, a lot of times when I'm working on a problem, I might have some like bigger goal that I want to accomplish. But usually what I do is I try to find the simplest, most embarrassing open problem that I still don't know how to solve. You know, that aspect of like digesting it down to something, boy, how can we not know this? It, you know, it feels like it's on the right track because then when we can solve that example, maybe we're assembling the set of tools we need to make attacks on other problems. So I do think we're going to have to become a lot more deliberate about, you know, what do we care about? How do we use these tools from a metacognition standpoint? You know, what exactly are we delegating to them and what are we not delegating to them? And we're really at the beginning stages where we're just trying to piece it together as we go along. You know, on the one hand, looking for a proof is kind of like exploring this tree of possibilities. You might have some intuitions based on some other proofs you've seen about, you know, which directions to go in. Maybe I should go down this branch. Maybe I should go down this branch. But sometimes we encounter, you know, points in the tree which have really high branching factor. We just don't know which way to go down. We don't even have a guess about if we went down them, what would we discover? And now in this centaur world of math, we suddenly, you know, have something else intelligent that we can interact with, right? And what is the right way to use that tool to search for the beautiful theories, I think, will be challenging and evolving. But it's important to not lose sight of the fact that we don't just care about theorems. So, you know, I'm a mathematician. I'm a theoretical computer scientist. But I've also been reading a little bit more broadly, you know, about social sciences, just to sort of make my peace with all of the changes that have been happening. One of the things that really stuck in my mind that I liked is there's something called the doorman fallacy, which is very interesting. So the idea is that you take a complex job and you boil it down to its simplest, most visible task. So what does a doorman do? I guess they hold the door for people as they're coming and going, right? And when we lived in New York, we really enjoyed talking to our doorman. You know, they were wonderful people. But, you know, if you digest it down to just that simple job, you can think, oh, I can replace them just by a sliding door that detects when people are going or coming. And you miss out on all the aspects of their job that, you know, add a lot of value and are harder to articulate. And I think that's a good way to think about mathematics in the age of AI, too. What does a mathematician do? You know, the simplest and most visible part of the job is we prove theorems and, you know, theorems about conjectures that were open for hundreds of years, like Fermat's last theorem. These are very visible things that we've spent a lot of time popularizing. But I don't think that that's all that a mathematician does. I like to think about it as a mathematician, as someone who thinks deeply and a byproduct of thinking deeply is, you know, proving old conjectures. But it's also building new theories, building new ways of looking at old things, making new conceptual connections. And these are parts that, you know, maybe AI will swallow up, too. But we're not there yet at this moment. So even though this is a momentous time, I don't think it's the right time to hit the panic button. I think it's the right time to really recenter in terms of what our values are and have these conversations as a field and remember that what we do really has a plurality of different aesthetics and goals, and that when one thing becomes cheap, like problem solving for well-defined things, it makes the other things that are orthogonal to it all that much more valuable. That's actually really beautiful. The dormant fallacy going to perplexity here is the mistake of reducing a person's job to its most obvious task and assuming that automating that task replaces the value of the whole role. A hotel decides, quote, we're paying someone to open a door, an automatic door can do that more cheaply. But in fact, the doorman also welcomes guests and recognizes returning customers, handles luggage and hails taxis, provides security, and discourages unwanted visitors, and signals prestige and makes guests feel cared for. That was my favorite part. I looked up what a doorman does, and boy, we have a lot of commonalities with mathematicians. If I don't signal brand value, I don't know what I'm doing. That's funny. But I think the task for us humans in general, but also mathematicians, is to come up with that list, like actually make more explicit exactly what it is that humans do. And I haven't heard any of you mention beauty. I apologize if I've mentioned beauty excessively already. But there's something that mathematicians are, from my outsider sort of fan perspective, are kind of expositors, scholars of things beautiful, broadly defined. That's exactly right, is until five seconds ago, there was a near perfect correlation between the ability to solve new deep theorems and what we actually want as human mathematicians. And what we want is exactly what you said, beauty and understanding and depth of thought, and the fact that you can press a button and get what used to be our proxy without any effort, without any deep thought, and without any necessarily knowledge. We have people saying, do a breakthrough, and then putting whatever they find on Twitter, and now what are mathematicians supposed to do with that? I would go one step behind answering questions and doing problem solving is asking the right questions. So that is how I think you measure good problem solving is asking the right sub questions, asking a question that is interesting and can be solved and can be interpreted. And so I feel like the value of asking good questions has always been something we value in our colleagues, and that's gonna become increasingly more valuable. I agree completely. And just to emphasize a point that was sort of hidden in there, I think asking the right questions requires a lot of mathematical insight. You'll see people make that comment all the time, what are we gonna value going forward? It's asking the right question, it's giving great talks, but I wouldn't minimize, not that you were, but just to be explicit, I would not minimize the role in mathematical insight in both of those tasks, because it's when we say that talks will become more important, I'm not imagining that we're all gonna become Twitter influencers and TikTok whatevers. I hope not, because really what stands out is having ideas that get adopted by your community that end up becoming powerful over time. So I wanna tie this back to something that Lex brought up earlier on, which is he asked am I being overly optimistic? Probably, right? Compared to how grad students view these things. But I think one of the things that I'm most worried about in the near term is what's gonna happen to our systems and institutions. It's not just hiring and tenure and all these things, but it's really that when a grad student starts off their PhD, they're a new entrant to the field. They're trying to absorb what are the techniques in their field, what are the important problems, and they're trying to figure out how to make a name for themselves. And what we used to do is we would give them a blueprint that was very simple to follow. Take a problem people care about and improve it. And then people will know who you are, you'll get papers in top venues, doors will open for you. But now I think that blueprint has changed in a lot of ways because there are so many results that are happening right now that we're all just overloaded with it. And I think it's okay to take time to rethink that blueprint. We still care about the same things like ideas that are gonna influence your community, but they're just harder to operationalize in terms of some simple rubric that we can give them. So I'm still optimistic. I just think it's gonna take reworking a lot of our systems and institutions to do that. By the way, and so to go back to your point, I don't actually believe the Saldaxian. I don't think we're there. I can prove a theorem, but I don't understand why the proof works. There's some magic that happens over here. And until I generalize somehow the phenomenon and understand it more deeply, then I'll see why the proof works the way it does. Maybe that's one of the reasons why I don't actually believe the Saldaxian because the things that we do are much more exploratory and what constitutes a good question beyond just here's the Riemann hypothesis, make some progress on it. So I think you also brought up a really important point, this like generality and generalization of our concepts. So related to beauty and mathematics, I think one of the gold standards has always been how simple can your result be and how much does it generalize. And out of this infinite space of theorems that could be proven, finding some structure that is simple and generalizes, like that is I think still the gold standard. And unless I'm mistaken, humans are still asking the questions. So like we prove existence, uniqueness of Navier-Stokes with AI, but humans are guiding this. Humans are still asking the questions. Like I would love to see the next breakthrough if you had a machine that can prove any theorem you pose. I want it to start telling me what questions I should be asking. Do you think it's possible that that machine just does a for loop and finds all the simple and generalizable, basically all the E equals MC squared? That's happening today actually. So like there are absolutely research programs that are systematically automatically discovering kind of minimal physical laws. So this is less mathematical theorems, more physics governing laws and equations. So observing natural systems and trying to see in those natural systems if there's simple physical laws that model how they operate. Yeah, distilling it down to like the basics. So you can kind of recapitulate known theories. If you observe balls dropping, you can deduce gravity and F equals MA. If you observe fluids, you can deduce Navier-Stokes. And then taking that to data, we haven't written down simple equations, weather, turbulence, solar flares, things like that. And I'm assuming similar things are happening in math where there are probably these like minimal pieces that can be deduced. Do you think there's a large space of simple and generalizable things in math that could be discovered automatically? So my version of the solved axiom is that anything that I do as a human mathematician, I've tried on several occasions to sit down and say, okay, you know, LLMs will never do this that I do. And I'm like, well, why not? I can ask a question, you know, I can ask it a question. I can ask it, what questions should I ask? Right, there's sort of, because it's such a generalized system, I'm not saying that it's good at all of those things now. But my version of the solved axiom is anything that I can contribute as a human being, it could also have contributed on its own. And so what I really mean by that is, in terms of value, what do I bring to the community? It can't be, I asked a great question. It can't necessarily be in this world of the solved axiom, which again, I don't believe. But even in that world, it should be something other than, I asked a great question, I proved a great theorem. And even in that world, I think you need mathematicians, people with human mathematical expertise. Encore, do you believe in the solved axiom? We're on the spectrum of like how close we can get to it. And in that spectrum, what's the role of the human, do you think? I guess I don't really believe in the solved axiom. I think, maybe it's half true in some ways and figuring out which half of it is true will be very interesting and challenging. So I think there'll be major surprises and more to come in terms of what AI can solve. But for me, when you're facing a new technology that's really, really good, the danger is that when you try to conceptualize why it's working, you do so by anthropomorphizing it. So, maybe one of the earlier lessons here is from deep learning. So there are these amazing examples, you can take neural nets and you can do human level or better object recognition. But in general, the way that it succeeds is less robust than how a human does, because there are all these examples like adversarial examples, where you can add very slight patterns to natural images that can completely fool these object detection systems, but not a human. And I had a former colleague at MIT, Alex Modgery, they had some really nice work, which I really enjoyed. It was, I think the title was something like adversarial examples are features, not bugs. So they did the following cool experiment. So you take a bunch of natural data, maybe pictures of cats and dogs, you train a deep network to identify them very well, you add some adversarial perturbations to them that make to the AI system, a cat look like a dog and vice versa. So it's kind of think like a small little pattern that changes how the object detector works. And then you create a totally fake data set. It's got pictures of cats with dog patterns on top and you label it as dog and vice versa for cats. And amazingly, when you train a deep network on this with all the wrong labels, it's gonna learn something that does well on natural data. So somehow when we recognize what a cat and dog is, we do it in a way that we can articulate in terms of its legs and its tail and its shape in general. But a lot of times when deep networks or AI can do things that they might be able to find trapdoors or backdoors solving the problem, that use a lot of statistical relationships in the data that a human doesn't. So to me, it's important to not draw this boundary very sharply between what you can do and what AI can't do, because you're gonna be wrong. But that doesn't mean that I don't believe there will emerge some kind of boundary, if only for the fact that I think we over anthropomorphize the way that they solve problems. And when I'm looking through the open AI solutions, they just don't look like a human has solved those problems. So to me, there still seems to be a very big qualitative difference, and that's an opportunity really for collaboration and centaur math more than it is, dread about the AI overloads, but who knows, could be wrong next week. What's your intuition about how much of this is AI just tying together a bunch of thousands of disparate papers and pulling at the threads, sorry for bringing back the metaphor, but the threads of the different papers, and then just basically articulating the proofs that were already there in the different papers and you're making minor connections, or is it actually developing new mathematical ideas? Have you seen glimmers of whatever that genius alpha zero move in the game of Go? Yeah, move 37. Move 37. So I haven't read all 700 of their papers, the ones that I have looked at. Some people speak about this dismissively like, oh, it was in the convex hull. You do the moves that we know as humans and you just do them really well and do a really good tree search. I wouldn't call it the convex hull. I think Tim Gowers had a really good way of talking about it instead as the semigroup generated by all of these things. Yes, it's the same notion of putting together known moves, but the space of known moves, what it can accomplish is sort of much, much bigger than this kind of derogatory convex hull. So in terms of the quasi-RH, the moves that it took are all things that were known to human beings and there's a very long cascade of them that is kind of shocking when you look at it and say, oh my God, that's so beautiful. I'm anthropomorphizing it, of course. How did you think to do this move after that move and after that move? And yeah, that does it. Could you articulate to the more general public the intuition, the line between semigroup and convex hull, the derogatory convex hull? And quasi-RH, of course, is the quasi-Riemann hypothesis, that's one of the big proofs. Yeah, let me try to do it like this. You have, think of like a balloon. So the inside of the balloon is its convex hull. And so we have like all of these results, all these ideas that mathematicians have thought up and then anything inside of that is some linear combination of it. But it feels compact. It's like there's this kind of closed, enclosed area and within that area, like AI hasn't really made new progress because it hasn't moved the balloon out. It hasn't given us a new idea yet. And it's derogatory because the intuition there is we could have done it, mathematicians could have done it because it's already quite obvious. It's already within the space of things that have already been thought about and developed. Yeah, so it has read every paper. It's exactly what Ankur was just saying. It has read every paper. It has internalized all of that. That's more than any human brain not even could hold but has time to read and learn and internalize. So in terms of breadth, yes, it beats any human being. And then in terms of depth, has it really generated a genuinely new idea? And you can argue that, at least in the case of quasi RH, I would not say it's a genuinely new idea. It's a beautiful combination of a lot of things that were very well known. There are a lot of people who are kicking themselves that had some idea in that direction 20 years ago and two years ago, there was a big breakthrough on cubic theta functions and cubic Gauss sums. And that ended up being the path that the AI took to the solution. But so to contrast that with a semigroup, I think of a semigroup as like tiles. So you're just playing tiles on the board and what you're allowed to do is whenever you have a tile, you can put another tile to the left, to the right, to the front or to the back. And then you can just start tiling things. And that's like a semigroup action. And the question is, well, how far can you reach with that? You can go, I don't know, you can go really, really far out with existing ideas might prove a whole lot more than we, as humans, have been able to explore. So your intuition is more in the semi-group direction, meaning like there is at least glimmers of expansion versus linear combinations inside the convex hull. You know, how we choose to interpret what it's doing, I think, is up to us, but throughout the history of mathematics there have been very seemingly difficult and deep problems that then someone comes up with a three-page solution that is very elementary and could have been explained to an undergrad and could have been discovered 20 years before when the problem was first proposed. And had it been discovered 20 years before, it would be an exercise in a textbook somewhere. And because that solution wasn't discovered and really bright people worked on it for a long time and made incremental progress and made the problem really famous, then by the time it was actually solved, it was a really big deal. That happens from time to time in mathematics. And what I said a year ago when I wrote, so there was this International Congress of Mathematicians this summer and I was writing my proceedings that they wanted 10 months in advance. So I said, you know, what am I going to write about math and AI now that will still be relevant 10 months from now? And I think what's there is somewhat evergreen. I stand by what I wrote. But one of the things that I wrote there was exactly that we're going to discover that a whole lot more problems that we thought were unreachable by current technology actually are reachable by current technology, if you can put it together in just the right way. And that's what AI is amazing at. Again, someone could interpret that as, oh, so it's inside. You know, these solutions were inside the convex hull. You want to call it that? That's fine. I think that's, you know, it's not invalid. So I like all these metaphors, you know, also the balloon. So the important thing about the balloon is the boundary between what's inside and outside. And what makes it derogatory is the fact that, you know, you haven't seemingly changed the boundary. But, you know, definitely these problems have changed the boundary. And, you know, maybe a more concrete thing is just that the types of things that people are going to work on after that drop of open AI problems is forever different. You know, there were at least a couple of results there that are like, you know, field altering results that will change what types of problems you give to young researchers, what areas you want to explore, because your conception of the landscape is just that much different. So not just because it expanded, but because the very expansion changed the direction of the interesting questions. That's right. So, in fact, you know, one of the problems which probably, you know, maybe it's the easiest to explain what kind of effect it had on people is not actually from the open AI drop. It was an anthropic result that was written up by my colleague Virgie Vasilevska-Williams and then Josh Allman. So this was about the threesome problem. And I'd say that there it has a major effect on what's called fine grained complexity theory, because there was a big surprise that, you know, a very basic problem that you give people in like coding interviews can actually be solved asymptotically much faster than the solution anyone knew. And that was counter to what the community was betting. In fact, there was a whole theory built up around the idea that if this problem really is that hard, then all these other problems inherit its hardness. So now as a result of that problem being easier than we thought, you know, having a bit better of an algorithm, you know, yes, that gives you a lot new algorithms across the entire landscape. But it also changes the character of the field because the goal there was to put all those problems together and said that we should be studying them all together. But now we're sort of back in the state of affairs where maybe the right way to study them is separately after all. So I think these are examples where, you know, irrespective of how much, you know, deep new generalizable ideas we get out there, even just knowing the truth of the theorem is going to change the way that people organize research in that area. And, you know, there's no putting the genie back in the bottle on some of these problems. I mean, imagine one of the 700 was factoring in polynomial time. That would absolutely change. Everybody would know. Like you can't use your credit card on Amazon anymore. All of e-commerce is gone. All of secure communications is gone in like that. Can you explain factoring polynomial time? Yeah. So if you give me two very large numbers, a million digits each, I can multiply them together and do that in a split second. If you ask me to go backwards, say you took two large prime numbers, you multiply them together. You give me the product and you ask me, figure out which two numbers, you know, just factor. Don't we ask our kids to learn how to factor in school? You know, it's a lie. We do not know how to factor. And the fact that we don't know how to factor is why we have the capacity, the ability to communicate with one another in all kinds of secure ways. So all of modern basically the modern world depends on the fact that we don't have an algorithm for taking a large number and pulling its prime factors out of it. So a lot of e-commerce does, because that's the basis for a lot of the schemes and public key encryption. But, you know, famously, there are quantum algorithms for factoring. So there's a lot of emphasis these days on, you know, finding post-quantum, you know, bases for cryptography. A lot of space methods. And, yeah, we're already concerned about what will happen when we have quantum computers large enough to. But we still have a little bit of time before before quantum computers get large enough to make up, make this an actual problem. But there's no theoretical reason. I mean, OK, factoring is probably not NP hard. But, you know, I mean, let's say let's say prove that P doesn't equal NP. Maybe that doesn't. That would be amazing, obviously. But that doesn't if it proves that P equals NP, that would be completely. That would be very interesting. That would be very interesting. I'll definitely ask about more broadly in the context of complexity theory, but presumably open AI or any other company would not drop that result carelessly. That's the kind of thing that should go through the government. Sometimes certain certain mathematical results quite obviously lead to dramatic impact on the world. Other results might take maybe months to digest, but that one would just, yeah, would affect the economic systems of entire nations. So how many mathematicians does the NSA employ? Thousands. Thousands. Something like this. I've always done this thought experiment. You know what? What might they have that they just wouldn't tell us about? And this was 10 years ago. You know that some evidence showed that collaborative teams of mathematicians could solve much, much harder problems than any one individual mathematician. They like broke it down in the right way. So. I don't know. Do you want to just revisit the balloon for a second? Because I actually like the semigroup analogy better than the convex whole. But I don't think it's as it should be interpreted as derogatory as it is, even if it was just that these machines are interpolating human knowledge rather than extrapolating. Because even if there is this convex whole of ideas and things that could be pieced together from bits that humans have invented, we're not doing that fast enough. There's a lot of opaqueness, and this balloon is not see-through. And so even if you can make this convex whole much more transparent, kind of like lift the fog, give mathematicians radar so they can see what's over the horizon, that's pretty incredible. And I don't think we should be interpreting that as derogatory at all. I do think it is more, more extrapolatory than that. Yeah, we should say probably the size of the convex whole. We don't know actually how big it is. Well, that's part of the point. We don't know how big it is. We don't know what's next to each other. Like, there's so much fog of what we know, like, and that's exciting to lift that. But human beings do make new definitions, you know, bring new concepts in. And that fundamentally feels different from combining existing ideas and repackaging them. When human beings do that, and they do that from time to time, it really feels like, OK, there's a new boundary point. And now what's the convex whole of that, of what you could prove? So it's like you can take the balloon and there are ways to pull on it. And we know when it expands, like somehow we have an intuition for when it's getting bigger, like you can recognize that. In many ways, the archetype of what the researcher looks like is going to change because of this new technology, too. You know, I'm always amazed that when you look back historically, how much very different fields of math really spawned from similar ideas. And a lot of those connections drove the development in the beginning. But over time, they become a bit more specialized. And I think certainly the way that we train graduate students, you know, we'll need to make sure that they're not just over specialized in the same way. You know, I don't know what the future of hiring will look like. I don't know what the future of journals will look like. But I think people who think broadly about, you know, a bigger swath of mathematics will be in the best position to benefit from these technologies. You know, that would be my guess. Yes. Go. Almost silly sounding question. But so a large part for many decades of research has been. The paper, if you were to guess in the years to come, how does that change how a human contributes their idea, maybe collaborate, how a team of humans contribute an idea into the public space? So, I mean, I don't know about in pure math, but in engineering and some of the applied sciences, we already have a pretty rapidly shifting view of what an academic contribution is. So if you write code in part of a larger infrastructure that is reusable, that, you know, supports larger projects. So think of LIGO or some of the big, you know, billion dollar astronomy projects or, you know, cancer research projects that require hundreds of people and huge resources already. It's not necessarily one 15 page PDF. You know, there's lots of different pieces, code, data, reusable bits that can be fit into a larger framework. And in my mind, that is kind of obviously even more relevant with AI is having a little workable piece or widget or API or code base or data set that can be leveraged by the community or by AI is increasingly valuable, even more than just like one result or one paper. And I don't know how that looks in pure math. Can you comment on your view of how everything's changing, even just observing the insane evolution of even archive, which is similar in some ways to what's happening on GitHub? For example, a lot of people in the open source movement are really struggling to know what to do. Some are predicting the death of GitHub. Why? Just because there's so much code being added to GitHub that it can't sustain the growth? So one of the things that makes the open source movement successful is there's a kind of one or a few folks that really manage each repository, the human beings, and they are psychologically and on many levels are becoming overwhelmed by the all the pull requests. So it becomes impossible to manage and they get exhausted. And there's this whole feeling the open source movement, maybe similar to the academic environment, requires the humans to be motivated. And if you get worn out by the just similar the ideas that Terry Tao mentioned, that you just quit. And so that that's a failure mode that you want to kind of avoid. So journals were already in math under very, very severe strain from the number of things before AI having nothing to do with AI. Just there was more people producing more knowledge, sending it to journals and journals had to referee it. And referees are exhausted and being asked to work really hard to understand these theorems and check them. The only thing worse than that system, I think, is not having that system, because the way that like our mathematical epistemology, the way that we arrive at truth is deliberately slow and careful and precise. Go back to Perlman's solution of the Poincaré conjecture. He puts these papers on the archive and then people around the world are shocked by this. And so they invite him to come give lectures. He gives he comes to Stony Brook. He comes to NYU, comes to Princeton. He's going all around the country and the world. And he's giving three hour long impromptu explanations of, you know, maybe an hour is him explaining what he's doing. And the next two hours are people saying on page 247, lemma 412, I don't understand how you do this. And then he goes to the board and gives them a half hour lecture on how he does that. So the entire thing lives in his mind. He's done the best he can to communicate it on paper. And then it's up to us, the community, to go back through it very carefully, very deep in a detailed and precise way and agree, yes, this is a real proof of the Poincaré conjecture. And that took years to do and a whole lot of human effort. So coming up with the solution is only the first step in the process. And that process has to be deliberate and it has to be community driven. And so that's what the journal system is trying to accomplish. So you make a really interesting point. And it kind of highlights this traditional view of a pure mathematician is this like lone genius who, you know, solves the singular problem and then goes explaining it to the world. In many other fields, we have moved past the point where we can't individually advance, you know, circuit design or aircraft design or protein folding or any, you know, astronomy. And we have to work together in these billion dollar collaborative projects. And again, I'm an engineer, so I'm thinking leadership is obviously the thing that a community needs when it's having this kind of crisis of organization and individuality moving towards a more collective model. And that's interesting because I know so many mathematicians. Like, by kind of organization, this is a very independent group of people who don't like being told what to do, who don't like being told what to study, who are very independent by their very nature. You know, we like solitude and working alone. That's why we, you know, spend so much time doing it. I think that's a really good point. I mean, it's interesting just to see the differences in working styles, even between math and physics, you know, with things like the Large Hadron Collider. So, I mean, let me take a contrarian view for these things, which is, you know, you should never let a good crisis go to waste. I agree the journals were overloaded, the conferences were overloaded. It always strikes me that we're doing so much work, you know, to review these things, to handle all these things. And now we're faced with the task that someone's got to review all these AI-generated papers. And some of them were important, but, you know, it's really hard to get your community on board with doing that. I would really, you know, I don't know what's going to happen, but I would really be in favor for finding new models, not to replace the old ones, but to relieve some of the pressures that have come about. So, you know, I'd love to see models where, you know, maybe we take a more collaborative effort to verify some of these AI proofs, right? So, some of the OpenAI results that have dropped, you know, improvements on Fast Fourier Transform, you know, there's been tweet streams with people improving the constant. I don't really know what to make of that, you know, I'm not even on Twitter. And it seems to me that we need to figure out other modes of working, you know, for organizing what are the state of all the improvements, but also how do we organize and tame these ideas? So, it's kind of striking that, you know, the way that we think about publications, as you work hard on your draft, you post on an archive, and that's your draft. You know, you might update and create a new version at some point or a third version, but the draft belongs to you. It's not really up to the community to add notes. Hey, I think you can do this argument differently. Hey, I think this is simplified this way. But maybe there's some kinds of things that we need other models for, you know, allowing crowdsourcing for our understanding to evolve. I'd be all for that. You know, the other thing is I love giving research talks. You know, one of my favorite parts about giving research talks is you can take all of the messy complexities of the paper, the things where you just knew you had to dot the i's and cross the t's, but that wasn't really what excited you. There was some key idea, and there was some special case that really illustrated why that idea was powerful. And that's the part you really want to explain to people, because when someone asks me about Lemma 57, I don't even remember it, right? And maybe, you know, this idea that a paper is just this monolithic thing designed to be, you know, written down and completely formalized. Well, now we can trust lean formalizations to some extent for some of the technical building blocks, and that can allow us to rework the way that we do the writing in the first place, so we can put more of the emphasis on the work on, you know, the promulgation of those ideas. The Mathlib model is exactly what you were describing, where instead of a paper, you have a PR, you have a pull request, the maintainers and the reviewers will go through the pull request, they'll make suggestions here and there. I mean, the referees make suggestions, you can improve this and that, but it's not a public process in a journal. In a GitHub open workplace, you can see exactly who contributed what idea that simplified an argument, or you're, you know, you're reusing this, you're reinventing this idea that already exists in this part of the library, you should just quote that and refactor this part, whatever. And Mathlib, working formally, allows one for the first time. Like, one of the problems with mathematics is that we don't have a perfectly well-written specification to our theorems, our technical lemmas. There's, if you open up a random page in a random math book and read a lemma, there's so much context that the author, because it's so annoying to write something that fills in every bit of context. So we just assume you've read the previous 60 pages, now I can just tell you what the next step is in that lemma. But then when you try to, that means you can't just take it off the shelf the way you can a part, a physical part when you're building a rocket. And so if you, if all of mathematics is done in the future formally, then the specifications are built into the object, you know you're not using it in the wrong way, because you can't use it in the wrong way, because Lean won't let you. And that is the sort of crowdsourced version of mathematics. And that was the dream of Mathlib when it was started 10 years ago, was to run mathematics more like a science. As a quick aside, can you mention what Mathlib is and Lean, just because we mentioned it a few times for folks? Sure. So mathematics is, in the ideal, a formal process starting with an axiomatized system and then logical rules of inference from there. So if people think of, I don't know, do people still learn Euclid's geometry in high school or something? That's supposed to be the kind of model for kids to start seeing proofs for the first time and seeing an axiomatic structure. And you can take that idea and write it in software and say, here are the actual axioms of mathematics. Here are the rules of logical inference. And Lean, there's a long history of this kind of software. It's called interactive theorem provers, where you're writing line by line in a proof. And the computer says, yes, this line does follow by logical inference from the goal state on the previous line. So Lean is software that allows you to check all kinds of things. You can check the specifications of your code, align with what you what you were trying to define. That's what it was. What it was invented for was bug free code. And hopefully in the future we will have bug free code because AI systems are really good at finding bugs in code written by humans. So that's it's becoming ever more important for that application. But another application is to check mathematical results and allow you to write a mathematical result in a very formal way. And so now Lean itself is just the hardware that checks the logical steps. You do not want to start every math project by defining what the natural numbers are using zero and the successor, you know, the piano axioms of the natural numbers. So it's nice to have a large general library that's extremely well maintained, extremely well organized and so on. And that's what Mathlib is. Mathlib is the Lean library for formalizing mathematics in the Lean ecosystem. And again, Lean is just one interactive theorem prover. There are lots of others. It's the one that so far has been winning the math wars, if there are any math wars, the competition of two people or something. And it's the technology used in part to show that the OpenAI release of all the different proofs, that the proofs are correct. That's right. It has become now standard to because auto formalization has gotten to the point where AI systems can not only prove a theorem in natural language, you can't trust their natural language proofs. These are stochastic processes. And so the fact that they're saying things doesn't mean that they necessarily have a model of mathematics in their heads, whatever that means, the same way that we do. But that's OK, because they'll write a Lean code that builds on top of Mathlib, builds on top of the prime number theorem plus project or whatever, and checks that it has indeed solved this problem down to the axioms. So this is where I'll just take the tangent and the thought experiment may be a bit philosophical at this moment. But what if you have that black box that gives you proofs that you can't understand? Let's say the more interesting case where not because it's so big, but because the ideas just don't make sense to you to any of the world expert mathematicians in that field. What do you do with that? So the only way sort of almost by definition, because the logical rules of inference are written down and there there's only a few of them, there's a handful of them, the axioms, there's a handful of them. So the only way that you have mathematics that's not understandable is scale. Let me add a contrarian take. One of the things I also wonder is what would unformalizing math look like? Certainly formalization helps tremendously that we have confidence that the results are correct. But much of the way that we understand the results ourselves is really at some unformalized level. What is an interesting special case? Where does this come from, this technique you're using? And maybe that's what that world would look like. If we have these black boxes, we would try to tease apart and figure out how to ask them easier questions to develop some kind of system of understanding of how they actually got to the answer. Yeah. So there are really two different things that we conflate when we write a paper. One is the engineering of the proof and the other is the communication of that to someone else. And they're almost at odds with one another. I had this experience. I was taught real analysis last fall and you open up any real analysis textbook. And this was my experience when I took real analysis. The first couple of lectures are what are the real numbers? We have to define them or construct them as equivalence classes of Cauchy sequences or rationales, whatever the hell all that means. It takes like two weeks for a first semester course in real analysis to get there. And then we have the real number line. And now we can start doing calculus with formal epsilon and delta definitions, yada yada. And it always struck me as bizarre that we do this because before the class, I knew what the real numbers were. It's this number line. And then we spend these two weeks talking about Cauchy sequences of rationals. And then on week three, they're the number line again. What the hell was the purpose of that? And so it's because the way things are written or had been written were very engineering. You need to construct the real numbers before you go start using them. But the real numbers, human beings use the real numbers long before we understood that they needed to be constructed formally. So what I did in my class is I jumped straight into epsilon delta calculus. We did six weeks of calculations until we got to a point where, wait a second, here's a theorem. It's true in the real numbers. There's some convergence, something called Bolzano-Weierstrass theorem. You have a sequence of real numbers that are jumping around, but in a fixed range. There has to be some subsequence that converges to a real number. Well, that's not true in the rationals. Why is it true in the reals? What are the reals? So you don't start asking that question until you hit a problem that needs to be solved. So the way I taught the course was sort of backwards from how it needs to be taught if it's an engineering project. Because you can't just start talking about the real numbers until you've constructed them, until you know what they are. Well, that's correct if what you're doing is building up the theory. But if you're telling a story, then you absolutely can just start with the real numbers. You have an intuition for what that is. And until your intuition breaks, you don't need the resolution of how you actually construct them. So I think the way that we write papers in the future is going to be very different, where we spend a lot more time just talking about the story of what have I learned in this process, what are the interesting... You do the standard thing here, here, and here. But then here, you're sort of stuck, and where do you go? And here's an idea for why you go left and not right, and not up and down, or whatever. And there's a lean formalization that accompanies this story that lets you know that the engineering actually works out. Because you can't just tell stories. If you just tell stories, math is not... It's not music where you can tell a beautiful story, and it's okay that you have parallel fits moving somewhere, and you've broken some rule, but Bach does it all the time. So... So I think that's a very important point, just to sort of emphasize the point that Alex is making. There's this view of mathematics, which I don't believe in, where it's just this rising tide of theorems. How can AI be bad? It's contributing to this rising tide. But really, mathematics is not just about the number of theorems we have, far from it. It's also about that story of fighting with problems. Because when you fight with a good problem, it fights back at you, and you learn something about it. It's really about the system of beliefs and abstractions you come up with, in route to... When you're fighting this problem, you discover some more general meaning that applies more broadly. And I think it's a great opportunity to sort of disabuse ourselves of this idea that we're just, you know, axiomatizing things until we get to the next theorem. You know, telling that story of how we actually got to these problems, what the parallels are to other problems, is now, at this pivotal moment, more important than it ever has been. Yeah. So to get all the way back to your question about, you know, is it a matter of scale? Well, yes, it's just a matter of scale. But exactly, to Ankur's point, the way that we think about mathematics is about stories. And it might be that we don't understand the story of this. You can follow a proof line by line and not understand what's really going on, you might say. And you just ask some other LLM to explain to you the story, if it can come up with such an explanation. But that's what we want. We want stories. We want understanding. When you fight with a problem, it fights back. I think there's a deep truth to that about all the great pursuits that humanity has taken on. And on that topic, I would like to talk to you about some of the greatest fights of all time with some of the hardest problems in mathematics that have fallen. But first, a breath of break, if it's okay. Sure. So, Alex, you're a number theorist. Maybe it's fair to say the central problem in your field is the Riemann hypothesis. And you showed a lot of excitement on the aforementioned Twitter X about the quasi-RH, the quasi-Riemann hypothesis proof. Can you explain that? Can you explain what the Riemann hypothesis is? And can you explain the nature of your excitement about it? Sure. There's sort of a one-minute version of the Riemann hypothesis or a five-minute version or a 30-hour version. I don't know how far you want to go. Let's start pulling at that string. So, we know what the prime numbers are. People are interested in the distribution of the prime numbers. And what Riemann discovered is that there's a function. This is totally crazy, that there's a function that takes complex numbers as inputs and has complex numbers as outputs. And that function governs the behavior of the prime numbers. It's called the Riemann zeta function. And by the zeros of the zeta function, we mean complex inputs where the output is the number 0 in the complex plane, so the origin in the complex plane. And the question is, where are these complex zeros? What inputs, what complex number inputs, give you a complex number output of 0 when you apply this zeta function? Why in the world should that... complex numbers? What are you talking about? We have these discrete numbers that aren't divisible by anything else sitting in the natural numbers on the number line, and now you want to talk about functions from a complex space to a complex space. That was the great breakthrough. The great breakthrough was to figure out how to do signal processing on the prime numbers. If you think about, you know, the reason people can hear you, your podcast, is that my voice box is producing pressure waves of sound. That's the natural space. Then that goes into the microphone, the microphone analyzes it, does the Fourier analysis on it, converts it into amplitudes and frequencies, and it's just a bunch of numbers, stores those numbers or sends those numbers, whatever. The speaker then takes those numbers and reconstructs back out what sounds to people like my voice. The same thing is going on, that's what the zeta function does for the prime numbers. It takes the prime numbers, you can think of the prime numbers as particles of air molecules wiggling around, and it does a signal processing analysis on it that can recover the same prime numbers back. And the frequencies and the amplitudes correspond to the zeros of the zeta function. The zeta function has that information in it. And the Riemann hypothesis, so the amplitudes of all of these, you can call them Riemann waves, and the superposition of those waves recovers the distribution of the primes. And all of those amplitudes are supposed to be exactly the same. Like the orchestra that plays the music of the prime numbers has every single instrument sounding at exactly the same amplitude. That's what the Riemann hypothesis predicts. So can you actually, we know what the prime numbers are, but maybe going into the space of philosophy and the romantic notions of aesthetics, why are they so interesting and important and central and beautiful at the core of mathematics? And connected to why Riemann hypothesis is exciting is why is it so difficult to figure out if and what the pattern is in the prime numbers? Let me start with your second question first. Why something is difficult, I only know how to answer once I know what the answer is. Maybe it's not difficult. Maybe it's not as difficult as we thought it was. I did not think I would see a proof of the quasi-Riemann hypothesis in my lifetime. So the fact that I saw it on Tuesday was extraordinarily shocking. What did you do the moment you saw it, in 30 seconds after? Do you remember? I remember where I was the second I saw the drop. We were at this National Museum of Mathematics gala in New York City. I had a whiskey in one hand and I kept refreshing. I was talking to people, socializing, but I also kept refreshing because I knew it was dropping around six o'clock eastern time. Then the drop hit and in fact I was talking to a quantum reporter there and we were just chatting. Oh yeah, they're going to drop these 300, 400, we didn't know how many, 700 it turns out, problems. I was like, oh wow, that's really cool, it dropped. So I'm looking at my phone and the second thing I see is quasi-RH. I'm like, what do you mean quasi-RH? You don't mean for what? I look and it's the zeta function and it's all Dirichlet L-functions, so they've solved no zegal zeros. I'm like, that can't be what's in this drop. I could not compute in that moment what I was seeing. That speed of progress in that little time, yeah. It was so far outside my model of what was possible. It's been a month of things happening way outside of my model of what was possible. The Fermat's Last Theorem formalization, I was sure we were three years away at least from being able to do something like that. So maybe you can elaborate on that. It was in lean, it was formalized. How big was the Fermat's Last Theorem in lean? 13 million lines. 13 million lines, okay. So I just, two days before I gave a colloquium in the Princeton math department where I explained in painstaking detail what lean is, what MathLib is, but also that AI systems today, there's this MathLib library where everything is kind of perfectly done, exactly the right levels of generality. There's not supposed to be any code that's reusing code, that's reproving theorem somewhere else. Everything is the most general, the most abstract. And what AI systems are capable of doing is building one or two layers on top of that. But what they're doing is jerry-rigging. They have not been trained yet. I'm not saying they can't do it, but they're not good at it right now. What I call canonization, which is taking kind of sloppy code, refactoring it, figuring out exactly where in the library it belongs, what's the canonical version of this statement, of this lemma, and then everything, every proof should just be a combination of you apply this lemma from over here, you apply that lemma from over there, put all these things together, there's your theorem. And then it's really clean, it's really beautiful, very elegant. That's what MathLib is going for. And what I said in my colloquium where we are today is that you can get entire textbooks formalized as long as they're in the MathLib halo, as long as they're one, maybe two steps removed from things that have already been done in MathLib. But when you start, and this had been my practice, my experience had been, when you start asking it to do things that are three or four steps removed from the library, the way that it came up with a definition of this object and the way that it came up with some other definition of the same object, because it was doing it for two different purposes, then they clash, you can't stitch these things back together. You really need the library, MathLib, to grow in order to get to a place where you can do research manuscripts that are really synthesizing tons of different areas of mathematics. Which is why you would expect Fermat's Last Theorem to take, because it's so far outside of the halo of MathLib, it would take quite a bit of time. Exactly right. So Kevin Buzzard had a million dollar grant to spend the next five years, or maybe he's already a year or two into that grant, to reduce the formalization of Fermat's Last Theorem to things that were known in the 1980s. So he wanted to do that last step, those seven years of Wiles' work and Wiles Taylor, to finally finish, you know, Fermat's Last Theorem, but built on top of things that we knew would be extraordinarily difficult. He wasn't going to do Langlands Tunnel, he wasn't going to do Maser, there were all kinds of things that were expected to be even much more difficult than what these five years would take. And everything I said in that colloquium was wrong. And I think my colloquium was on a Wednesday, on Friday there was a drop of 13 million lines of code, from the axioms Fermat's Last Theorem is proved, the statement of Fermat's Last Theorem is completely elementary, there's no semantic alignment to check, you just glance at it, yes that's the statement, as Lean says it's sorry free. Can you explain how it was done? How it was done? So it was anthropic, and how it was done was AI systems just got really really good, it's still not a nice proof, it's not the clean proof, it's not, there's still a ton of work that needs to be done to make that proof a mathlib quality proof, and I think that's important work and work that we want to do. But in terms of just getting that sorry free theorem, they just blasted through all of these, they jerry-rigged this, they jerry-rigged that, and you know, 13 million lines later they had a sorry free formal lean proof of Fermat's Last Theorem, it's just absolutely shocking. By the way, it's beautiful to always, because of how rapid everything is, there's going to be a lot of statements that everybody makes, the world experts make that are wrong, that's the whole point of it, and that's the fun thing about human discussion, is we have disagreements, and I would venture to say somewhere between 40 to 80 percent of everything said today will be wrong in five years, because everything's changing so fast, but that's the whole point of human discussion, then deliberation and discourse and even the scientific process is saying shit and seeing what sticks over time, over 10 years, over 100 years, and so on. I was describing my experience, I had a lot of experience with lean and with using AI, I started doing lean before AI existed in the sense that we understand it now, and it was very painstaking, and the second AI systems came on, you know, the first LLMs were coming online, they were crap at lean, but then they got a little bit better, a little bit better, and I was like, oh my god, I can do library search, I don't have to sit there looking, I know there's a lemma that does this, but I don't know what it's called, and I don't know which exactly, is it in the manifolds library, is it in the algebraic geometry library, like give me this, give me this lemma, it was great at that, and I was like, oh this is amazing, I'm going to keep using these systems, and every bit of my experience told me that they can do like x but not 10x, and this was a million x. It was just completely outside of the distribution in my mind of what systems were capable of. Do you have a ballpark estimate of how many humans were involved in that, and like what is the level of human involvement in generating those 13 million lines of code? I believe it was two or three humans, and I believe most of their involvement was keep going, all caps. Okay, all right, so you think it's really just like gentle guidance, not like, you know, pathway creating? As far as I understand, there was no domain expertise on the parts of the people who were doing this. As the meme goes with OpenAI or Anthropic, you know, the prompt is solve math or something. Right, do a breakthrough. Yeah, do a breakthrough. Do a breakthrough, that's the prompt, and it goes and does it. Of course, you never know from a company perspective, there's a lot of incentive to sort of sell it as that, even though there might have been a lot of intermediate steps that led, whether in any stages of the training or maybe some things left unsaid, even though the final prompt is do a breakthrough, there could be a lot of groundwork before that that led to, we don't know. So I remember that was a Friday because I just was completely, like my mind was completely exploded, and it was Labor Day, and we just took the kids to the beach, and I needed a few days of, you know, I'm not checking my phone, I'm not on Twitter, I'm just throwing, you know, throwing my kids into the ocean. Did you have that moment where you're just sitting and looking at the ocean and contemplating? Yeah, like what does it all mean? Yeah, yeah. So I had already gone through that a month before, and I was like, so now what that says is that auto-formalization is basically solved. It's not solved in the way that I would want it to be solved. I want, and people are still going to work very, very hard on this. There's all kinds of other things that happen, just like there's all kinds of other things that happen when you prove theorems that are byproducts of that process for human understanding. There's all kinds of other things when you work on making a beautiful library of mathematics, and that process will continue, that process will be very difficult. It's going to be the, you know, thousands of man-hours, people around the world contributing in this open-source way, but in terms of can you get a sorry-free proof of something that human beings have written down on paper, I think that problem is done. And so you're standing there with a whiskey, having gone through this a month before, and now again. Right. This was round three. When I got back from Labor Day, it was Navier-Stokes. Navier-Stokes. I was just mourning if FLT, you're going to do this to me? Come on. But now, I mean, this really hit home, right, with the quasi-RH. So maybe, can you speak to the importance and the beauty of the primes, and also how big and hard the leap is from quasi-RH to full-RH? Yeah, so the primes, the reason the primes are so valuable is that they are the fundamental particles in the theory of multiplication. Every whole number is broken down into a product of primes. You know, we know about the elementary particles in physics, and all of matter is made up of that. In the same way, the primes are the particles that make up the multiplicative structure of whole numbers. And their distribution had been a mystery and remains a mystery. Even the Riemann hypothesis doesn't solve everything we want to know about prime numbers. But the Riemann hypothesis was so beautiful because it pointed the way to what the distribution should be, and it has a thousand applications in very faraway fields. If only the Riemann hypothesis were true, then we would know X, Y, Z. So let me interject a little story that came up last week, which was pretty entertaining. So we had Grant Sanderson who was visiting IES, and we had a long discussion about what is AI doing to the math community. And I really enjoyed this comment that Camillo Delellis made, which was I'll do my best impression. I can't do the accent. But there was a play written in the 1950s by a mathematician. So who knows that mathematicians are writing plays, but apparently this play, the premise of it was that von Neumann died, and he went to heaven, and he met God, and he got to ask him a question. So he asked him, is the Riemann hypothesis true? And of course, everyone holds their breath as they're waiting to hear the answer, and God says, yes. So finally, everyone can exhale, and then von Neumann asks, but how do you know? And God thinks and says, well, I know where all the zeros are, and they're all in the R equals a half line. So my understanding is the premise of the play is that that's not enough for von Neumann. He gets a little bit agitated to try to find out more about why it's true, and I think he ends up at the end in the other place. That's great. So I don't know what draws a moral from it, but it seems to speak to the mathematical experience. Math is not the fact. It's the why. It's always been the why. It's our deepest quest for understanding. And if you just tell me a fact is true, but I don't understand why it's true, that has contributed zero to my knowledge. So primes, I think, are also interesting because they relate to things that a three-year-old can understand. Fundamentally, we know how to add one to numbers, and there's this breakthrough moment when you're a kid. I can add one forever. That's a big idea. I can keep adding one forever. Then you learn how to multiply and group, and that's also a big idea. And primes are interesting to me, and have always been interesting, because we don't understand how adding and multiplying work together. The two most fundamental, basic, simple ideas in all of mathematics, we don't know how they work together. I can add one to a number that I can factor, and I have no idea what happens. I know if it's even it becomes odd, and vice versa, but I don't know if a number that has many, many composites becomes prime. We just fundamentally don't know that simple, adding one, how that changes multiplication. And primes are basically the distillation of that three-year-old mystery, and it's a mystery that all of us still wrestle with. That would be the interesting bit, is if the proof told me something fundamental about that structure that I could actually use. Yeah, to the three-year-old, I think my older one was something like two or three. We had these little blocks, you know, cubes that kids like to play with, and there was like 20 of them in the pack, and so he would make a little four-by-five grid, and he liked that, and he would make a ten-by-two grid, and that was kind of cool. And then one day I look over, and he's like really frustrated. I'm like, what's going on? He's like, it's not working. And one of them had rolled onto the couch, and he only had 19, and he couldn't figure out a way to make a nice-looking rectangle out of the 19. Yep, it still bothers me today. Yeah, so absolutely, kids love primes. Kids can discover primes. And so the solution is to lose another, right? Well, exactly, yeah. Do you think, so the leap between quasi and RH, I presume you think humans will not, in your lifetime, be able to solve without AI's help, the full agreement hypothesis? I don't know. Can AI, I guess, is the follow-on question. Like, how hard is it? I don't know. I don't know. I did not think I would see a proof of quasi-RH. Now that I see what the proof is, and there are a lot of technical details that I still have to go through for my own satisfaction. There's a lean formalization. The lean statement was already in the library, was already in MathLib. So the fact that they have a sorry-free proof gives me 100% confidence that the proof is right. And moreover, having read the proof, you know, at a high level, the ideas that it's using are all completely valid. They're brilliant, you know, to embed the Mobius function, twist it by sextic characters. This is all stuff that's the bread and butter of analytic number theorists that was chained in just the right way and was absolutely incredible. And so what Riemann showed is that, remember, we're talking about this function that has complex inputs and has these outputs, and we're worried about when the output is zero. So he showed that there's what's called the critical strip, which is a vertical infinite strip in the complex plane where the real part is, so you can think about the x-coordinate, lies between zero and one, and the imaginary part, the y-coordinate, can go arbitrarily high or arbitrarily low. And the hypothesis is that the zeros all lie right at the dead center of that strip, where the real part is a half. That's exactly what Ankur was just saying. And what we knew until Tuesday was region where there are no zeros of the zeta function, that's like getting thinner and thinner the higher up you go. It goes towards the one line. The one line is something called the prime number theorem, which tells you the basic distribution of the prime numbers is equivalent to the fact that there are no zeros of this function on the line real part of S equals one. So if the x-coordinate is one, as you go up and down that line, you will never find a value where when you apply the zeta function, it hits zero. Okay? So we had this zero-free region that was getting thinner and thinner the higher up you go, and that gave us the best known distribution on the prime numbers, the best error term in the prime number theorem, which I can go into a few. You tell me how much math you want to talk about. Let's go. But what I could have anticipated is that it would thicken that line a little bit. It would still be going to zero. The thickness would still be going to zero as you go up the imaginary axis, but it would be better than what we currently know. That thickness hasn't moved in 70 years, but I could sort of imagine that it could maybe make some progress. What it did is it gave us a vertical strip. So there are no zeros with real part between seven-eighths and one, an entire infinite vertical strip. That's a quasi-Riemann hypothesis. And if you can get that seven-eighths down to a half, that's the full Riemann hypothesis. Now, the methods that it used in the seven-eighths proof, and there's a second proof that's about eleven-twelfths, never mind these technical details, that method sort of fundamentally can't get down to one half. So a new idea is needed. You need a new idea, but quasi-RH needed lots and lots of new ideas, and it came up with them. As you're having gone through this three times already, is your intuition, this is for all three of you, about remeasuring in your mind the hardness of the various problems? Because you mentioned like picking easy things, because that presumably you have a good hardness radar. Like this problem is probably solvable, and this is not solvable today, solvable this week, solvable this month. Are you readjusting to this new world? I'm readjusting my inability to predict it. Yes, exactly. Well, you had the ability, no? It's noisy. Sure. It's a larger amount of noise than I thought. I mean, I think we've always known that there are problems we care about that are within reach of the tools that are out there right now, but it's just surprising the scale at which, you know, how many different problems really fall into that category. So we also don't know a lot about, you know, apparently they prompted thousands of problems, and what they dropped were only the positive solutions, that they only had about a 10% success rate. So we're sort of left guessing, you know, is there a very jagged boundary that we could squint at and say we can make sense out of that distinguishes, you know, what did fall versus what didn't fall? I think one of the other dangers is that maybe the problems which are the safest are ones which you look at and it's just painfully obvious you need a really creative new idea, you know, because you can sort of tell that maybe there's a barrier and the things that are out there in the literature just don't seem to fit into that hole. But now I think one of the things that's changed the landscape of math forever is, you know, maybe people become a little bit more secretive with those ideas than they were in the past, you know, because once those ideas are out there, then, you know, it might be a lot more straightforward for AI to be able to balance that one new idea, that missing ingredient, that new tile it needs, and be able to fully exhaustively explore what it can reach out of that. So you might be a lot more protective of your ideas than you were in the past. I think that's also a bit scary because traditionally the mathematics community has been really open with sharing ideas and that would be something bad to lose. But that's also because we had this first through the gate mentality and I think that's also going to change. That's right. I would prefer that the openness stay and the first through the gate is what goes. That's right, but that'll take time. It'll take time. Yeah. So that could be an effect, you know, that's more transient in nature that if we can learn how to value other things then maybe that'll go away. There are other communities that have models for how to value contribution even when it's not lionizing one individual's contribution. So it's not that there's not a way forward, but, you know, that is in the short term a real roadblock to doing mathematics the way that we used to and enjoying it the same way that we used to. So speaking of which, Steve, you told us a story about soup that made you maybe fall in love with the bigger problem of fluid dynamics. So going from math to applied math to engineering, the full thing. So maybe can you talk about Navier-Stokes, this set of equations that explain fluids, why it's such a big deal, why it's beautiful, similar as we talked about primes, why is it so hard and why is it such a big deal that Navier-Stokes was proven? So fundamentally the Navier-Stokes equations is just a way of stating the conservation of momentum for fluids, for objects that can move kind of freely like gases and liquids. So we know F equals ma is conservation of momentum, you know, classical Newtonian physics. The same basic principle applies to the gas in this room, to water flowing, and it's true in any volume. So it's true in little volumes, big volumes, imaginary volumes. And if you write down that conservation of momentum in a volume and then you kind of take the limit as that volume becomes arbitrary and small, you get these Navier-Stokes equations. It's a differential equation, a partial differential equation, that describes how the velocity field of a fluid changes in space and time, and also the pressure related to the forces acting on that fluid. Can you give an intuition about why fluid, given that F equals ma is a pretty simple equation, why fluids are such complicated systems? F equals ma is simple to write down, but it can get complicated really, really quickly. So if I have like a single pendulum, it's really easy to write down the equations for that single pendulum. If I put, you know, another pendulum on the end of it, now I have a more complicated equation and it gives chaos. Fluids is, you know, in a sense, if I try to model the air in this room, I could chop the room up into a billion little boxes and write down a little differential equation for how the momentum works in all of those billion little boxes. But now I have a really complicated system of equations. And so it's simple to write down. It's simple to conceptually understand what the equation is saying and representing, but actually solving it and understanding how the fluid actually moves in any kind of practical sense is very, very complicated. I read something like, you know, like one space where fluid dynamics is important is reducing drag for when you're flying for airplanes. I read something like airlines could save like something like tens of billions of dollars by reducing the drag by 1%. So why is this so hard? So that's one. And drag is maybe easier than something like turbulence. So, like, why is why is these things that are worth so much money so difficult? So the good news is, is that if you can improve our ability to predict turbulence, to simulate it even a little bit better, faster, more accurately, there are, you know, dozens of fields where you can have a huge impact, both economical, ecological, human health and safety, with a relatively small improvement. So drag reduction on airplanes, better artificial hearts, you know, more efficient wind turbines and gas turbines, and, you know, you name it, the world is, we live and work in a fluid, and so do all of our machines. The inverse design problem, I know it's actually very closely related to this kind of like math problem. I know what I want to prove. I don't know how to get there. I know I want a wing that has high lift and low drag, but I don't know how changing the geometry, changing the shape of that wing will affect this whole cascade of smaller and smaller and smaller vortices that, you know, this turbulent fluid that generates over the wing that eventually results in its lift and drag. I don't know how to inverse design efficiently. Part of it's because the equations are expensive to simulate or to test. You know, wind tunnels are expensive. You need supercomputers to simulate these things. This is actually kind of related to pure math. We'll get back to this, I think, but there's this really nice Feynman quote that, you know, we know how to write down these very general equations, Maxwell's equations, Navier-Stokes equations, Schrodinger's equations, but we currently don't know from the equation what phenomena is possible. You know, if you'd never visited Earth, could you have possibly predicted the sunsets and the tides and all the phenomena just from like a partial differential equation? Like, we're not even close to there, and there's some very fundamental math questions about, you know, if I can write down a differential equation, I can't tell you what's inside of it. We still have to simulate. We still have to test. We still have to do this very elaborate engineering process to understand how these equations manifest drag on an airplane wing or turbulence inside of an artificial heart. Sorry, can I just ask a question on that? I've always wanted—so, like, in the case of Einstein's theory of general relativity, he writes down these equations. A few years later, Schwarzschild finds a blow-up, and they're like, oh, come on, so what's wrong with these equations that there's a blow-up? You know, if you get enough matter into a small enough space, then, like, light can't escape it. We don't have any physical experience for this. This has got to be a mistake in the equations until we discover black holes, and then it turns out that actually the theory perfectly explains, you know, and predicts what's supposed to happen. In the case of blow-up of Navier-Stokes, are we going to discover something new about water, or is the fact that at some scale you get to particles, you know, you get to an atomic structure that does not follow the liquid equations anymore? Why would we have predicted that there should be solutions to a continuous process that's allowed to be pushed to its limits? Yeah, it's a great question. I'm going to give you the engineering answer, which is these equations are an exceptionally good model of how fluids work at a certain scale. And you're absolutely right. Like, you need some number of individual molecules to be averaged over in a volume for these to be constitutive equations that are valid. And so, if you zoom in small enough, you know, now we're talking about individual molecules, and Navier-Stokes breaks down. So, the mathematical problem of these equations blowing up is interesting from a mathematical standpoint. By the time they blow up, it stops being a fluid. There's no black holes. Like, other physics takes over. Yeah, but it's possible. Wait, I think it's interesting from a philosophy of science perspective that where equations break down could, and maybe statistically high likelihood, is a window, a door in a breakthrough of needing some other discovery. Like Newtonian physics breaking down needs Einstein. Right. And the grand unification is that we don't have gravity at the same scales as we have, you know, interactions of subatomic particles, but but this idea that We don't know something so fundamental about these equations There are so many people on earth right now simulating and testing Navier-Stokes in every industry imaginable I mean, it's it's probably one of the most studied and analyzed and used equations in the entire engineering world And we don't know this fundamental property of the PDE. Does it blow up in finite time or not? That's like the zeroth-order version of the bigger question is, you know, what other kinds of solutions does it admit? Does it you know, are there sunsets and hurricanes and tornadoes all the phenomenology that that could exist, you know? Existence and uniqueness is like the the simplest baby step and that's where it gets interesting to me is even if there's not direct tangible use of this mathematical result in an engineering sense It gets us that one step closer to understanding the nature of these equations and what is possible Which is very fundamentally interesting to us Could there be could this solution that that has the blow-up be used somehow in an engineering application? And you know, you want to drive it into this state and get some funny behavior that helps airplanes fly with less drag or something Yeah, I mean I think in principle a huge industry a huge part of what we try to do is manipulate Fluid flows like bend them into our will right? So if you watch a bird fly or an insect fly they are shaping vortices and carefully funneling, you know energy into structures for their You know objectives of their daily life to like perch on a rock or you know hunt and we try to do the same thing So flow control is is one of the highest challenges. It's very very challenging But all of our science fiction technology Would be kind of accelerated if we had better ability to control these flows and that this this proof is One very interesting construction of this kind of, you know, very specialized flow control in a way I'm reminded the Feynman was asked about intelligence Artificial intelligence in the 70s and he said something like the way that we solve problems with technology is Radically different from how nature solves these problems. Look at birds how birds fly and how airplanes fly he was exactly pointing to this and it's kind of interesting to think back to that comment now when we see what LLMs can do with theorem proving versus what human brains Can do with theorem proving and all kinds of other applications, but it's it's interesting that he pointed exactly to this issue of how How we've solved flight Using technology and how different that is from nature's solution to it. Our airplanes don't fly now I wonder if birds have existential crises when I see planes What am I even here for? Yeah What What are the different components of the navies of the question? Can you if you can you dig in like what what does it take to describe fluid? Yeah, so very very basic a fluid moves and it moves differently in different places and at different points in time and so The kind of basic building block of a fluid is the velocity field So at every point in space, there is a velocity vector that tells you how that that fluid is moving You know, maybe it's really fast here and slow here and it creates a vortex and It gets more interesting when the fluid velocity changes a lot over space So like a tornado is a very rapidly changing fluid velocity field in both space and time There's pressures that are involved. So again, like the core of a tornado is low pressure. So pressures are Kind of forcing this velocity field and causing it to move in the way it does that's the building block is a velocity field and its pressures and Then the equations tell you how that velocity changes in time and in space according to Kind of its own momentum. So fluid has momentum to carry itself. That's you know gusts of wind it has internal Friction and resistance that's viscosity. So honey has more internal resistance than water has more internal resistance than air And then all of the external forces like Heating and pressure and gravity and things like that So each of those can be quantified as a term in this this larger Navier-Stokes equation So speaking of external forces, can you describe the math side of this of the Navier-Stokes for the blow-up? What was the formulation of the problem and what was actually proven? And why was it so hard if you don't have? that internal friction that viscosity We have a strong suspicion that the equations can and will blow up under certain Conditions we have other examples kind of simpler problems that are like Navier-Stokes That do blow up So the simplest one is just a simple ordinary differential equation if you don't mind me taking a little tangent Let's go So x dot equals x squared x is the state of my system x dot just means The rate of change of x in time and if that rate of change of x's is growing proportional to x squared That differential equation does blow up in finite time. So it just takes a quadratic nonlinearity Think about if I have you know, x dot equals x This is like x could be the population of bunnies and we know bunnies like to Populate x dot equals x grows exponentially, you know The the state of my bunny population in time will grow exponentially get bigger and bigger and bigger. It'll double and double and double But it won't reach infinity Until infinite time has passed. That's just a normal exponential growth. That's not finite time blow up. That's x dot equals x But if I take x dot equals x squared that blows up in finite time So that quadratic nonlinearity that quadratic kind of feedback is enough to cause this equation to blow up And you can like if you write this down, you know You can solve this using kind of classic basic calculus techniques and you get this this finite time blow up And so there's a progression of partial differential equations that have the same behavior where a quadratic nonlinearity and that's really interesting is the Navier-Stokes equations One of the terms that can the convective term that tells how the fluid kind of moves itself with its own momentum is a Quadratic nonlinearity just like x dot equals x squared. It's kind of the self motion Without Diffusion without viscosity. We think that that should probably blow up to infinity and finite time have this finite time blow up, but we don't know if Viscosity if the kind of, you know internal friction and resistance like honey has Always damps out those blowups before they can happen My intuition as an engineer is of course, they do like if I have a flow where it's you know Accelerating, you know to infinity that should immediately Diffuse into its neighbors. It should create this like diffusion into its its neighboring fluid and kind of tamp down that that growth And so I think the math problem is essentially Are there any edge cases any initial configurations of the fluid where you perfectly orchestrate this thing? So that it could actually accelerate to infinity and finite time and then there's two kind of technical versions of this one is With no external forcing and one is with smooth external forcing Smooth being important because if the external forcing has singularities, then it's kind of trivial Like if I force it with a tornado, you know an infinitely tight tornado Then of course, it's going to blow up the problem is stated has smooth External forcing there's four versions of the yes on the clay Yeah website one of them being the forced version and that's the one that was proven and that's on those proven Exactly. What was proven and what are some interesting things about the proof that you saw? Yeah, so What is proven is that there is this way of Orchestrating a smooth kind of external forcing that perfectly cancels out various terms in the equation So that you get this Vortex this can only happen in three dimensions is really important We should probably say that we've been talking about three dimensions is a big difference between 3d and 2d Yeah, so I mean when we talk about a fluid we usually imagine it in 3d But there are these kind of analogies or simplified 2d flows whether on earth is pretty well approximated as a 2d flow Yeah, you've mentioned somewhere in a talk that I heard that Hurricanes a kind of like model dwell in 2d and tornadoes are 3d and I remember I remember different sort of analogies that you were drawing there Yeah, so so one of the interesting phenomena in three-dimensional turbulence is you generally have this this breakdown of big vortices into smaller Vortices into a little itty-bitty vortices until eventually they molecularly diffuse into heat That's kind of this three-dimensional turbulence cascade in Two dimensions like whether on earth you actually get a reverse cascade, which is why hurricanes form you actually have little vortices Coalescing into medium vortices coalescing into bigger vortices, but so the AI proof of This kind of counterexample that you can have finite blow-up basically creates this very very specialized vortex That in the third dimension is stretching out and thinning it so that it can essentially become infinitely Thin the radius gets smaller and smaller and smaller as it stretches out Kind of like a figure skater doing a spin you pull your arms in you go much much faster And there's this very specialized kind of rate of contraction of this vortex so that the energy and the fluid still remains finite But at this localized filament The flow kind of blows up and you get infinite vorticity. I was surprised when Mathematicians before the AI proof Started thinking it was going to be possible to find this counterexample, which only started maybe 15 years ago. That's right It's a relatively recent in the grand scheme of things. Yep, instead of looking for a solution looking for a counterexample Yeah, because all of our physical intuition tells us diffusion does solve the problem And so it is like very physically counterintuitive in the mathematics world. That's a really hard problem impressive to construct a counter example What does it teach to you as an engineer who stared and studied at a lot of different natural systems there The there are fluids and maybe take that as a step towards what are The equivalent kind of hardness and harder problems yet to be solved in fluids So it's a very important math result I'll say it like but day-to-day engineering fluids and applied math kind of analysis of fluids is almost completely unchanged fluids are just as hard to simulate and predict today as they were a month ago or two months ago, and so When I'm trying to actually solve an engineering problem with a fluid I want to design an airplane wing or an artificial heart or whatever it is I'm still having to simulate and predict and manipulate this flow in a very practical way that's expensive in terms of Supercomputer time and wind tunnel resources and the this mathematical theorem and proof are very interesting But it operates at a regime that's so non physical that it doesn't really affect my day-to-day Engineering there's still I know this is very pithy, but I like to say, you know, we solved the million-dollar math problem There's a trillion dollar engineering problem, which is like how do you predict these things in a fast efficient way and do this inverse design? That's a much harder problem And you say it's a trillion dollar because that literally literally you saw some of these problems That's how much money you'll make not because it's a prize, but like there's industries. Yeah, and and we could say, you know Trillion dollars of Possible benefit to humanity of like better faster solutions lower drag less emissions, you know all all of it. It's We have like transportation defense health energy all of these have you know fluids at the very core Technology and so I think those problems are still very hard and this math result is Important in in the sense I was talking about earlier that We don't know the fundamental Character of what is possible in these equations. We don't even know if eggs if the solutions exist. That's like so fundamental This is like a baby step towards a much bigger human goal of understanding. What's in an equation in a more Practical sense. Can you speak to some of the maybe? Beautiful fascinating phenomena in the space of fluids that you've been thinking about you mentioned turbulence What what are the big puzzles to sort of solve in the space to understand you've mentioned drag like air fluid What are like the big millennial prize type problems that people are thinking about? I mean, there's so many you can break them down into a few categories. So turbulence is very hard because it's multi-scale What is turbulence turbulence is this this phenomenon that? When you have a complex fluid roiling or mixing and moving kind of like Da Vinci's he has these beautiful drawings of fluids at all of these different scales All of the scales matter the big vortices and the medium vortices and the little vortices all of them actually matter to predict The lift and the drag and the thing so the gas in this room is a great Counter-example, it's the opposite of turbulence. It is complicated. The molecules are all moving around in a very frenetic way And there's a lot of molecules But that you know 10 to the 23 number of gas particles all of those individual configurations don't matter we know from Thermodynamics, this is like the great success of classical physics is that there's a small handful of numbers that actually determine how this gas behaves, you know, temperature, pressure, density. That simple description doesn't exist for turbulent fluids, for the flow over a wing, the flow inside your heart, for the flow on a river. All of the scales actually matter. All of the numbers in that simulation actually matter to predict the next behavior, to predict what's going to happen, and that makes it fundamentally challenging. So this is actually a great AI problem, is there a coordinate system, is there a set of numbers, just like we have for gases, you know, temperature, pressure, density, is there a set of numbers or some representation of a fluid, of a turbulent fluid, that allow it to be simply described in a way that is useful for engineering to build, you know, motors and engines, and just like we have for gas dynamics. So then the interesting question, if it's a problem for AI, and AI is listening, I'm sure, the solution to that problem, would it, would the path lie through mathematics or through having an AI system sit there and watch some fluid for a while? I think yes to both. Because fish, they figured it out. They figured it out. Well, so I think you have the right keyword as coordinates, and I, so that's kind of why I said both, and I'm not trying to be like trite, but all of our great physics breakthroughs were predated by a coordinate transformation. So Newton's F equals ma came after we put the sun at the center of the universe instead of the earth, right? You had to do that coordinate transform first. So I think what, the way I see it is, we need to take all of this data, these simulations, these observations of fluids, and what we're looking for is a simple coordinate system so that the description of that fluid becomes very simple and generalizable, beautiful in a physical sense. And that is both a brute force, you know, get a lot of data, try a bunch of different coordinate transformations and representations until you find the simplest one that generalizes the best. But there's also a lot of deep mathematics in what the form of that coordinate transformation might take. In general, if you find a question that really smart people asked, you know, in the 70s and 80s, but they hit a wall computationally, that's very fertile ground right now. So, like, we know what the form of these coordinate systems probably should take from some fundamental principles. We've just never been able to write them down. They're too complicated to write down. That's, like, a perfect problem for AI is, you know, now let's give it a lot more data, a lot more compute, and see if we can start learning these transformations we know should exist. Just to understand, the formation of vortexes, so that's like a, like, little, like, tornadoes. Little tornadoes. Their formation is, is that part tricky to understand? Yeah, absolutely. And it's beautiful, too. Like, I love watching these videos of, you know, if you shoot two vortices together, they'll, you know, influence each other and then merge in these very interesting ways. You have, like, a red vortex and a blue vortex, and you can see where they merge and join. But they also stretch and fold and, you know, turbulence is chaotic. So the trajectories get tangled like spaghetti. It's not random. There's a huge amount of structure, but it is very multi-scale, very, very tangled. So one of the things I would wonder, you know, scientifically, I mean, these are really interesting problems, and that was a great explanation. So I'm always curious about what computational problems nature can solve already, right? And there are a lot of examples of this. You know, there's a famous one with, like, slime mold finding shortest paths in mazes. In fact, some of the precursors to Navier-Stokes were, you know, like, some of Terry Tao's work on showing that you can have blow-up for Navier-Stokes-like equations because they can simulate Turing machines. So I guess I would also wonder what sorts of computational tasks these computations with vortices colliding can do and how that imposes limits on what sorts of coordinate systems ought to exist. Because, you know, you mentioned PV equals NRT. That's an amazing dimensionality reduction because you don't keep track of all of these positions and velocities of particles, but there's just, you know, a small list of numbers that describes a metastate, and that's it. But if I really had a tremendous reduction, you know, like that in the case of turbulence, then it would seem to me very difficult to compute very interesting things, too. Could it be like the game of life where you have this very simple cellular automata system and it's a universal Turing machine and you can cook up whatever behavior you want in it? Yeah, that's right. I mean, it's a game of life. A lot of times people think about, you know, what are minimal rules to have all of these exotic behaviors, and it's really shocking how small the rules need to be to get that right. But I guess, you know, I'm a theoretical computer scientist. One of the biggest exports from our field is really the algorithmic lens. So it's not just about, you know, computation when you have a well-defined problem and a laptop and you want to solve it, but it's also about, you know, computation in the natural world and what that teaches us about what the rules of nature might be. So, you know, some famous examples of this are things like equilibria in multiplayer games, and Nash equilibrium is a great solution concept. But of course, the fact that it's hard to compute Nash equilibrium, you know, throws into question, is it really a universal mechanism that explains how people play games? There was this amazing quote by Kamal Jain that I really like, which is, if my laptop can't find it, then neither can the market. And, you know, to me, what's really exciting is the nexus of a lot of these different perspectives. You know, when you were describing, and it sounded to me like they're very interesting computational problems under the hood, and that one can kind of get a dual perspective on these things by thinking about how that computation imposes limits on what a good coordinate system ought to look like. So I think that's a really nice way of framing the problem. And I would take two extreme examples. So Earth's weather and climate systems are actually just complex. All the scales do matter. The weather in this zip code matters to me. You know, everything matters. All the variables matter. And you're right. Like, if I overreduce that system to, you know, five numbers, then I lose the descriptive capability of what I actually care about. That's one extreme. The other extreme, let's think about how, you know, a moth flies or a butterfly or a fruit fly. They are proof by existence that you can interact with an extremely complicated turbulent flow and do very advanced engineering control. They can control their forces and moments in gusts, in rain. They can live very rich lives. And they presumably don't have a PhD in fluid dynamics or know the Navier-Stokes. They're probably not running supercomputer simulations. There might be five or six numbers that actually matter to their daily life of maintaining a leading-edge vortex and, you know, stalling their wings to perch and to land. And those are kind of two extreme examples, and I really want to be able to handle both of those. We want to be able to handle the full complexity of truly complex systems, and we want to be able to distill down the absolute bare minimum essentials for fast decisions in those kind of insect-extreme cases. And biological flyers still dramatically outperform, you know, human-designed fluid systems in many regimes. So we still have a lot to learn about these simple… So maybe we're, like, veering into research now, but, you know, to me it's always amazing how nature not only solves very hard optimization problems, it seems to find very robust solutions. And I'm always struck by the fact that, you know, when we model it, we tend to model it assuming we have full knowledge, you know, about what the boundary and initial conditions are for things like Navier-Stokes. But when a bird flies, you know, it doesn't know all of that. And to some extent, you know, modeling things with uncertainty is really how you reframe the optimization problem to get very, you know, rich and robust solutions. That's kind of amazing to me how a lot of everyday examples sort of fit into this. I remember, you know, I'm a big baseball fan, and I remember reading a math paper about baseball that when you've got an outfielder, you know, once the ball has been hit, a lot of stuff is known. You know, the velocity, the trajectory. You would think the outfielder would just take a straight-line path to where they expected the ball to be. But when you observe outfielders, they don't take straight-line paths. They take a slight curve to it. So why is that? You know, at least one way to wrap your head around it is that that's actually a very robust strategy, because when you're taking a curved path, you can continually reevaluate the trajectory of the ball and then adjust as needed. So there are very interesting solutions that nature finds, but it's not always obvious what is the right formulation for the optimization problem in the first place. And sometimes modeling it in the case of uncertainty so that you can design solutions that are robust to the fact that you don't have full system knowledge seems to be an important piece of the picture. So there's a fun aside related to this. Insects, in particular, don't have the ability to measure their full flow field. They can react to gusts of wind, gusts of wind faster than the information that hits their wings goes to their brain and back to their muscles. So researchers have studied this for decades in biology, and what they've found is that there are strain-sensitive neurons embedded on the wings, maybe a few dozen to a few hundred of these, in very stereotypical locations. So brother fruit flies will have the same locations of these strain-sensitive neurons, and those strain-sensitive neurons fire when the wing bends in a certain temporal pattern. So it gets hit by a gust of wind, the wing bends, neurons fire, and a local computation is happening in that shoulder muscle of this insect that goes directly back and corrects robustly based on partial information of a system it doesn't have equations for that is non-linear and turbulent. So it's like, again, proof by existence of exactly what you're saying is maybe the robust strategy of actually relying on partial information and uncertainty. They can do more than we can at this point with way less information. Yeah, I have a couple of, I apologize, pothead thoughts, because Alex mentioned black holes. I have a fun thought that maybe the singularity observed with a blow-up in Navier-Stokes equations comes into play when we're, as opposed to airplane wings on Earth, we're traveling through two connected black holes in a warp drive that we figure out, and then there's the flow control of space-time, and that becomes relevant because now you're in a black hole, hence the singularity. For people just listening to this look Alex just gave me, it hurts my soul. Okay, so moving on to the next topic. But the forcing, you need the black hole vortex to also be forced at exactly the right, yeah, the forcing, where's the force, I mean, at that point, you could try to write it. Yeah, there's a way, you can make something out of this. There's some non-zero probability that future asses that will listen to this, that guy is the one time he said something that was actually correct. Okay, if it's okay, a quick bathroom break. Yeah, sure, sounds good. Cool. Ankur, one of my favorite topics is theoretical computer science. You're one of the world experts in many of its problems, so I'd love to ask you about one of the big problems that's been solved in this OpenAI job, which is the unique games conjecture. Can you give people an intuition about what this problem is, why it's a big deal, and what are some aspects of the proof that are interesting? Yes, I'd love to. So let me give you a little bit of a background. So let's start off with complexity theory 101. So one of the most important problems in theoretical computer science is the satisfiability problem. So the thing to keep in your mind is imagine you've got variables x1 to xn, and they can take on either true or false values. And what I'm going to do is I'm going to create a big formula out of them, and usually the way we do this is with an AND of ORs. So I start off with a clause that's, you know, an OR, x1 OR not x3 OR x7. So depending on my assignment, that clause might be satisfied, as long as at least one of those things is true. And then I take the AND of many of these different clauses together. And the question is, if I give you that formula, called a conjunctive normal form formula, you know, can you find a satisfying assignment? So can you find an assignment of x1, x2, up to xn, setting them either to be equal to true or false, so that all of those OR clauses are on, so that the entire formula evaluates to true. So this is the canonical example of a problem that's easy to check someone else's answer, but it might be hard to find an answer yourself. So if you told me that it's satisfiable, all I would do is I'd ask you, okay, give me the satisfying assignment. You tell me what x1 is set to, what x2 is set to, and I just go clause by clause and check to make sure that you really have satisfied everything. Now, the, you know, the central problem in theoretical computer science is the P versus NP problem. So are problems that are easy to check, is it easy to find their solutions to? And the presumption in our field is that, no, it's not easy. And that's the basis, you know, of a lot of things like the, you know, cryptography and e-commerce and what have you. Now, really, the starting point for getting to something like the unique games conjecture is something called the PCP theorem. So this was a landmark result proved in the 90s. It actually was such a big deal that it made the front page of the New York Times. And the crux of it is really this idea that it's even robustly hard to fool a verifier. So you start off with a formula, and either it's satisfiable or not satisfiable. And what does that mean? If it's satisfiable, you can write down an assignment and I can check one by one that the entire thing evaluates to true. But if it's not satisfiable, that means someplace you're lying. You write down this purported satisfying assignment. And when I go clause by clause, I'm quickly going to find one clause that is not true and that invalidates the whole thing. And I say, aha, I caught you. That's not actually a satisfying assignment. So the idea behind things like probabilistically checkable proofs without getting into a lot of the technical definitions, the idea is really gap amplification. So can you start off with a formula that is hard to decide whether it's satisfiable or not? And can you engineer a new, maybe much bigger formula that's robustly easy to check in the sense that if it's satisfiable, it'll still be satisfiable. You can write down an assignment and it'll be fine. But if it's not, I should actually be able to just flip a few coins, query random variables, check a random clause, and with good probability find that you're lying. So the point is that with very sublinear inspection, with only constant amount of inspection, I can find that you're cheating. So this was a momentous result in theoretical computer science because it laid the foundation really for what's called hardness of approximation. So a lot of algorithms and complexity theory is what do you do when you're faced with problems that are genuinely hard to solve? And, you know, one of the things we do is we try to approximately solve them. So maybe we can't find an assignment if there is one that satisfies all the clauses, but can we find one that satisfies many of the clauses? How large a fraction of the clauses? Now, even there, we would expect problems to become computationally hard after a certain point. It should be that it's not just that, you know, deciding whether it's completely satisfiable or not completely satisfiable is hard, but maybe even deciding is it 100% satisfiable or is it only 99% satisfiable? The best case over all assignments satisfy at most 99% of the clauses. That too might end up being hard to decide. So these are what probabilistically checkable proofs give us. In places where we knew that the optimization problem was hard, they tell us that it's even hard to approximate. And the unique games conjecture is a strengthening of the ordinary PCP theorem because it has to do with a certain type of label cover instance. It's a bit technical to go into the details, but the point is that if that version of a probabilistically checkable proof were true, it would have this great organizing effect, where it would turn out that for a huge swath of problems, we would actually know what is the best approximation algorithm, and it would be something very simple. So when I started off in the field, the unique games conjecture had been posed, you know, a few years before that. And there was still a lot of debate about whether it should be true or false. And, you know, people worked hard to try and give proofs of it. People worked hard to try and disprove it. There was this amazing time period, maybe in the 2010s, where there was quite a lot of back and forth. And, you know, there would be new results came out that people's opinions would change whether they believed unique games or not. I like this quip. I forget who said it, so I can't properly attribute it. But someone told me, you know, maybe God's going to decide whether it's true or false based on which side is working harder. Because it really seemed to change year to year. And, you know, one of the results, one of the headlining results in the OpenAI drop of proofs was that the unique games conjecture is true. Now, I should mention that this is not without spectacular human progress on it. So one of my colleagues at MIT, one of my colleagues at IIS, they both contributed, among other people, to a lot of the major strides. So Dor Mintzer and Irit Dinur, they proved a weaker version called the two to one games theorem. And this really was a, you know, shot heard around the world in theoretical computer science, because at that point, I think a lot of people weren't sure what was a path forward for unique games conjecture. And I think their work really gave a viable path forward to something that was in the vicinity of unique games. So it sounded like, you know, maybe humans could get there with another 10, 20 years. But, you know, just when OpenAI released the results, it turns out that the unique games conjecture is actually true. So this is quite remarkable, because, you know, at some points, I guess I did find myself on the side of the doubters, because unique games just sounded too nice to be true. Because, you know, a lot of work like Prasad Raghavendra's work, it implies optimal hardness of approximation for so many different types of constraint satisfaction problems, that the conclusion just looked to me too strong to be true. What did you feel when you found out that night? Okay, well, for starters, there were already rumors, you know, in the weeks leading up that OpenAI had solved the unique games conjecture, and that it was a true. And these weren't even just completely arbitrary rumors. One of my colleagues, Dorb Minsur, he actually posted a paper where he wrote something in the footnote at the bottom that, you know, we would have liked to polish this paper more, but because of the rumors that were going around, we feel the need to put out our approach sooner rather than later. So for me, it was, in some sense, a bit of a relief that the results were finally out there. Now, they're not very readable papers. It's not like you can point to them and really quickly identify what's a new key idea. But I'm glad that they're out there, because now we can move forward on the next phase, which is digesting those proofs, and figuring out where the field moves after that. Now, in terms of how did I feel that night, you know, as Alex mentioned, we, you know, we all had a drink in hand. And then I saw, you know, on my phone, the news that that these proofs had dropped. And, you know, people kept up coming and talking to me at the gala. But it's already hard to socialize as a mathematician, let alone, you know, when you have math, you really want to read. So it was doubly hard. The other funny thing about it was, you know, I was in New York already. So I took it as an opportunity before the gala to visit a lot of people I knew at NYU. And I just happened to be meeting with Subhash Kot, the one who introduced the unique games conjecture. And I just so happened to ask him, you know, just a few hours before the gala, before the OpenAI paper is released, Subhash, what would you do, you know, if they release a positive proof of unique games? Are you going to read it? And he told me, no. Which, you know, it's, it's, that's a lot of people's reactions, unfortunately, it's just kind of true that there are these grand challenges we work on. And sometimes when they get solved, you know, sometimes it feels like, you know, someone has stolen your thunder. You know, it's, it's not like people didn't have ideas about how to solve the unique games conjecture. And I bet you those, those blueprints would have panned out if we'd given them more time. I think the other thing I worry is that, you know, that, that we lose out on the human effort of trying to reach those solutions to begin with. You know, math is not just about what is the furthest I can reach with the current set of tools I have. Is it a convex hull? Is it a subgroup? I don't know. But, you know, sometimes when it's a human driven solution, we learn a lot more in the process. So I worry about the fact that there were these great landmark problems, field defining problems that a lot of people are reaching towards. And that decisive progress was made, you know, and the results all dropped in one night and that people lose their enthusiasm for going down those paths. And I wonder if we'd given those people that space to go down those paths, what other exciting things we would have discovered. So I think one of the things I mentioned, you know, we were all trying to process it at MoMath and, you know, there were a lot of really nice talks. Not all of them went up on YouTube. Some of them were about benefactors and awards named after them. And the thing I kept coming back to is this idea that, you know, why do you do math in the first place is to play and learn about the world. And we saw so many manifestations of it that night, you know, because there were awards named after people who, you know, why were they donating to the museum was because they wanted their kids to have a place to play with mathematical ideas and learn about the world. It was also the opening of an Escher exhibit. And what a great visual of, you know, what happens when we play with mathematical concepts. Right. And so it felt to me that in this moment, it's really important to remember as we're going through all of this change together as we don't know what the future holds, what AI and mathematicians collaboration will look like in the future, what hiring will look like. But the reason we all got into it was because we were very curious. You know, we wanted to understand the world. And mathematicians are very good at abstracting things. You know, they don't just understand one little piece, but they find some abstraction and some set of definitions that turns out to be unreasonably powerful in some other corner of mathematics. So I think it's really important, whatever the future holds, to make sure that we organize our field in a way where we still encourage people to play with ideas. I don't know what kind of reward system will work. But, you know, when I look out and I talk to young people in the field, they have a lot of anxiety. You know, they hear a lot of stories even about people they know, you know, running harnesses to, you know, scrape the web for open problems to dump as many papers they can to flood archive. I don't know if that's a good strategy. It doesn't seem like it is. But it certainly doesn't feel like fun to me. It doesn't feel like we're playing with ideas and learning about the world. And that's sad. I just want to get back to the things I really enjoy. The other thing maybe I just want to add in there too is like, you know, it's a big identity crisis for a lot of people. Because, you know, some people view themselves more as problem solvers. And that can feel very comforting that you can draw a little circle around the types of problems you work on, the types of techniques you use, the types of problems you care about. And that feels very comforting because the rest of the world is messy and you've got blinders on. This is the part that I care about. Now, I think is the time to remove those blinders because I just don't find that to be a productive way to do it. You know, we love playing and learning about the world around us. And it's important to remember, as you know, a young mathematician, are you smarter than you are curious? Or are you more curious than you are smart? It's very important to be more curious than anything else. And you mentioned fun. There could be another perspective of it, of more about the struggle, from my maybe outsider perspective, is some of the most fun in the moment sure feels like struggle. I've talked to a lot of Olympic gold medalists, a lot of them feel something approximating depression after they win the gold medal, because it's like you have given everything and you realize everything you've given is not over. 30 years ago, there was a documentary of Andrew Wiles. And there's a great scene, I think they even run it right at the very beginning. To your gold medalist, you know, what do you do once you've won the thing? He's sitting there at his desk, and he's describing the moment when he realizes what the solution is, not his initial announcement, which was flawed. And then for many months, he was trying to correct it. And he finally saw how to how to do it. And he's trying to say, you know, I will never again in my life, have a moment as powerful as the moment that I experienced at this desk that day. And he's choking up. He can't get the words out. So that can you get that if you press the button and it gave you the solution? Absolutely not. So exactly to both of your points, like the journey, you know, if you see a giant hill, a giant mountain, and you say one day, I'm going to get to the top of that, and you start training for it, and you're sort of doing little by little, whatever. And then a helicopter lands and just drops you at the top. Can you actually speak to that? The interesting thing about Andrew Wiles, and a little bit about Grigori Perlman as well, is working alone for a prolonged period of time. What do you make of that? How common is that in mathematics of sitting there and grinding a problem for, in his case, seven years? Until the 20th century, almost every math paper that had ever been written was written, and it had one author, I think now the median is two in mathematics. And so we're not a lab science, it really is a solitary process. Now, see, the things that I give my graduate students to do are not the things that I'm working on, sort of by design, I want them to have their own field, I want to open doors for them. It's different in different fields, right? So when they have a success, it really should be their success, even if I've sort of, I had an idea that this would go somewhere, and they executed on it, that's theirs. But in terms of sitting there, the paper that I put out a month ago is something I've been thinking about for 10 years. It's a five author paper. Over those 10 years, we made incremental progress to the point where we finally, you know, had a solution. And that paper was an instance of there's a phenomenon here, and I don't understand what it is. It wasn't here's a problem, solve it. Like, we could not at the start of the project, give you The lean formalized statement that needs a sorry like you discover what the statement is through the process of Hacking away at the problem little by little until you see enough of it to to say this is how it comes together Actually that makes me think can you describe how your thinking process of how you think the really hard problems? Is there some aspects to it that are interesting like a lot of coffee in? In the morning at night alone brooding with a cigar or on a Together at a whiteboard with others. Is it is there some? interesting color texture insights you can give sure I Think I think a lot through world models in some way. So, you know, I have some Mental model of how I think results in some sub area go and I think that colors a lot of how I see Surrounding areas too. So for me a lot of my work is really about, you know Building bridges between algorithmic thinking and other problems that usually aren't studied through that same algorithmic lens so I think when I first started as a Advisor, I sort of made the mistake that I gave students questions that were a hundred percent in my area And the problem then was, you know when they would come back and they'd have an idea I could shoot it down very quickly and I was either writer. I was wrong. But either way, no one was having any fun and You know over time what I started to do was I would give students questions that were 10 20 degrees off from my own area Things that I wanted to understand better myself papers that I thought were important that I just hadn't had time to sit down and read I'm gonna have the student explain the result to me and What's fun is we would go through that process together not, you know by printing out the paper and going through it line by line But trying to reverse engineer Why do we think this works in the first place? And there's so many details that once you get the feel for it You kind of have the sense that they ought to work out But at least in important papers, there are one or maybe even more than that key ideas that really change your perspective Oh, this really might be doable So what I found was, you know as soon as we start trying to understand one of these results we find our own way of looking at things and Then it's not long after we find some other questions that we now think are approachable because we found our own way of Organizing our thoughts. So that's often the way I do It is the earlier that I can have the student in the driver's seat not necessarily with a well-defined question But with them teaching me something and them being responsible for distilling and digesting and abstracting it down to its essence You know reasoning through with them analogies that aren't meant to literally be true But help us figure out what to expect in neighboring areas You know, then it feels like they bring a unique perspective and we're off to the races so for me in in a lot of ways the technology is Only gonna enable that kind of work because it means that a much richer set of problems are approachable You know for me when I find experts and related areas that can really explain things in my own language I hold on to them for dear life because you know They can speak the gospel. They can tell me how this this area is organized and explain it in my own language but now, you know, I have technology that can do that too and It also feels a bit sad that you lose that human element that was there once before I Think also students have so much anxiety about you know What publications are gonna look like that one of the hardest parts is you know when I'm thinking about a project? I'm really trying to think about it in a sort of multi-year time horizon Because I have some meta vision that I want to realize like how can we use learning theory techniques and quantum many-body systems? And there are a lot of stepping stones on route to that, but some of the most exciting problems are you know many years out? But students are so worried about the incoming AI tidal wave, you know the future of work universal basic income are any of us gonna have jobs that they're really thinking in three and six month horizons and Then it's hard to find problems that we both care about just because you know, we're thinking about different worlds You don't think we should encourage them to continue to think long-term, I mean, I understand that oh I do But they don't always listen to you. I Mean, I don't know if I'm right or wrong, you know, maybe AI will solve all of our problems But it seems like the only ones that are actually worth us working on at this moment in time Are the ones we're still gonna care about in a bunch of years and you should only be working on future-proof problems If you're working on things that you know You're in a rush because you think the next generation of models will definitely solve those problems Then maybe just go on vacation and don't solve that problem Can you speak to how you think you do work a lot of natural systems? What are the interesting insights that can be pulled out of these natural systems? so I find a tension between Focusing very hard on an engineering problem. I'm trying to solve which can be very Singular you put on blinders you only think about this problem and then giving yourself the kind of mental freedom to look around and just observe and be curious very often you find the solution when you turn over a rock and when it's Not something you're thinking about You know with the full front of your mind, but in the side of your mind if you're looking at a problem You know through your peripheral vision you have a much better chance of solving it sometimes and so this is a constant balance for me is Observing all of the fields that I don't work in and all of the nature that I don't understand and all of the things that I'm Curious about but then also having very focused problems that you don't relent you don't let go and I this must be true for all Of us we I've heard from from both of you You know focus problems you've been working on for years and decades that you haven't let go of but presumably That is informed by a million other pieces of information that you happen to pick up through interesting conversations And I just do you have example because it sounds like neighboring areas is a common theme you have Examples of things you're like you're the curious peripheral vision of your brain went Okay, that's an interesting system might be relevant and that ended up being so or maybe in the future will so for me as a fluid dynamicist biology and its messiness and complexity and also material science and its messiness and complexity are Neighboring in a way that's very fruitful I learn a lot from from both of those fields and a lot of the kind of basic math that we develop to handle complexity does seem to transfer and sometimes you have an idea just because you see something is possible or You start to develop something and you can explain something that we didn't understand before but I think For me biology and fluids and materials have this really interesting kind of commonality of multi-scale nonlinear Messy nebulous objectives things like that, by the way on that topic Do you think there's hope that AI can have some significant breakthroughs in the messy world of biology? I certainly hope so and I think so there are large efforts in biology to collect very large data sets and to understand Complexity in systems. So the connectome where you have the full wiring diagram of the brains of Insects, it looks like we will start to have connectomes of other animals and eventually it'll be probably like genomes where you know It is cheaper and more accessible. I have this like feeling that we'll find out speaking to the resilience of you know Birds being resilient that in order to understand how the human biology system works You have to like pull that thread And you're gonna have to understand how the entirety of life on earth emerged and functioned as an entire Like mechanism like it's all tied together like you can't replicate It's ridiculous. What happens with humans from a single cell this whole thing goes It builds and that the intelligence that's built into the whole building mechanism in the first few Days weeks months of life This whole thing is like I think could be like many orders of magnitude more complex than some of the hard mathematics problems We've been talking about I guess I suspect If turbulence is hard Like whatever the hell's going on in here. I don't mean the brain I mean like the fluids are on the brain the whole thing the immune system all of it The everything interconnected together and I am very deeply connected to the history of the bacteria that was here billions of years ago, I suspect the complexity of the entirety of the mechanism of life on earth is many many many orders of magnitude higher than any System that we that is amenable to AI today Just thought sorry, I know I tend to agree I think you're right like It's it's incredible to just imagine all of the complexity and scales that inside one cell It's a whole city. It's a whole infrastructure, you know, one of your neurons has garbage collection and power production and information product like the whole thing is at every single scale is Beyond our ability to possibly fathom you're absolutely right And I think that's what makes it so interesting and it is related to math in the sense that Finding the right scale where we have any hope at all and there is something interesting That we can use to understand that that whole picture. I think that's very fascinating and that Up until now has been a human endeavor of finding the bits that are interesting and understandable in this mess of complexity and Yeah, I think I think you're right. I mean the fact that anything is Understandable about the natural world is is a miracle, right? There's no reason that Newton's laws can be written in physics that Newton himself Can describe the mathematics of and you know invent calculus for the purpose of being able to say what inverse square propagation? Results in elliptical orbits, you know solve the two-body problem yada yada Yeah, so the the fact that the universe speaks mathematics is completely mysterious There's no a priori reason that we we don't have a right to know anything that we know I Apologize for the question, but I'm obligated by the said universe to ask the question, but given this topic Do you think math is invented or discovered? Yeah, it's the age-old question I have a very sort of pragmatic answer to this which is The questions that we end up asking it seems to me are functions of what's going on in the world What questions we possibly can be asking given our current state of knowledge the questions? We're trying to answer today are not questions that Euclid could have thought about Math was sort of the the things that people got interested in were different in Greece versus India versus China But then the answers once you ask a question precisely The answer is what God decides the answer is so we don't have so we're discovering answers to questions that we've invented We're discovering answer. Okay See if you have worked with a lot of natural systems and look at the equations equations are I? Guess always approximations of the real world. So what was this math real? You got to say yes Yeah, how big is the gap between the mathematical world in the real world Surprisingly small I think Unexpectedly small. I like I like your answer a lot. Actually, I do somehow fundamentally think that math is both created and discovered We create mathematical formalisms to understand the world around us and Those approximations are unreasonably effective. It is It is a miracle that there are simple compact descriptions F equals ma equals MC squared that describe Everything that we observe at, you know All of the scales that we seem to observe we seem to be able to distill something simple mathematical that describes them That feels like we are creating those But that there is that simple Generalizable description underneath everything and that seems to be something we're also uncovering. Do you think that's a good guy? you know kind of searching for the beautiful simplicity For us. Oh, we are we just us like aesthetically Pleasing things. I think it's a good guide for us because we are fundamentally simple We can only handle so much in our minds. We can only Envision so much. I think simple is essential for humans to interact with the real world Okay, if I want to get like slightly deeper into the question of created for versus discovered Numbers are a good example integers. We all Fundamentally feel like you know counting numbers are real in a very Essential way I can have one rock and two rocks and three rocks and that is a number of rocks That is a real number of rocks But those are absolutely abstractions that I have created a rock is not a rock Is not a rock. It's a collection of atoms. That's a different collection of atoms. They're not there's not one rock and or two rocks Do numbers even exist? I wrestle with this. I don't know like it. Do we actually believe that integers are a human creation to interact with our symbolic abstraction of what we observe quantities and I mean took humans thousands of years to discover the number zero, right and And then negative numbers negative numbers were actually accepted by mathematicians in the West after complex numbers were accepted as A mechanism for study which seems insane, but there's a whole story behind that. What's your relationship like with infinity? Is it real to you or is it a nice thing to have as part of proofs? So much of modern mathematics is exactly dealing Seriously with infinity right one of the first times people really start to hit a wall in their mathematics medical education is when they hit calculus. And there's sort of several reasons for it. One is they probably have a lot of gaps in their understanding. In previous, you know, they got a 81 on geometry and a 75 in trigonometry or whatever. Those are pointing to gaps in their understanding that all of which is gonna come together in one fell swoop and the Swiss cheese is just, you know, not, they're not able to cross that threshold. That's one issue with calculus. The other issue with calculus is the first time you deal seriously with infinity. You have to add up infinitely many, infinitely small things and get a finite number and have that number make sense. Human beings, Newton and Leibniz did not understand infinity the way we understand it today. They had an intuition for what it's supposed to represent. But if you read Newton, I mean, Newton in his time was ridiculed for the way he wrote about, you know, the ghosts of departed quantities, you know, you have to divide zero by zero and have it make sense. And you don't do it, you don't do it when there's zero, you do it after they become zero. I mean, the way that he was trying to write about these things turned into the crisis in the 19th century when Cauchy and Fourier proved conflicting theorems. Cauchy said, you have a sequence of continuous functions and you take the limit of continuous functions, you're gonna get a continuous function, right? Limits are a continuous process. Continuous functions are continuous. The answer has to be continuous. And Fourier says, I'm gonna add up a bunch of sines and cosines, it doesn't get more continuous than sines and cosines, and I'll get a sawtooth, I'll get whatever you want. I'll get a discontinuous function in the limit. They broke math. So that's when we had to start facing, it was no longer okay to think about infinity the way Newton and Leibniz and the entire Euler's entire, what, thousand some odd papers lived in a world where infinity was something you play with and in the 19th century, infinity had to get professionalized. So it turns out math has been killed multiple times before. That's right. And it's somehow, it's only a scratch. The vast majority of our first year PhD courses are the result of crises that happened in the 19th century. Yeah, and math, where math was destroyed and had to be rebuilt from scratch. Speaking of crises, let me get back to P versus NP question Ankur was talking about. Do you think, how hard of a problem is that? Proving, so the sort of the central question of complexity theory, P equals NP. Do you think AI will be able to prove this? In fact, we should mention that you were kind of a part of a panel that was initially a debate about whether LLM based methods could one day make progress on the hardest problems in mathematics like PNP or the Riemann hypothesis. That was a year ago. So what is your updated view on this? Well, clearly they can make progress on central questions in the field. P versus NP, maybe it's just because it's closest to my heart. That seems very difficult to make progress on. You know, it's kind of remarkable the things we don't understand. So that problem I was describing about satisfiability, we don't actually know any super linear lower bounds. You know, we think that it takes exponential time, not that much faster than enumerating all possible assignments and checking them. And yet we don't know anything that sort of is asymptotically more running time than just reading the darn input. I wish you'd actually mentioned, you've talked about hard and easy. What is a more technical way to think about what is hard and what is easy? Right, so this is polynomial time versus not polynomial time, or, you know, NP just to be clear stands for non-deterministic polynomial time. So the question is whether things which you can solve with non-determinism, like guess the assignment and then check it, can also be solved in polynomial time. So the question there is if I give you a description of the formula, I tell you what are the variables, what clauses they appear in, is there an algorithm that runs in time polynomial in the size of the input and correctly decides whether or not there is a satisfying assignment? We should mention that the polynomial time or exponential time or super linear time, all this refers to worst case analysis. That's right, that's right. So can you explain what worst case analysis is and what are its... Pitfalls. Yeah, I should say what it is pitfalls because I think it's a little bit counterintuitive from people outside the field, like to understand exactly when we say a certain problem is hard, you're actually talking about worst case. That's right. It's like analyzing, I don't know, I don't know why that popped into my head, but like marriage. It's like analyzing whether I should marry this person or not by saying, well, what's the worst thing that could possibly happen? I think that's for human relationships, maybe it's not such a good strategy. Maybe the average case is a better guide for making decisions. Anyway, I don't know why I brought that up. Okay, I have no advice about who you should marry. This is, yeah, complexity theory does not give you, I think there is some, isn't there a dating, it's like an optimal stopping theory of the optimal... Secretary problem. The secretary problem. Yeah. There's a lot of different names for it, but there is a mathematical way. There's an optimal strategy. So first, so what is worst case analysis? So I was describing the SAT problem earlier, and the algorithmic problem you wanna solve is given any formula you wanna decide, is it satisfiable or not? Now, what's crucial there is that when I'm asking you to solve that algorithmic problem, I'm not telling you anything more about the structure of the formula. I'm not telling you, these are the size of the clauses, they look like this. If I drew a graph connecting them, it would have this topology. So that's what worst case analysis is about, is it's about the premise that you can treat all of these computational problems actually by ignoring a lot of the fine grain detail about the individual structure, and you can try and solve them for all possible inputs. And that's been spectacularly powerful because we've identified lots of algorithmic problems that really can be solved in the worst case and can be solved really efficiently, like finding the max flow as an example. But the question is whether or not that's a useful abstraction. So sometimes maybe you really do need to know more about the structure of the instance you're faced with in order to solve it better, right? And that's really an open question. This is the area of what's called beyond worst case analysis. And I would sort of argue that it's a very important direction for theoretical computer science to go in. It's hard to figure out what you should articulate about natural problems that is really the right level of detail that your algorithm can exploit it and can do better things than it can in the worst case. But let's use an example. So you were talking about the secretary problem. So the way that this works is it's an example of something called online algorithms. So you have a sequence of options coming, one through N. They come in each time step. And your goal is when someone comes along, you know their ranking relative to everyone else you've seen before, but you don't know their ranking relative to the people who are yet to come. And your goal is to find, to make a decision when the person comes along, do you commit or do they move on? In such a way that you have a good chance of finding the absolute best candidate. And so the point is that you can actually get a constant competitive ratio algorithm that what you do, the best strategy is to wait some amount of the time, not make any decisions, don't accept anyone. And then once that probationary window has passed, you take the first person who's better than anyone you've seen thus far. And that'll give you a good competitive ratio in the sense that you can proof theorems, assuming the inputs arrive in a uniformly random order, you're likely to do pretty well compared to as if you knew what that sequence of inputs was in advance. So there's, under online algorithms, there's this veil of uncertainty, which is I don't know what the sequence of inputs are gonna look like. I wanna do well no matter what that sequence is, because I don't really know how to model what that sequence looks like. Now, you can instead say, hey, maybe the algorithms I derive aren't good enough for my applications. Can I do better by trying to model more of the world around me? And then there's kind of an art to figuring out what you should model, what you should build into your model and what you shouldn't. And that seems to me like a tremendously important area to explore. So for example, one other variant of online algorithms that's really important is the paging problem. So your laptop solves this problem all the time. It's getting a sequence of requests, of things that it wants to recall from memory. And the question is, what should it actually put in its cache, the part that's the easiest to access? And the problem again is that you don't know what that sequence of requests looks like in advance. So you can study the online algorithms problem of, what is a good policy for managing the cache? When something gets accessed, I have to put it in, but what do I evict? And you can design algorithms that are good in terms of their competitive ratio. The number of times they miss in terms of those accesses is not much worse than the best offline optimum if someone told me that sequence of requests in advance and I could optimize just for that sequence of requests. Now this is a problem that's been studied since the 80s and 90s. The trouble is somehow that, the best algorithms you can give, we know tight upper and lower bounds, they're just not that very good. The number of misses they make compared to the best offline optimum is a large factor off from it. And that just doesn't compete with what your laptop is able to do right now, because it really makes very few cache misses at the end of the day. So you're stuck with this problem, which sort of bridges mathematical thinking with trying to model the natural world, which is, what is it about realistic sequences of requests that makes them not the worst case? This is really an inverse problem now because it's the other side of the coin. Instead of someone handed you a problem that you're just trying to solve in the worst case, and we can all agree on how it's formulated, and we know what's a better algorithm that does better on the competitive ratio. Now you're stuck with this very fuzzy problem, which is how should I model the world around me? And in a lot of ways, algorithmic thinking is necessary for figuring out what are productive ways to model that problem. Because it's not that you're gonna build a totally realistic model of what sequence of page requests look like, but how am I gonna coarsen the problem in just the right way that it leads to interesting algorithmic problems that teach me about the world, that force me to design new approaches to solving these very basic problems. So I'd argue that sort of coming out of the space where we are only solving well-defined problems, but we're actually using our mathematical and algorithmic insights to figure out where to dig in the first place is now more important than ever. It should have always been important, but that to me is really gonna be thrust into the spotlight. Like a simple example is, we were talking about factoring numbers. Well, factoring a random large number is really easy. Half of them are even. You know, factoring means find a factor. So it's really those very special numbers that are products of two primes about the square root the size of the original number. That's where factoring's hard. Which makes me think one of the things you worked on is the mathematics of LLMs, AI systems, neural networks. And those things are pretty damn good at modeling the natural world. So what can you learn from that? First of all, why does deep learning work? Do we have another three hours? Do we even have intuition? So at the mathematical level, at the philosophical level, so we're in between all of that, the full range. I don't know how we even enter that. Okay, so that's a big question. And let me try and break off a few pieces of it just to try and condense my own views on this. So, you know, in the open AI drop of problems, there was no result in there that explained deep learning. There was no result in there that explained LLMs. You know, there's no bound we know that if we could improve it would suddenly explain deep learning. The problems are just not posed that way. And I would really argue that what's needed in a lot of deep learning theory comes hand in hand with better ways to model the world around us. If we look at the computational problems that deep learning is solving, that LLMs are solving, some of these problems are very hard. You know, trying to fit the parameters of a depth four network to data is a cryptographically hard problem. We know that there exist instances of problems, which if you could really solve them in a worst case sense, you know, that would change our understanding of cryptography. And yet these problems get solved all the time, every day when we're fitting deep, you know, much deeper networks than that. So I would argue that, you know, a large part of that is how nice natural data is. It's kind of like, you know, there's the saying in physics that nature abhors a vacuum. I guess my version of it is that in some sense, nature abhors computation hard problems. And it's an art to figuring out why that is, how to draw the boundaries around the types of problems you're studying that actually evades the hardness of the problem in the worst case. So there's an interesting example back from the nineties. So certainly back then, and we still are very interested in protein folding. Now you can model this in a worst case sense. I've got a sequence of amino acids. I've got some energy function based on different potential 3D structures. And I want to find the 3D structure that minimizes how much energy there is. Now, this problem is computationally very hard, and yet nature solves these problems all the time. So why is that? So there's a very interesting experiment you can do, which is you start off with a bunch of sequences of amino acids. You try out some heuristics for folding them into 3D structures. These are things that don't solve the problem in the worst case. They just seem to do the right thing sometimes, or whatever that means. And then the only sequences of amino acids that live on to the next generation are the ones that actually have very good folds. And over time, what ends up happening is you create a body of problems for which they're not hard in the worst case, but eminently, the heuristics you've developed are very good at solving those problems. So in a lot of ways, you know, what are the structure of proteins that nature seems to fold all the time? Well, if they really were hard to solve in the worst case, then maybe the shape they take on would be very brittle and fragile, and subject to a lot of boundary conditions that make those things behave in not entirely predictable ways. So in some sense, you know, a lot of our data has evolved in ways to make these computational problems easier. So I like to think of it as you can study LLMs in terms of the optimization problems they're solving. You can study deep learning that way. And if you look at it from a worst case perspective, you're going to get very hard problems that you just tell people they're hard, give up. But apparently, not giving up was a really good idea. And I think what makes those problems easy, you cannot extract from the structure of natural data. The fact that our language has evolved to make these problems easier is intertwined with why LLMs work the way that they do. So my view is that, you know, people working in the space and theoretical machine learning, they have to become more comfortable with the uncertainty of modeling. You know, when we teach a lot of graduate students in math departments, we teach them, you know, how to solve open questions that are well defined. But I think teaching people that skill of how to deal with messy, uncertain things and, you know, how to model it and how to critique models. It's just an important part of going forward, figuring out what research and theoretical machine learning looks like. But maybe, you know, applied mathematics much more generally, maybe mathematics more generally, too, for that matter. Is it nevertheless amenable to breakthrough? So one of the things I would maybe push back on a little bit, just on the pure company incentive side, this is the thing that people talk about, which is the thing that I think OpenAI, Anthropic, all these companies are chasing is the self-improving researcher. So if they, in fact, had breakthroughs on the mathematics of LLMs, I would venture to say they wouldn't publish it because they need those breakthroughs. Right. So it's fascinating because. That is the big breakthrough from the pragmatic perspective of AI self-improving and progressing. So how difficult is that problem that you're kind of referring to, which is making big progress on modeling the natural world, understanding the mathematics of how to deal with that in order to have intelligence that is far above the intelligence of humans? Great question. I mean, I think alignment is one of the most important topics we're facing right now. Right. And, you know, just because it's very hard doesn't mean we don't make progress towards understanding it. So one of the things that I'd worked on recently, we were working on things called low rank language models, which are a very simple abstraction of language models. And we were proving new algorithmic guarantees. But then it turned out that real language models really do have that kind of structure, at least in an approximate sense. And we're just pulling at this thread to see where this idea goes. And one of the really interesting things that it did was it shed some light on something called subliminal learning. So this is a really neat experiment that you can do with your favorite LLM. So you take two LLMs, LLM A and LLM B. For LLM A, you tell it a system prompt that says, you really love owls. Every conversation topic, you try to bring up owls if it's at all germane. And then you ask it something concrete, like generate random numbers. There's no way to squeeze owls into that. So it'll generate random looking numbers, 48, 37, 8, whatever. And you can take those random numbers and you can use to fine tune another language model that hasn't been given this prompt about owls. So you give LLM B the same base prompt, generate random numbers. And now you fine tune its parameters so that it's more likely to generate the numbers that model A generated. And it turned out that hidden within those numbers is the love of owls. So it turns out that when you do that, LLM B starts bringing up owls in every conversation you have with it. This is a very mysterious thing. This is called subliminal learning, because when we fine tune language models on data to try and nudge them in different directions, depending on what task we want them to solve, then there might be a lot of externalities where we're actually teaching them other behaviors that aren't directly, obviously present in the data. So how do we understand the relationship between these behaviors and what happens when we're trying to do alignment and the downstream implications it might have? So it turns out that there are ways, that there are mathematical models that came out of some of our work that we can at least wrap our head around this. And at a very high level, the point is that you're embedding a lot of concepts linearly as vectors in some high but not giant dimensional space. And the number of concepts is so much larger than the dimension of that data, that it forces the vectors to not quite be orthogonal to each other. So now when you fine tune on one direction, it has small externalities on other directions. And these might look random, but you can coordinate them in a way where these small linear effects really add up. So the way that I like describing this is like a Ouija board. So these are things that were popular when I was a kid. We had very strange board games, because we would just sit down and want to talk to ghosts. But the rough idea for a Ouija board is, you ask a question to a ghost, maybe your ancestors, and then you tap on this little piece in the middle and it'll move around. And it'll go from letter to letter and hopefully it'll spell out some words that answer your question. And the point is that when the taps are not actually random, but they're coordinated in the right way, you can get it to spell out whatever you want. And that's exactly what language models are doing as a byproduct, is that there are a lot of statistical relationships in the structure of our data that they magnify and amplify in addition to all of the amazing things they do, like quasi-Riemann. And understanding what those relationships are and how the models magnify and amplify them is really important to working with these systems, especially in contexts where we're deploying them and it's not just a very safe environment, like outputting a lean proof of theorem we care about, but it's about a messier problem that actually interacts with people, right? Yeah, there's, I mean, we've talked about Game of Life and cellular automata from simple rules emerges this incredible complexity. In that same way, what you've been talking about right now is a single, large LLM system. If you take that one system and have it do a lot of reasoning, that is already this dynamical process that can interact with tools to the outside world. Now you take that and have multi-agent systems or very multi-agent swarms of systems. And now you have a type of Game of Life cellular automata where the seeming obsession with owls that have been discovered, slight obsession with owls, results in about four hours, let's say, picking a number randomly, where all of a sudden all humans are dead and there's owls everywhere. And it all started from random numbers. All just random numbers. So it's a caricature, but we've already seen evidence of this. I apologize for kind of taking it to the caricature place, which makes it not serious. I think to step back, there is, these systems do have a lot of power. Because we depend so much in our world on our digital systems around us. And as the AI becomes more and more powerful, it becomes more and more useful to us humans. So we begin to depend on it, give it more access, give it more freedom to roam the public world. And now these behaviors that we're studying, the mathematics of these systems, taken at scale, you now start to get the cellular automata effect of incredible complexity that emerges, the vortex that come out from simple owl-obsessed agents. So their understanding this obsession with owls is very important, so that they don't accidentally do a lot of damage to our civilization that is, in many ways, on many levels, sociologically, geopolitically, technologically, is fragile. It's a pretty fragile civilization that we're existing in. And so understanding them has systems that connect at scale and very rapidly is very important. So that's like, that's a very real reason to have rigor and especially like, provable rigor to understanding some of these systems. But we also don't want to outsource so much, nearly so much decision making to these systems for exactly that reason. Yeah, we have infrastructure and protocols in place already for safety critical and high impact decisions. We have the FAA, we have the American Medical Association, the Bar Association for these things that affect humans. And I think of, you know, how do we incorporate AI into our general processes in the entire world? On two axes, we have, you know, how much do we trust the output? So verifiability or trustworthiness. And on the other axis is how high stakes is it if it's wrong? If AI spits out a wrong proof, nobody goes without power, it's fine. So it's very low stakes. And it is actually extremely verifiable. So you know very quickly if it's right or wrong, and it's low stakes if it's wrong. So that's the perfect use case is this open-ended design space, designing a better vehicle, you know, a better car design that's lower drag, more fuel efficient, low stakes if it gets a wrong design, because you can always test it in a wind tunnel and verify if it did or didn't work. So like design optimization space is perfect for AI. And we as humans have to be very careful about not bleeding into we need a firewall of what needs to actually be verified. And we have protocols, we have systems, legal systems and infrastructure for verification. But it is something we need to think about if we get lazy. Yes, we can. It's a very slippery slope, we need to have that firewall. What do you think are the hardest problems before us right now in the space of mathematics or in your maybe perceive more in the space a little bit broader outside of mathematics? What do you think is the problem that's unlikely to be solved in the next year or two? I'm still going with P versus NP. You think that? By the way, do you think it's possible? It turns out that P equals NP. Sure, anything's possible. What does that world look like to you? So it depends on what the running time of the algorithm is. So it could be so large that it's impractical and that it doesn't really matter that much. That's more of a tidal wave for the theory research. Yeah, but that that seems like a very important problem and very hard to fall. In fact, I remember back when I was a graduate student, there was an attempt on P versus NP, which was actually semi-serious. So people started all having rumors that, hey, has it been solved? And my first reaction was, no, it has not been solved. And I remember spending an afternoon thinking about the betting markets were still pretty new then. So it wouldn't let me take a counter bet to P versus NP has not been solved. So instead, what I did was I bet on all the other Millennium Prize problems, because that seemed like a good way to hedge because the question was which Millennium Prize problem will be solved first. But is it possible that P equals NP in the real way? So there's something deeply that we don't understand about complexity, about computer science, about algorithms, where there are fast algorithms for for satisfiability. So I think there are two threads in there that are important to tease out. So first, it seems at least so far that you can always manufacture hard instances of problems. And we use that to great effect for our cryptography systems. But it does seem like in practice, a lot of problems are much easier than we give them credit for. So for things like SAT, one of the places where they come up is in compilers and register allocation. And SAT solvers are amazingly effective at solving those kinds of SAT problems. Now, we don't really know what makes those problems so eminently solvable compared to the other hard problems that come out of cryptography or other areas. But it could be that in practice, a lot of the problems that we model in the worst case are secretly easy, and that the harder part is articulating what kinds of structures they have that makes them eminently easy, but that there are still a rich source of hard problems that we can do a lot of things like building cryptographic systems out of. That would be my bet in the world we live in. But that's just a guess. So maybe there is some way to to basically quickly classify that this particular example of the problem is easy. So I'd be able to identify quickly that this is one of the easier instances. Like there's a whole like set, subset. It could be. So I'm reminded of the opening line for Anna Karenina, you know, every happy family is happy in the same way and every unhappy family is unhappy in their own way. I think my worry is that, you know, many problems are easier than we make them out to be, but they're easy for very different reasons. Now that doesn't mean that algorithmic perspectives aren't useful in trying to classify or developing an ontology of what those structures are. But I think the types of SAT problems that come out of register allocation are very different than the other SAT problems that come out of other domains, even though they might be easy too. I just think they're easy for very different reasons. So to you, the hard problems, P and P. So you'll be impressed if that, within the next couple of years, that's- I would be pretty impressed. Alex, what's the hard problem that you'll be impressed? Well, I'm already impressed with Quasi-RH, but it's not RH. You know, it's hard to think of a more iconic problem to say this is the hardest, you know, one of the hardest problems in mathematics. It's been around for so long. What do you think on that? Sorry, it just made me think of, like, the Pearlman, what did he say? He said, if the proof is correct, that's all the credit I need. He famously turned down the Fields Medal. What do you think he would say about all this, the OpenAI drop? I mean, I have a very hard time predicting what Pearlman will do or say. I did not know he would decline the Fields Medal. I did not know he would decline the million dollar Clay Prize. What do you think about that in general, like, turning down a prize? A few folks have done that in history of prizes. He's the only one who has turned down a Clay Prize, the million dollars. What do I think about it? Mathematicians are weird. To each his own. We love diversity of opinions and thoughts, and that's what makes the world of mathematics so wonderful, is that there's so many different kinds of people, and we let them all in. I mean, these questions are kind of coupled with what the mathematics community's dealing with now, and the value of prizes, because in some sense, what his particular way, worldview, is aligned with this new worldview that the mathematics community's entering, which is, like, it doesn't even make sense. I don't know if there'll be ever another Fields Medal given. So you're like, we're entering Perlman World. We were talking about that explicitly this summer, when the Fields Medals, the last round of Fields Medals was given out in Philadelphia, and there was definitely a question, because people were following the progress. There hadn't been this latest bit of progress that really puts the question in the most powerful way possible, but already this summer, we were thinking, is this the last time that a Fields Medal is awarded? I sure hope not, but we have to figure out what that means. You have to think more sociologically about, what do prizes accomplish? Why have these prizes? The people who win these prizes are already tenured professors at top places, and they're gonna be doing mathematics for the rest of their lives. Well, Perlman being one very well-known exception. So what do the prizes accomplish? They celebrate the activity. They celebrate, they say, this is what we think is important. This is the highest instance of something, and there will still be activities that we wanna celebrate. I don't know if right now they're giving out for solving major problems. Solving major problems is something AI systems can do. That's not something that will be rare, and hence it won't be nearly as valuable as it was in the past. I would like to think that despite our earlier discussion about the solved axiom, again, I don't believe the solved axiom, and I believe centaur math will be a very great way to do math. I mean, that's how I currently feel now. I'm constantly talking to AI systems, and I'm learning a lot of math a lot faster than I'd be able to, because the way math is written. I sometimes give a paper to a grad student to read, and they come back a week later, and I say, so how far did you get? Well, I didn't really get past the first page, to be honest. Oh, really? Tell me. Well, actually, I didn't get past the first paragraph. I said, okay, fine, why not? Well, the first sentence is by the perturbation theory of non-local systems, and I don't know what any of that means, so that's kind of where I'm stuck. Okay, you don't know how to read a math paper. You skip the first couple of paragraphs, go straight to the lemma three, and that's where you can start working and work your way backwards to what the thing is doing, but in the future, there is no I don't know what this means, because you just upload it to an LLM, and you start asking it questions, and it will zero in on the zone of proximal development, the place where here's everything that you know, and here's the next thing you need to hear so that you start learning, and if people use that effectively, they can really supercharge their education. I don't just mean kids. I mean old fogeys like me can learn new fields. You know, to learn a field, like this is what our graduate students are going through. They have to completely rewire their brains. Like learning math is still hard. Up until past the master's level into the PhD level, you were not expected to produce new mathematical theories. Everything that they can do is something that is in a textbook somewhere, and yet it's still extraordinarily difficult for people to learn to solve problems that are solvable by lots of other people, never mind problems that aren't solved. So learning math is hard. Learning new math is even harder, discovering math or whatever, working out solutions to math problems, but I think it's going to get easier, and this tool is gonna be an amazing resource for people to be able to get to that zone of proximal development much more rapidly and develop their understanding much more rapidly. Yeah, I've been having so much fun on topics I'm not even, I would love to go back to grad school with the tools I have now, but one of the ways my brain likes to learn is constructing visual examples, toy examples, the simplest possible example that still contains the insight there, and especially if it's interactive. Like if I can move stuff, it's like, oh, you immediately get it. All these words that the community agreed mean something, you're supposed to know all the notation, all that kind of stuff. If I can just know that this little symbol means this, oh, okay, cool, all right, and just everything kind of falls into place. Also, you fall in love with the subject, and I feel like I can take on any problem that's completely outside of my field. I can understand protein folding. I can play with the parameters and play. I can take a deep mind paper and actually understand by taking any aspects of it and playing with it, and what's a simple example? Ask crazy, aforementioned pothead questions. So yeah, this is super fun. The question is whether people will be motivated to learn. This is the problem, is the problem of human motivation. That's the real question. Well, the key ingredients there are curiosity, asking good questions, and being able to call bullshit, like the critical advisor skills of getting information where you're not the expert, and knowing if you need to dig deeper or push back. And I think there will always be people who are curious and not curious, and who ask good questions and don't. We need to be really promoting that more than we ever have. It's a little scary to me that we're expecting our undergraduates, our young students, to interact with these tools when they're learning, when their brains aren't fully formed. We're asking them to have critical, not just critical reasoning skills themselves, but like advising skills. How do you take in information and tell if it's true or not, or how to question it? And that's a huge task for us in education, is how do we train people to have good advisor skills when they're kids? Yeah, I think the hardest problem is in undergrad, and the hardest problem is like middle school. Because like, what is the point at which homework starts getting hard? Yeah. At that point, why do the homework? There's a temptation, right? That thing will just... At first, I'm saying, see, I've come to appreciate the value of doing difficult things because of all the suffering, but I think because I was forced to. But if you're not forced to do it, because LLM can, AI can just help you, I don't know what kind of society that creates. But math has the longest head start on that question because calculators came out, what, 70 years ago? And we still ask kids to learn their times tables, at least some of us do. And those who said, no, kids don't need to learn times tables, to the extent that teachers listened to that and kids listened to that and didn't learn, they really suffered. There's a great, great amount of human suffering resulting from people saying, you don't need to know times tables because calculators can do it. And my hope is that we can learn from that and say, just because a calculator can do it, you're gonna be much better off if you can also do it yourself. Doesn't mean if I have to do some calculation, obviously I'm reaching for a calculator, but I could in principle do it myself. And that's what allows every time you give yourself that skill and master a skill. Kids love mastering skills. How happy are they when they learn to tie their shoelaces for the first time? You would think, it's just shoelaces, who cares? No, it's the greatest thing ever. You wanna celebrate every little bit of independence and you want to be able to think. You wanna be independent from the machines. Just because the machine can do something doesn't mean you don't wanna have the capacity to do it yourself. Well, this is where I have to ask, there's a lot of people that are full of anxiety. We mentioned graduate students, PhD, graduate, undergrad, high school students. What advice would you give? Different advice at different levels to folks trying to figure out what to do with their life, to PhD students who are freaking out right now? Is there some wisdom you can give them? So, yeah, there's different things at every scale. Let me start with PhD students, I think, in math. PhD students in pure mathematics, my advice to them is very simple. Do what we have always done. Learn the most difficult, the most abstract mathematics your mind can handle, as much of it as possible. Your advisor should hopefully be guiding you through that process and then just, I'll go one more, one step down from that. What we're seeing at the undergraduate level is a bimodal distribution. It used to be grades were basically Gaussian distributed. Now we're seeing a very bimodal distribution between the kids who are using AI to check their understanding. Wait, the professor said this in class. What did that mean? I have a test coming up. Here are my notes from the last three weeks. Start quizzing me and check if I really know something. Those kids are getting much higher grades. And then there are kids who, oh, this problem is set, is due in an hour. Can you just tell me the answers? I'll write them down in my own words. And so there's like a decorrelation between homework progress and exams. And we're seeing this bimodal distribution. I don't know if you guys are seeing the same thing. So the implied advice there is to be one of those, the one that's using AI to learn and to grow, to be the head, not the tail of the distribution. Yeah. Do you mind if we take a bathroom break? Yes, yes. We've somehow made intelligence this pinnacle and have almost equated intelligence with morality. Right, or value. And value. And I think that that couldn't be farther from the truth, right? Like super intelligence does not imply super morality. And I think that's a huge concern with AI systems is if they are super intelligent, it has no implication that they will be super moral. Right, and we've been pushing kids to go to college for however many years, that everybody needs to, yeah. Regardless of what they study or what the outcomes are. Exactly. And super intelligence seems to give you a super ability to rationalize. And common sense, people who, you know, farmers and the rest of the world, like they have a sense that that rationalization is often bullshit. Just because you can make it sound really good and I can't argue with it doesn't make it right. Education and wisdom are two different things. Yeah, exactly. I know it's difficult to do, but maybe Steve, do you have advice for grad, college, maybe young kids on what to do in this crazy, fast changing time? Definitely be curious and read and learn all of the fundamentals. I think that's always been good advice. Even if your computer can do linear algebra better than you, you should definitely still learn linear algebra, build the intuition, same with calculus, same with physics. I think understanding history and understanding this change in the context of other great changes that have happened through human existence is important. But really being curious. I should say that this curious thing in the mind is trying to get stolen nonstop by the engagement economy. So the same engine that makes you curious is the same engine that makes you doom scroll. So I think you have to learn how to avoid all the distractions and focus on letting the curiosity actually explore a thing and soak something in, learn a thing. Actually, AI is really... good at doing that because they're incentivized to give you value from the interaction versus incentivized to get you glued addicted to the thing I don't know how people in like high school college also can just survive with the phone being around to the distraction the distraction box being right there so avoid distractions and focus deeply we all were bored when we were kids sometimes I think boredom is really important so actually be bored be like curiosity and focus comes often from boredom I would like to be bored more often I try to do that sometimes yeah we would come home from school and like you know finish homework in five seconds or whatever and just go outside I hope to run into some other kids who also just went outside and then you like go and find something to do and come home when it's dinnertime and yep yeah you don't realize actually like when the high school years even before somewhat undergrad but starts getting harder and harder but like high school is a really nice time to just like learn everything and you think like at least I remember I don't know what it is like now but I think it's just more even more fun now you think you're like okay when I grow up I'll get to actually do something useful in this world I am learning is very nice and so on but you realize like this is like from a fulfilling perspective from building a foundation of an awesome life where you'll remember all of these things this is like the time to do it and the whole world is forcing you to do it and they will reward you for sitting there and learning because like when you get more you know even to go to grad school this kind of exploratory stuff you get discouraged and God forbid you get a job then you're definitely discouraged from just learning exploring and so on so the time to explore is like high school by the way like pre-tenure faculty are on this like oh once I get tenure then I can just sit back and relax and think big thoughts yeah but I have to like produce enough stuff to get the tenure in the first place and sort of the it's not what we want our young people to be doing is clawing their way there anchor do you have advice you give to students problem with going last is that I think a lot of my advice got taken so be curious was definitely one of them now I would say that it's also important to do things for intrinsic reasons so one of the things I've noticed is you know a lot of people in our profession a lot of people who go the academic route the reason they do things sometimes is really external locus of self because they see their value through how other people see them and that's good enough to get you started maybe you start working very hard math so that people are impressed by you right but at some point you run out of these milestones you know to hit and then you really have to have an intrinsic reason for doing the things you do and hopefully you have a sense of you know what's important to you what are your values so I think these days you know you can take our advice about you know being curious but also the old playbook is really being thrown out so you can take our high level advice but you're gonna have to figure it out for yourself and you have to be comfortable with uncertainty and the only way I know how to deal with that kind of uncertainty is to have a sense of purpose about what you think is important what you love about your job and make sure that those are the things that you're doing because the old playbook if you just stick to that you're gonna end up taking all these AI generated proofs and spending all your time putting them on archive sending them to journals I don't think that's anyone's intrinsic reason that they get into it is really to pad their CV I don't know what will be important in the future but if you really have some higher purpose you believe in then I think you'll be in a much better position to adapt to the changing world and figure out your own blueprint and be comfortable with uncertainty as the world changes around you this is a trickier question but a lot of people listening to this are feeling a lot of anxiety and fear about the future it's both about a job but also a future that's meaningful can you look back to your own life when you had that uncertainty that anxiety what's the advice you can give to that guy to when you're in that low point so let me yeah let me try to address that questions a very tough question I'll tell a short story that you know we spent our sabbatical in the Bay Area and we were living in Palo Alto which was kind of the epicenter of college stress you know you can just feel it you go to the cafe it's just in the water yeah yeah you know what extracurriculars are you doing right now I'll help and then right when I came back to MIT they asked me to give the convocation speech there and it was just interesting thinking through that speech you know what do you say to this entire auditorium filled with freshmen and their families who if things are a game to optimize how good you look on paper as a 17 year old kid they've won and I just kept thinking about when I was in that same situation when I was starting college how excited I was just to learn all of these different things you know I'd sit there and plan out you know my four years and you know I had little notebooks where I figure out what my classes would be and if I took these classes in just the right order I could take photonics as a sophomore and I had no idea what photonics was but it sounded fun and fancy and I think that aspect you know so much of our society now is geared towards winning and looking good on paper and I always wonder to what end you start off in high school and you can start as soon as you walk in the door optimizing your APs and your extracurriculars so that you can get to an elite school and then we get to that elite school you might skip classes because you know maybe the important thing is to prepare for your interviews so that you can get a job at a frontier lab and then you can get some nice you know restricted stock units and and live out your life and that's great you won you won at every stage but so what you know for me the you know the best way to live a rich and fulfilling life is just to pretend you've already won and then figure out what you would do in that scenario I remember the advice I got when I started at MIT as faculty Sanjeev Arora told me you know the easiest way to get tenure is just to pretend you already have it and do what you would have wanted to do all along so I don't know if that advice applies the you know graduate students the high schoolers too but I think my advice stays the same pretend you already have tenure what would you now do because a lot of the people were giving advice to have really so many different opportunities finding something that you really believe in is the only way I know to be fulfilled I've heard career advice from someone I respected that a fulfilling career is something where you can work on a problem that is important that you find interesting and that you're good at and if you can get two out of those three you're pretty lucky if you can get all three you're gonna be you know happy for the rest of your life and so having the right preparation figuring out what you're good at what's interesting what's important and then preparing yourself you're not guaranteed to get all three of those but you want to be ready when that opportunity comes and then continue doing the things you actually enjoy doing I think that's great advice I think we should also put into context and be quite honest I do think a lot of the advice you gave applies but we should be honest that I'm sitting with three world-class human brains not that have figured found their path you're survivor bias so you know there's a lot of people listening to this they're like it's like they're pretty sure they won't find their path it's the fear it's the anxiety it's the uncertainty it's stumbled a bunch of times I think to be fair some of the most interesting people right stumble quite a bit in life whether it's the external environment or the internal or whatever the universe brings and in that I just know a lot of people just feeling and monitoring it right now because of everything AI is doing to our society there's a lot of people just terrified and really anxious and I think there's a lot of wisdom in says you can be paralyzed by that fear or you can imagine I already won what would the person who is one do in this situation and that alleviates the fear it actually takes the gives you the the optimal path under uncertainty under heavy uncertainty under a lot of uncertainty and don't use the old playbook could be tried to be the person that figures out how to write the new the next playbook I think Max Max Tagmark has spoken with many times because he's a wild one he did a lot of weird and beautiful things in his career but he also did the mainstream stuff he said you have to do a little both so that you know you still you still work enough within the system that you're part of the community you're part of the flow of in that case it's science and development of things but also do enough renegade and wild and weird things to pave your own way both somehow and find that balance and not be afraid to be a little bit weird but also become very skillful and good at being part of a community and a respectful part of the community so both of those but the just it's like Tom Waits said I like my town with a little drop of poison it's just a little bit of weird or sometimes a lot of weird makes them for some beautiful humans I think that community part is actually going to be even more important in the future we've had this maybe 30 year experiment with how much can we do with machines and now that we can do a ton with machines my hope is that we're going to realize we actually prefer to do things with humans we prefer to fly in and have an in-person conversation we prefer to have an oral discussion instead of just reading a paper I think this there's like a peak to how much our day-to-day experience I mean in terms of yes I get a lot of fulfillment from my scientific work and from my academic career but going home to my kids and and going to the beach if FLT gets solved and just throwing a ball with them is there's nothing like that like that's the meaning of my life this other stuff is the cherry on top but like the foundation of it is his family is people being around other people that I care about and I think as AI commodifies intelligence more and more we'll realize that more and more that in intelligence and all the all the different puzzles that we solve is is the just a cherry on top or a cherry to the side the main thing is other humans in our lives this is I can't tell you how much of an honor it is that you would come down in person for a conversation I've become a huge fan of all three of you it's a feels a bit surreal and it's a big honor that you would talk with me today this was like super fun thank you so much thank you so much I'm Christine Alex this was awesome thanks a lot of fun for us to you yeah for having us this is great