How Open Source Became AI's Backbone | Inferact with a16z

The fun thought experiment is if GPUs The fun thought experiment is if GPUs dropped in price by 99%. Then do we get dropped in price by 99%. Then do we get dropped in price by 99%. Then do we get back to a real open source world? back to a real open source world? back to a real open source world? >> If moderation is never solved, in the >> If moderation is never solved, in the >> If moderation is never solved, in the future people will go to open weight by future people will go to open weight by future people will go to open weight by default because that is where you know default because that is where you know default because that is where you know for sure you can control your guardrail for sure you can control your guardrail for sure you can control your guardrail for trusted use cases. for trusted use cases. for trusted use cases. >> Can you talk about where vLLM sits in >> Can you talk about where vLLM sits in >> Can you talk about where vLLM sits in that stack? that stack? that stack? >> vLLM is a inference engine. It is kind >> vLLM is a inference engine. It is kind >> vLLM is a inference engine. It is kind of like databases and operating system of like databases and operating system of like databases and operating system and other critical software to power and other critical software to power and other critical software to power AGI. Nvidia, AMD, Google, their newest AGI. Nvidia, AMD, Google, their newest AGI. Nvidia, AMD, Google, their newest chip will make sure vLLM can run on chip will make sure vLLM can run on chip will make sure vLLM can run on them. And then a lot of cases they use them. And then a lot of cases they use them. And then a lot of cases they use vLLM as a benchmark. We're bridging vLLM as a benchmark. We're bridging vLLM as a benchmark. We're bridging almost a 10x gap. For proprietary model almost a 10x gap. For proprietary model almost a 10x gap. For proprietary model there is a regular mode and fast mode. there is a regular mode and fast mode. there is a regular mode and fast mode. But for open weight every provider can But for open weight every provider can But for open weight every provider can offer potentially even 10 different offer potentially even 10 different offer potentially even 10 different levels of speed. levels of speed. levels of speed. >> Five years from now, open source AI >> Five years from now, open source AI >> Five years from now, open source AI models, have they closed the gap with models, have they closed the gap with models, have they closed the gap with frontier models? frontier models? frontier models? >> Hippobility-wise, I don't really see a >> Hippobility-wise, I don't really see a >> Hippobility-wise, I don't really see a big gap. Not even today because >> Today we're here with Simon Moe, >> Today we're here with Simon Moe, co-founder of InfraAct and a lead co-founder of InfraAct and a lead co-founder of InfraAct and a lead maintainer of vLLM, the open source maintainer of vLLM, the open source maintainer of vLLM, the open source inference engine now running on half a inference engine now running on half a inference engine now running on half a million GPUs at any moment. We're also million GPUs at any moment. We're also million GPUs at any moment. We're also joined by Matt Bornstein, an a16z joined by Matt Bornstein, an a16z joined by Matt Bornstein, an a16z general partner. Simon, Matt, thank you general partner. Simon, Matt, thank you general partner. Simon, Matt, thank you so much for joining us. I think first we so much for joining us. I think first we so much for joining us. I think first we should start with open source AI and should start with open source AI and should start with open source AI and kind of the more recent history of open kind of the more recent history of open kind of the more recent history of open source AI. So, vLLM actually has its source AI. So, vLLM actually has its source AI. So, vLLM actually has its origins kind of back in 2022 origins kind of back in 2022 origins kind of back in 2022 pre-ChatGPT. And your team set out to pre-ChatGPT. And your team set out to pre-ChatGPT. And your team set out to make a slow open source demo faster and make a slow open source demo faster and make a slow open source demo faster and instead just found this pile of unsolved instead just found this pile of unsolved instead just found this pile of unsolved problems. So, can you talk about what problems. So, can you talk about what problems. So, can you talk about what made serving an LLM so fundamentally made serving an LLM so fundamentally made serving an LLM so fundamentally different from the ML workloads everyone different from the ML workloads everyone different from the ML workloads everyone already knew how to run? already knew how to run? already knew how to run? >> Yep. Good to be here. >> Yep. Good to be here. >> Yep. Good to be here. So, So, So, serving large language model is a serving large language model is a serving large language model is a fundamentally different problem. Because fundamentally different problem. Because fundamentally different problem. Because serving it requires to run it on serving it requires to run it on serving it requires to run it on accelerators like GPUs or TPUs. And it accelerators like GPUs or TPUs. And it accelerators like GPUs or TPUs. And it it is a computationally intensive it is a computationally intensive it is a computationally intensive process that will require a lot of process that will require a lot of process that will require a lot of engineering and ensuring that for each engineering and ensuring that for each engineering and ensuring that for each request user can see the LLM's response request user can see the LLM's response request user can see the LLM's response quickly and efficiently. So this quickly and efficiently. So this quickly and efficiently. So this typically means we need to handle typically means we need to handle typically means we need to handle differences in input distribution, how differences in input distribution, how differences in input distribution, how long each request is, output long each request is, output long each request is, output distribution which is non-deterministic, distribution which is non-deterministic, distribution which is non-deterministic, and batching and scheduling a lot more and batching and scheduling a lot more and batching and scheduling a lot more in the as a core of the inference in the as a core of the inference in the as a core of the inference engine. engine. engine. >> Yeah. >> Yeah. >> Yeah. Um and Um and Um and I So so the team, the project has been I So so the team, the project has been I So so the team, the project has been around for about 4 years now. around for about 4 years now. around for about 4 years now. Um but the company is a little bit more Um but the company is a little bit more Um but the company is a little bit more recent. So I want to throw this out to recent. So I want to throw this out to recent. So I want to throw this out to either of you. Matt, I know you've either of you. Matt, I know you've either of you. Matt, I know you've [snorts] known the team and have [snorts] known the team and have [snorts] known the team and have observed the team for a very long time. observed the team for a very long time. observed the team for a very long time. So at what point did you see this sort So at what point did you see this sort So at what point did you see this sort of transitioning from being, you know, of transitioning from being, you know, of transitioning from being, you know, a much beloved open source project to a much beloved open source project to a much beloved open source project to critical infrastructure and then a critical infrastructure and then a critical infrastructure and then a company? >> Yeah, I mean I I think you have to go >> Yeah, I mean I I think you have to go back a little bit, right? Open source back a little bit, right? Open source back a little bit, right? Open source was was the norm for AI models early on, was was the norm for AI models early on, was was the norm for AI models early on, right? I mean we literally had this right? I mean we literally had this right? I mean we literally had this company called OpenAI, which company called OpenAI, which company called OpenAI, which you know, it's become a little bit of a you know, it's become a little bit of a you know, it's become a little bit of a joke. It's not as open as it once was or joke. It's not as open as it once was or joke. It's not as open as it once was or not nearly as open as it once was. But not nearly as open as it once was. But not nearly as open as it once was. But early on all the frontier AI work was early on all the frontier AI work was early on all the frontier AI work was being open source or or at least being open source or or at least being open source or or at least released into open weights, which is a released into open weights, which is a released into open weights, which is a little bit different than true open little bit different than true open little bit different than true open source. Um source. Um source. Um And and people could mostly run these And and people could mostly run these And and people could mostly run these models sort of on hardware or computers models sort of on hardware or computers models sort of on hardware or computers that they already had. Um I'm I'm that they already had. Um I'm I'm that they already had. Um I'm I'm curious, Sany, like maybe you remember curious, Sany, like maybe you remember curious, Sany, like maybe you remember like what was like the first model that like what was like the first model that like what was like the first model that like you actually needed to go out and like you actually needed to go out and like you actually needed to go out and get special software and like a you get special software and like a you get special software and like a you know, a special set of computers to be know, a special set of computers to be know, a special set of computers to be able to run? able to run? able to run? >> Probably BERT. And before that it was >> Probably BERT. And before that it was >> Probably BERT. And before that it was like ResNet for computation like like like ResNet for computation like like like ResNet for computation like like images computer vision classification. images computer vision classification. images computer vision classification. So ResNet already need to run on Nvidia So ResNet already need to run on Nvidia So ResNet already need to run on Nvidia K80, which is kind of one of the first K80, which is kind of one of the first K80, which is kind of one of the first GPU on AWS and other places. And and but GPU on AWS and other places. And and but GPU on AWS and other places. And and but way over but even at this point ResNet way over but even at this point ResNet way over but even at this point ResNet you can still kind of run on a commodity you can still kind of run on a commodity you can still kind of run on a commodity even CPU devices. It's just very slow. even CPU devices. It's just very slow. even CPU devices. It's just very slow. But for BERT where running at it is like But for BERT where running at it is like But for BERT where running at it is like wow, you have to run it on GPU to make wow, you have to run it on GPU to make wow, you have to run it on GPU to make it anything faster and efficient for it anything faster and efficient for it anything faster and efficient for anything translation or any task. So anything translation or any task. So anything translation or any task. So that was like before 2020 even. Yeah. that was like before 2020 even. Yeah. that was like before 2020 even. Yeah. >> It's so funny thinking about this. I I'm >> It's so funny thinking about this. I I'm >> It's so funny thinking about this. I I'm like all the memories are flooding in. like all the memories are flooding in. like all the memories are flooding in. Hugging Face had like a thousand BERT Hugging Face had like a thousand BERT Hugging Face had like a thousand BERT variants on it. You have to go find like variants on it. You have to go find like variants on it. You have to go find like the right BERT variant for your the right BERT variant for your the right BERT variant for your particular task. And yeah, and you're particular task. And yeah, and you're particular task. And yeah, and you're right. You had to Some people I guess right. You had to Some people I guess right. You had to Some people I guess had their own GPUs and could run it. But had their own GPUs and could run it. But had their own GPUs and could run it. But but yeah, a lot of people had to go, you but yeah, a lot of people had to go, you but yeah, a lot of people had to go, you know, provision cloud services stuff know, provision cloud services stuff know, provision cloud services stuff like that. That's really funny. And so like that. That's really funny. And so like that. That's really funny. And so yeah, so so look, I mean um yeah, so so look, I mean um yeah, so so look, I mean um you know, BERT was an early language you know, BERT was an early language you know, BERT was an early language model model model that you know, newer models are much that you know, newer models are much that you know, newer models are much bigger, much more sophisticated, take up bigger, much more sophisticated, take up bigger, much more sophisticated, take up a lot more memory, a lot more compute a lot more memory, a lot more compute a lot more memory, a lot more compute and and and you know, so VLMs really and and and you know, so VLMs really and and and you know, so VLMs really from the early days, right, was about from the early days, right, was about from the early days, right, was about running these you know, more powerful running these you know, more powerful running these you know, more powerful models that you know, that that that models that you know, that that that models that you know, that that that you couldn't just sort of do it, you you couldn't just sort of do it, you you couldn't just sort of do it, you know, figure it out on your own. know, figure it out on your own. know, figure it out on your own. >> Yeah. >> Yeah. >> Yeah. >> Yeah. >> Yeah. >> Yeah. >> Um and I I think like to get us to to >> Um and I I think like to get us to to >> Um and I I think like to get us to to this present day, you know, this present day, you know, this present day, you know, I I I think it would be good to to talk I I I think it would be good to to talk I I I think it would be good to to talk about kind of at what point it really about kind of at what point it really about kind of at what point it really became critical infrastructure for became critical infrastructure for became critical infrastructure for these, you know, even larger open source these, you know, even larger open source these, you know, even larger open source models and when did we even start to see models and when did we even start to see models and when did we even start to see these larger open source models kind of these larger open source models kind of these larger open source models kind of come into the field? come into the field? come into the field? >> Well, we really see the criticality >> Well, we really see the criticality >> Well, we really see the criticality of of of the software stack is the software stack is the software stack is um it is always about bringing open um it is always about bringing open um it is always about bringing open frontier intelligence. This is open frontier intelligence. This is open frontier intelligence. This is open Frontier Intelligence is a catch Frontier Intelligence is a catch Frontier Intelligence is a catch catchphrase for the K3 model release catchphrase for the K3 model release catchphrase for the K3 model release this time. And it's also about VRM's this time. And it's also about VRM's this time. And it's also about VRM's mission about influencing mission about influencing mission about influencing infrastructure. So, if you look at it, infrastructure. So, if you look at it, infrastructure. So, if you look at it, when do they start being critical is when do they start being critical is when do they start being critical is when people are relying on it for their when people are relying on it for their when people are relying on it for their application, for their day-to-day application, for their day-to-day application, for their day-to-day productivity. So, maybe 2023 when GitHub productivity. So, maybe 2023 when GitHub productivity. So, maybe 2023 when GitHub Copilot and ChatGPT are where it is Copilot and ChatGPT are where it is Copilot and ChatGPT are where it is something that people just cannot live something that people just cannot live something that people just cannot live out with anymore. Uh so, and that point out with anymore. Uh so, and that point out with anymore. Uh so, and that point open weight model is already backing and open weight model is already backing and open weight model is already backing and become a cornerstone of the how people become a cornerstone of the how people become a cornerstone of the how people are living every day. are living every day. are living every day. >> Yeah. >> Yeah. >> Yeah. >> And at that point, we will need the >> And at that point, we will need the >> And at that point, we will need the accelerator running open weight model accelerator running open weight model accelerator running open weight model and open weight software to open source and open weight software to open source and open weight software to open source software to make sure it works well software to make sure it works well software to make sure it works well yeah, and have all flexibility and yeah, and have all flexibility and yeah, and have all flexibility and control over it. control over it. control over it. >> Yeah, it's sort of it's sort of >> Yeah, it's sort of it's sort of >> Yeah, it's sort of it's sort of interesting from a startup standpoint. interesting from a startup standpoint. interesting from a startup standpoint. Um Um Um uh you know, like like you mentioned uh you know, like like you mentioned uh you know, like like you mentioned sort of GPT-3 or like early ChatGPT, um sort of GPT-3 or like early ChatGPT, um sort of GPT-3 or like early ChatGPT, um those closed source solutions were those closed source solutions were those closed source solutions were starting to become critical to like a starting to become critical to like a starting to become critical to like a small group of people around that time. small group of people around that time. small group of people around that time. And open source existed, but it was a And open source existed, but it was a And open source existed, but it was a little bit of a curiosity or sort of an little bit of a curiosity or sort of an little bit of a curiosity or sort of an enthusiast thing. Um as the frontier has enthusiast thing. Um as the frontier has enthusiast thing. Um as the frontier has expanded, you know, particularly with expanded, you know, particularly with expanded, you know, particularly with closed source models, closed source models, closed source models, more and more open source has been more and more open source has been more and more open source has been dragged in as kind of like critical dragged in as kind of like critical dragged in as kind of like critical behind it like if that makes sense. Like behind it like if that makes sense. Like behind it like if that makes sense. Like it like at any given point in time it like at any given point in time it like at any given point in time including now, I think models from including now, I think models from including now, I think models from OpenAI and Anthropic are are kind of OpenAI and Anthropic are are kind of OpenAI and Anthropic are are kind of more more more widely used and more critical kind of in widely used and more critical kind of in widely used and more critical kind of in general than than open source models, general than than open source models, general than than open source models, but I do think we passed a threshold in but I do think we passed a threshold in but I do think we passed a threshold in like like like I want to say about a year ago I want to say about a year ago I want to say about a year ago >> Yeah. >> Yeah. >> Yeah. >> where a bunch of smaller companies or >> where a bunch of smaller companies or >> where a bunch of smaller companies or like new application companies as they like new application companies as they like new application companies as they were trying to figure out, how do I were trying to figure out, how do I were trying to figure out, how do I really build an AI really build an AI really build an AI without just being a wrapper on top of without just being a wrapper on top of without just being a wrapper on top of OpenAI, the answer to that question OpenAI, the answer to that question OpenAI, the answer to that question turned out to be open source. I mean, turned out to be open source. I mean, turned out to be open source. I mean, this is what Cursor did. This is what this is what Cursor did. This is what this is what Cursor did. This is what sort of Decagon and Harvey are are in sort of Decagon and Harvey are are in sort of Decagon and Harvey are are in the process of doing now and a and a the process of doing now and a and a the process of doing now and a and a bunch of other like really really strong bunch of other like really really strong bunch of other like really really strong application level startups sort of made application level startups sort of made application level startups sort of made the determination we can't build just on the determination we can't build just on the determination we can't build just on closed source. We need to do our own mid closed source. We need to do our own mid closed source. We need to do our own mid training, our own post training, our own training, our own post training, our own training, our own post training, our own sort of inference and deployment tricks sort of inference and deployment tricks sort of inference and deployment tricks and all of that means it must be built and all of that means it must be built and all of that means it must be built on top of open source like you know that on top of open source like you know that on top of open source like you know that the closed source vendors won't won't the closed source vendors won't won't the closed source vendors won't won't give you the access to do this. So give you the access to do this. So give you the access to do this. So my read is like kind of a year issue ago my read is like kind of a year issue ago my read is like kind of a year issue ago open source became really central in a open source became really central in a open source became really central in a way that's not always visible because way that's not always visible because way that's not always visible because it's it's deeply embedded in some of it's it's deeply embedded in some of it's it's deeply embedded in some of these products but you know some of the these products but you know some of the these products but you know some of the most innovative products and most innovative products and most innovative products and applications now you know really depend applications now you know really depend applications now you know really depend on this very deeply. on this very deeply. on this very deeply. >> Yeah. Yeah, and can you talk about where >> Yeah. Yeah, and can you talk about where >> Yeah. Yeah, and can you talk about where VL LM sits in that stack where where we VL LM sits in that stack where where we VL LM sits in that stack where where we do have these larger enterprise do have these larger enterprise do have these larger enterprise companies that are choosing to use open companies that are choosing to use open companies that are choosing to use open source models like where where does VL source models like where where does VL source models like where where does VL LM sit in the stack for them? LM sit in the stack for them? LM sit in the stack for them? >> Yeah, I mean you should >> Yeah, I mean you should >> Yeah, I mean you should just about everybody uses VL LM. You just about everybody uses VL LM. You just about everybody uses VL LM. You should describe it. should describe it. should describe it. >> Just about everybody uses VL LM. Is a >> Just about everybody uses VL LM. Is a >> Just about everybody uses VL LM. Is a inference engine. That means its job is inference engine. That means its job is inference engine. That means its job is to turn available GPUs into a running to turn available GPUs into a running to turn available GPUs into a running and point for intelligence. So that and point for intelligence. So that and point for intelligence. So that means it is kind of like databases and means it is kind of like databases and means it is kind of like databases and operating system and other critical operating system and other critical operating system and other critical software to power software to power software to power this economy or power the AGI that this economy or power the AGI that this economy or power the AGI that everybody really uses today to ensure everybody really uses today to ensure everybody really uses today to ensure they can have they can have they can have cost effectiveness, efficiency, cost effectiveness, efficiency, cost effectiveness, efficiency, reliability and also always stay on the reliability and also always stay on the reliability and also always stay on the frontier because for VL LM we support frontier because for VL LM we support frontier because for VL LM we support more than 1,000 model architecture more than 1,000 model architecture more than 1,000 model architecture active today and a lot of those are active today and a lot of those are active today and a lot of those are proprietary but also a lot of those are proprietary but also a lot of those are proprietary but also a lot of those are open weight, right? And a lot of those open weight, right? And a lot of those open weight, right? And a lot of those model architecture when they're becoming model architecture when they're becoming model architecture when they're becoming transitioning from a research prototype transitioning from a research prototype transitioning from a research prototype to world accessible open weight model to world accessible open weight model to world accessible open weight model architecture they are live on VL LM architecture they are live on VL LM architecture they are live on VL LM immediately. So there's what a process immediately. So there's what a process immediately. So there's what a process we call day zero model release. And we call day zero model release. And we call day zero model release. And additionally VL LM also work closely additionally VL LM also work closely additionally VL LM also work closely with all the hardware vendors so that with all the hardware vendors so that with all the hardware vendors so that means across like Nvidia, AMD, Google means across like Nvidia, AMD, Google means across like Nvidia, AMD, Google and Amazon, Intel and a lot more. Their and Amazon, Intel and a lot more. Their and Amazon, Intel and a lot more. Their newest chip will make sure VION can run newest chip will make sure VION can run newest chip will make sure VION can run on them. And in a lot of cases, they use on them. And in a lot of cases, they use on them. And in a lot of cases, they use VION as a benchmark to make sure it runs VION as a benchmark to make sure it runs VION as a benchmark to make sure it runs well on them. So, [snorts] this kind of well on them. So, [snorts] this kind of well on them. So, [snorts] this kind of fusion of where models run and where fusion of where models run and where fusion of where models run and where hardware where it can meet gets to meet hardware where it can meet gets to meet hardware where it can meet gets to meet the hardware is where the magic happen. the hardware is where the magic happen. the hardware is where the magic happen. And this where VION meets. And this where VION meets. And this where VION meets. >> And you've told me some of the >> And you've told me some of the >> And you've told me some of the behind-the-scenes stories. Like, it's behind-the-scenes stories. Like, it's behind-the-scenes stories. Like, it's actually not easy these days where a actually not easy these days where a actually not easy these days where a model releases. It's like a lot of human model releases. It's like a lot of human model releases. It's like a lot of human drama in addition to like technical drama in addition to like technical drama in addition to like technical work. I guess are there any stories work. I guess are there any stories work. I guess are there any stories there that you think are okay to share? there that you think are okay to share? there that you think are okay to share? >> Oh, it's actually a very fun co-design >> Oh, it's actually a very fun co-design >> Oh, it's actually a very fun co-design process because from model labs point of process because from model labs point of process because from model labs point of view, right? These are brilliant view, right? These are brilliant view, right? These are brilliant researchers who have built this model. researchers who have built this model. researchers who have built this model. Now, their biggest question becomes, how Now, their biggest question becomes, how Now, their biggest question becomes, how do we get this out of the world and make do we get this out of the world and make do we get this out of the world and make sure everybody's able to use it and run sure everybody's able to use it and run sure everybody's able to use it and run it well? And we have worked with model it well? And we have worked with model it well? And we have worked with model labs that are very just because they labs that are very just because they labs that are very just because they just use VION just use VION just use VION already in production or in their already in production or in their already in production or in their research process, they will just dump research process, they will just dump research process, they will just dump everything for you. Cuz this is a moment everything for you. Cuz this is a moment everything for you. Cuz this is a moment when we go to them and say, like, "Hi, when we go to them and say, like, "Hi, when we go to them and say, like, "Hi, we're the VION team we would like to we're the VION team we would like to we're the VION team we would like to have support your open-source model. We have support your open-source model. We have support your open-source model. We would like to offer in a way that's kind would like to offer in a way that's kind would like to offer in a way that's kind of open-source but wide loss service to of open-source but wide loss service to of open-source but wide loss service to get your model running well on get your model running well on get your model running well on architecture." And then you return and architecture." And then you return and architecture." And then you return and look at the model labs like, "Oh, we got look at the model labs like, "Oh, we got look at the model labs like, "Oh, we got it working already because we're running it working already because we're running it working already because we're running it for the RL process. Here you go, just it for the RL process. Here you go, just it for the RL process. Here you go, just review our code and merge our pull review our code and merge our pull review our code and merge our pull request." And then on the other end, we request." And then on the other end, we request." And then on the other end, we really have model labs that just don't really have model labs that just don't really have model labs that just don't know how this can work. So, know how this can work. So, know how this can work. So, >> Because systems is not like their core. >> Because systems is not like their core. >> Because systems is not like their core. >> Yeah, cuz systems is not their core and >> Yeah, cuz systems is not their core and >> Yeah, cuz systems is not their core and they have been training or maybe they they have been training or maybe they they have been training or maybe they have their internal inference engine have their internal inference engine have their internal inference engine that just don't know how it will adapt that just don't know how it will adapt that just don't know how it will adapt to the open way ecosystem. And by the to the open way ecosystem. And by the to the open way ecosystem. And by the way, this is also a very much a way, this is also a very much a way, this is also a very much a multi-party kind of involvement process. multi-party kind of involvement process. multi-party kind of involvement process. Every model release typically involves Every model release typically involves Every model release typically involves the the model lab, involves a primary or the the model lab, involves a primary or the the model lab, involves a primary or secondary hardware vendors, involves us, secondary hardware vendors, involves us, secondary hardware vendors, involves us, involves Hugging Face who are the model involves Hugging Face who are the model involves Hugging Face who are the model format and like model hub vendors. And format and like model hub vendors. And format and like model hub vendors. And then, depending on the appetite of the then, depending on the appetite of the then, depending on the appetite of the model lab involves 10 or 20 different model lab involves 10 or 20 different model lab involves 10 or 20 different kinds of release partners. These could kinds of release partners. These could kinds of release partners. These could be inference clouds, these could be be inference clouds, these could be be inference clouds, these could be public hyperscalers, whoever is going to public hyperscalers, whoever is going to public hyperscalers, whoever is going to run this model, and you want to them to run this model, and you want to them to run this model, and you want to them to ensure success of the model are running ensure success of the model are running ensure success of the model are running successfully. Right? So, successfully. Right? So, successfully. Right? So, even up to today, if you look at the even up to today, if you look at the even up to today, if you look at the case three model release, it's a whole case three model release, it's a whole case three model release, it's a whole partnership and a drive to make sure partnership and a drive to make sure partnership and a drive to make sure that the model is Once the model is that the model is Once the model is that the model is Once the model is released, because it's just a few released, because it's just a few released, because it's just a few terabytes of files sitting on the terabytes of files sitting on the terabytes of files sitting on the internet, that people are actually going internet, that people are actually going internet, that people are actually going to be using it really, really well. Even to be using it really, really well. Even to be using it really, really well. Even from the beginning of 2023, 2024, if you from the beginning of 2023, 2024, if you from the beginning of 2023, 2024, if you remember when Mistral dropped their remember when Mistral dropped their remember when Mistral dropped their first model, they just dropped a torrent first model, they just dropped a torrent first model, they just dropped a torrent link for PGT, and then everybody's like link for PGT, and then everybody's like link for PGT, and then everybody's like struggling and trying to get it up and struggling and trying to get it up and struggling and trying to get it up and running. And then, we're working behind running. And then, we're working behind running. And then, we're working behind the scene with Mistral team trying to the scene with Mistral team trying to the scene with Mistral team trying to get the inference engines to work get the inference engines to work get the inference engines to work working in VR. This was the most working in VR. This was the most working in VR. This was the most probably early on exciting weekend that probably early on exciting weekend that probably early on exciting weekend that we are able to spend on this. And then, we are able to spend on this. And then, we are able to spend on this. And then, and after the weekend, when everybody's and after the weekend, when everybody's and after the weekend, when everybody's trying to really analyze what's going trying to really analyze what's going trying to really analyze what's going on, and Monday, Tuesday, well, Mistral on, and Monday, Tuesday, well, Mistral on, and Monday, Tuesday, well, Mistral and us just announced, "Here, you can and us just announced, "Here, you can and us just announced, "Here, you can run it on VR and successfully here." And run it on VR and successfully here." And run it on VR and successfully here." And everybody will able to easily reuse a everybody will able to easily reuse a everybody will able to easily reuse a lot of work and be start building on top lot of work and be start building on top lot of work and be start building on top of it. of it. of it. >> That was sort of a fun time where like >> That was sort of a fun time where like >> That was sort of a fun time where like enthusiasts like me could just like enthusiasts like me could just like enthusiasts like me could just like scramble to like download the model and scramble to like download the model and scramble to like download the model and like get it running somewhere. like get it running somewhere. like get it running somewhere. >> Yeah, yeah. >> Yeah, yeah. >> Yeah, yeah. >> I'm glad the profession is taking >> I'm glad the profession is taking >> I'm glad the profession is taking [laughter] over, because it never worked [laughter] over, because it never worked [laughter] over, because it never worked like very well. But, it was like very well. But, it was like very well. But, it was It was like a fun moment in time. It was like a fun moment in time. It was like a fun moment in time. >> Yeah. >> Yeah. >> Yeah. >> So, so to bring things, you know, >> So, so to bring things, you know, >> So, so to bring things, you know, forward to the present, I think forward to the present, I think forward to the present, I think open-source models and also, you know, open-source models and also, you know, open-source models and also, you know, distillation have have been in the news distillation have have been in the news distillation have have been in the news recently. Infra Act signed the NVIDIA recently. Infra Act signed the NVIDIA recently. Infra Act signed the NVIDIA Open Weights and American AI Leadership Open Weights and American AI Leadership Open Weights and American AI Leadership Letter that that was signed by also Letter that that was signed by also Letter that that was signed by also A16Z, Meta, Amazon, dozens of other A16Z, Meta, Amazon, dozens of other A16Z, Meta, Amazon, dozens of other companies. Can you just talk about companies. Can you just talk about companies. Can you just talk about you know, your decision to sign that and you know, your decision to sign that and you know, your decision to sign that and and sort of what what you were really and sort of what what you were really and sort of what what you were really kind of responding to kind of responding to kind of responding to in the market and and kind of in the in the market and and kind of in the in the market and and kind of in the news. news. news. >> Yeah, so for us, what we really want to >> Yeah, so for us, what we really want to >> Yeah, so for us, what we really want to stand behind is open way absolutely stand behind is open way absolutely stand behind is open way absolutely matters in the ecosystem. The world matters in the ecosystem. The world matters in the ecosystem. The world cannot just be controlled by proprietary cannot just be controlled by proprietary cannot just be controlled by proprietary APIs and where open way open development APIs and where open way open development APIs and where open way open development and research of these models are blocked and research of these models are blocked and research of these models are blocked or banned, right? Uh the pledge that or banned, right? Uh the pledge that or banned, right? Uh the pledge that Infra agrees enough for is we want to Infra agrees enough for is we want to Infra agrees enough for is we want to help and foster this ecosystem where we help and foster this ecosystem where we help and foster this ecosystem where we are typically in little bit downstream are typically in little bit downstream are typically in little bit downstream of this ecosystem, right? Inference of this ecosystem, right? Inference of this ecosystem, right? Inference engine are not part of the pre-training engine are not part of the pre-training engine are not part of the pre-training process, nor the RL process, but where process, nor the RL process, but where process, nor the RL process, but where where the model actually meets the where the model actually meets the where the model actually meets the world. And from what we're seeing, world. And from what we're seeing, world. And from what we're seeing, people are just really using their people are just really using their people are just really using their imagination and ability to materialize imagination and ability to materialize imagination and ability to materialize this imagination of open way model. this imagination of open way model. this imagination of open way model. They're able to leverage this open way They're able to leverage this open way They're able to leverage this open way model so much effectively. model so much effectively. model so much effectively. >> There's almost two pieces to this, >> There's almost two pieces to this, >> There's almost two pieces to this, right? There's like the the cost thing right? There's like the the cost thing right? There's like the the cost thing where it's like the closed models are where it's like the closed models are where it's like the closed models are too expensive, and then there's sort of too expensive, and then there's sort of too expensive, and then there's sort of the control thing where I I I want to the control thing where I I I want to the control thing where I I I want to sort of be in control of my sort of be in control of my sort of be in control of my infrastructure and and and in control of infrastructure and and and in control of infrastructure and and and in control of the model, right? If I need to extend it the model, right? If I need to extend it the model, right? If I need to extend it or or put on my own guardrails or or or put on my own guardrails or or or put on my own guardrails or anything. I I'm just curious, have you anything. I I'm just curious, have you anything. I I'm just curious, have you heard from heard from heard from from customers like are are both those from customers like are are both those from customers like are are both those things important to them or or like are things important to them or or like are things important to them or or like are they kind of willing to pay as long as they kind of willing to pay as long as they kind of willing to pay as long as they have the control or or maybe they have the control or or maybe they have the control or or maybe they're different use cases? they're different use cases? they're different use cases? >> I think like it fluctuate over time. So, >> I think like it fluctuate over time. So, >> I think like it fluctuate over time. So, control matters a lot over the last few control matters a lot over the last few control matters a lot over the last few years, and then cost just start to years, and then cost just start to years, and then cost just start to matter over the last few months. So, matter over the last few months. So, matter over the last few months. So, cost really matters starting from people cost really matters starting from people cost really matters starting from people trying to migrate off, trying to migrate off, trying to migrate off, right? Their expensive coding plan and right? Their expensive coding plan and right? Their expensive coding plan and like every skyrocketing token maxing like every skyrocketing token maxing like every skyrocketing token maxing spend, but control has always been on in spend, but control has always been on in spend, but control has always been on in the background of this is they want to the background of this is they want to the background of this is they want to even you know, wait to control the cost, even you know, wait to control the cost, even you know, wait to control the cost, right? But also is about controlling the right? But also is about controlling the right? But also is about controlling the system performance against what they're system performance against what they're system performance against what they're paying for. So, there are for example paying for. So, there are for example paying for. So, there are for example for a voice agent company, they want to for a voice agent company, they want to for a voice agent company, they want to control their own model so that they can control their own model so that they can control their own model so that they can make sure the model actually respond by make sure the model actually respond by make sure the model actually respond by their required time. So, the customer their required time. So, the customer their required time. So, the customer when they're on the phone, they can when they're on the phone, they can when they're on the phone, they can ensure the agent is responding according ensure the agent is responding according ensure the agent is responding according to a SLA. And this sometimes is only you to a SLA. And this sometimes is only you to a SLA. And this sometimes is only you can do with your control intelligence can do with your control intelligence can do with your control intelligence because you know the whole hardware because you know the whole hardware because you know the whole hardware you're running and the whole system you're running and the whole system you're running and the whole system you're monitoring. Versus signing up for you're monitoring. Versus signing up for you're monitoring. Versus signing up for relying on your critical infrastructure relying on your critical infrastructure relying on your critical infrastructure with a proprietary API where they might with a proprietary API where they might with a proprietary API where they might go down anytime or have violation of the go down anytime or have violation of the go down anytime or have violation of the contract anytime. contract anytime. contract anytime. >> Yeah. Simon, you also >> Yeah. Simon, you also >> Yeah. Simon, you also to to go back to the cost point, you to to go back to the cost point, you to to go back to the cost point, you actually make the point in an essay you actually make the point in an essay you actually make the point in an essay you recently wrote about the release of recently wrote about the release of recently wrote about the release of Gemini K3 that actually the economics is Gemini K3 that actually the economics is Gemini K3 that actually the economics is besides the point and it's actually you besides the point and it's actually you besides the point and it's actually you know in in the case of these you know know in in the case of these you know know in in the case of these you know just really really great open weight just really really great open weight just really really great open weight models that are on the frontier that models that are on the frontier that models that are on the frontier that were designed by really brilliant were designed by really brilliant were designed by really brilliant researchers like like researchers like like researchers like like these models are in some cases just as these models are in some cases just as these models are in some cases just as expensive as you know the closed source expensive as you know the closed source expensive as you know the closed source models. So, in those cases kind of what models. So, in those cases kind of what models. So, in those cases kind of what is the point of running them and kind of is the point of running them and kind of is the point of running them and kind of what do we learn architecturally in the what do we learn architecturally in the what do we learn architecturally in the in the course of running them? in the course of running them? in the course of running them? >> Yeah, so first on cost, it's not >> Yeah, so first on cost, it's not >> Yeah, so first on cost, it's not necessarily they are as expensive as a necessarily they are as expensive as a necessarily they are as expensive as a proprietary model but rather first the proprietary model but rather first the proprietary model but rather first the cost cost discourse has been discussed cost cost discourse has been discussed cost cost discourse has been discussed over and over again with even Gemini 1.5 over and over again with even Gemini 1.5 over and over again with even Gemini 1.5 a few months back. a few months back. a few months back. So, open weight model are sometimes So, open weight model are sometimes So, open weight model are sometimes definitely a lot cheaper. And but for definitely a lot cheaper. And but for definitely a lot cheaper. And but for this model, there's a this model, there's a this model, there's a big sort of step change where we're big sort of step change where we're big sort of step change where we're bridging almost a 10x gap but strike bridging almost a 10x gap but strike bridging almost a 10x gap but strike somewhere in the middle. Where Gemini K3 somewhere in the middle. Where Gemini K3 somewhere in the middle. Where Gemini K3 is not as expensive as Claude or GPT is not as expensive as Claude or GPT is not as expensive as Claude or GPT sole but it is a lot more expensive than sole but it is a lot more expensive than sole but it is a lot more expensive than Geo and 5.2. Why is that? And I do Geo and 5.2. Why is that? And I do Geo and 5.2. Why is that? And I do believe this is the point of where believe this is the point of where believe this is the point of where pricing intelligence with the market pricing intelligence with the market pricing intelligence with the market correctly and understanding where it is correctly and understanding where it is correctly and understanding where it is and and and but then the majority part of the but then the majority part of the but then the majority part of the discourse should be focusing on wow, discourse should be focusing on wow, discourse should be focusing on wow, this model is bringing a Opus 4.8 level this model is bringing a Opus 4.8 level this model is bringing a Opus 4.8 level model to our own infrastructure that I model to our own infrastructure that I model to our own infrastructure that I can use, I can run, I can fine-tune, I can use, I can run, I can fine-tune, I can use, I can run, I can fine-tune, I can be able to can be able to can be able to understand exactly how many tokens do I understand exactly how many tokens do I understand exactly how many tokens do I need, understand the exact performance need, understand the exact performance need, understand the exact performance profile. The reason here, for example, profile. The reason here, for example, profile. The reason here, for example, is for proprietary model, there is a is for proprietary model, there is a is for proprietary model, there is a regular mode and fast mode. And that's regular mode and fast mode. And that's regular mode and fast mode. And that's only the two switch here. But for open only the two switch here. But for open only the two switch here. But for open weight, when you are running it, every weight, when you are running it, every weight, when you are running it, every provider can offer potentially even 10 provider can offer potentially even 10 provider can offer potentially even 10 different levels of speed going from different levels of speed going from different levels of speed going from like the slowest mode, which can be a like the slowest mode, which can be a like the slowest mode, which can be a lot cheaper, to um 400 tokens per second lot cheaper, to um 400 tokens per second lot cheaper, to um 400 tokens per second uh almost up to 500 in many cases that uh almost up to 500 in many cases that uh almost up to 500 in many cases that for some workloads. And this is uh for some workloads. And this is uh for some workloads. And this is uh typically 2x or 3x faster than the fast typically 2x or 3x faster than the fast typically 2x or 3x faster than the fast mode out there today. So, this kind of mode out there today. So, this kind of mode out there today. So, this kind of level control even in terms of level control even in terms of level control even in terms of performance, and then let alone control performance, and then let alone control performance, and then let alone control over how customer interacting with the over how customer interacting with the over how customer interacting with the model, control over data retention. Keep model, control over data retention. Keep model, control over data retention. Keep in mind, Fable doesn't have the zero in mind, Fable doesn't have the zero in mind, Fable doesn't have the zero data data retention policy and at least data data retention policy and at least data data retention policy and at least the a lot of data need to be staying the a lot of data need to be staying the a lot of data need to be staying there. And let it control security and there. And let it control security and there. And let it control security and compliance and lot more. Um yeah, this compliance and lot more. Um yeah, this compliance and lot more. Um yeah, this is why I'm particularly excited about is why I'm particularly excited about is why I'm particularly excited about case three not just from the cost case three not just from the cost case three not just from the cost perspective, but a lot more on bringing perspective, but a lot more on bringing perspective, but a lot more on bringing this level intelligence to something this level intelligence to something this level intelligence to something people can own. people can own. people can own. >> In terms of, you know, calibrating >> In terms of, you know, calibrating >> In terms of, you know, calibrating things like speed, calibrating other things like speed, calibrating other things like speed, calibrating other things just sort of on the back end, things just sort of on the back end, things just sort of on the back end, what needs to happen and and kind of what needs to happen and and kind of what needs to happen and and kind of what are you seeing your users do and what are you seeing your users do and what are you seeing your users do and like like who who is being really clever like like who who is being really clever like like who who is being really clever about this? about this? about this? >> So, we do see users are able to get the >> So, we do see users are able to get the >> So, we do see users are able to get the maximum benefit out of this model when maximum benefit out of this model when maximum benefit out of this model when they enable fast mode. I like what I'm they enable fast mode. I like what I'm they enable fast mode. I like what I'm talking about here, of course, is Vion's talking about here, of course, is Vion's talking about here, of course, is Vion's own fast mode getting up to 400 and 500 own fast mode getting up to 400 and 500 own fast mode getting up to 400 and 500 tokens per second because it is really a tokens per second because it is really a tokens per second because it is really a big step change from like that big step change from like that big step change from like that especially when developer interacting especially when developer interacting especially when developer interacting with the model, they can see, "Oh, I can with the model, they can see, "Oh, I can with the model, they can see, "Oh, I can really just get my task done faster really just get my task done faster really just get my task done faster here." And the model are not no longer here." And the model are not no longer here." And the model are not no longer stuck in sinking, rather it is just stuck in sinking, rather it is just stuck in sinking, rather it is just executing, executing, and interacting executing, executing, and interacting executing, executing, and interacting with the environment. So, for premium with the environment. So, for premium with the environment. So, for premium developer block and focus with task, developer block and focus with task, developer block and focus with task, we're seeing is very benefiting. But we're seeing is very benefiting. But we're seeing is very benefiting. But also case three are just be able to also case three are just be able to also case three are just be able to have the ability for people to modify it have the ability for people to modify it have the ability for people to modify it and fine-tune on top of it allow them to and fine-tune on top of it allow them to and fine-tune on top of it allow them to make it better for their own workload. make it better for their own workload. make it better for their own workload. And this is definitely happening today And this is definitely happening today And this is definitely happening today as well. as well. as well. >> Can you just explain what the um >> Can you just explain what the um >> Can you just explain what the um licensing term is for the most recent licensing term is for the most recent licensing term is for the most recent open source models compared to the past open source models compared to the past open source models compared to the past and and and like why you think they're and and and like why you think they're and and and like why you think they're doing that? doing that? doing that? >> Oh, yeah. So, historically the um the >> Oh, yeah. So, historically the um the >> Oh, yeah. So, historically the um the open way model are just like Apache 2 open way model are just like Apache 2 open way model are just like Apache 2 like our software, which is like take like our software, which is like take like our software, which is like take it, modify it, do it whatever you want it, modify it, do it whatever you want it, modify it, do it whatever you want with it. Here is a gift to the world. with it. Here is a gift to the world. with it. Here is a gift to the world. And then recently the model lab are And then recently the model lab are And then recently the model lab are trying to understand a way to trying to understand a way to trying to understand a way to economically fund their own model economically fund their own model economically fund their own model development after all model training and development after all model training and development after all model training and researcher. And the data are very very researcher. And the data are very very researcher. And the data are very very expensive. So, we have been starting to expensive. So, we have been starting to expensive. So, we have been starting to see uh terms even to the Llama days for see uh terms even to the Llama days for see uh terms even to the Llama days for for when Meta it was releasing Llama, for when Meta it was releasing Llama, for when Meta it was releasing Llama, they do have a term of if your daily they do have a term of if your daily they do have a term of if your daily active user or like annual recurring active user or like annual recurring active user or like annual recurring revenue exceeds some threshold, please revenue exceeds some threshold, please revenue exceeds some threshold, please enter into a commercial agreement with enter into a commercial agreement with enter into a commercial agreement with Meta specifically, right? Meta specifically, right? Meta specifically, right? >> remember that the numbers were like >> remember that the numbers were like >> remember that the numbers were like specifically chosen at that time that specifically chosen at that time that specifically chosen at that time that you could go find it was like two you could go find it was like two you could go find it was like two companies in the world that like fit the companies in the world that like fit the companies in the world that like fit the definition that they had excluded from definition that they had excluded from definition that they had excluded from their license. their license. their license. >> Yeah, exactly. But like people have >> Yeah, exactly. But like people have >> Yeah, exactly. But like people have taken a hint from that, especially now taken a hint from that, especially now taken a hint from that, especially now the labs are trying to figure out a way the labs are trying to figure out a way the labs are trying to figure out a way to economically fund it, especially when to economically fund it, especially when to economically fund it, especially when they open source the model, everybody they open source the model, everybody they open source the model, everybody can just take it of themselves, whereas can just take it of themselves, whereas can just take it of themselves, whereas nobody will use your API anymore in many nobody will use your API anymore in many nobody will use your API anymore in many cases, while their API currently still cases, while their API currently still cases, while their API currently still taking up shape, right? And now we're taking up shape, right? And now we're taking up shape, right? And now we're seeing a very healthy ecosystem seeing a very healthy ecosystem seeing a very healthy ecosystem development starting from even MiniMax development starting from even MiniMax development starting from even MiniMax recently had when they're releasing recently had when they're releasing recently had when they're releasing their M2.7 model, they have a term their M2.7 model, they have a term their M2.7 model, they have a term specifically focusing on usage. And Kimi specifically focusing on usage. And Kimi specifically focusing on usage. And Kimi initially also has like if you have initially also has like if you have initially also has like if you have derivative derivative works, like this derivative derivative works, like this derivative derivative works, like this is kind of big news back then with is kind of big news back then with is kind of big news back then with fireworks and curvature about how they fireworks and curvature about how they fireworks and curvature about how they build on top of Kimi model. Yeah. build on top of Kimi model. Yeah. build on top of Kimi model. Yeah. >> And and it's >> And and it's >> And and it's if I if I if I like I just expand on that a little like like I just expand on that a little like like I just expand on that a little like I don't think it's greed, at least what I don't think it's greed, at least what I don't think it's greed, at least what I've seen from open source model labs, I've seen from open source model labs, I've seen from open source model labs, right? right? right? Open source models, really what we're Open source models, really what we're Open source models, really what we're talking about are open weights, right? talking about are open weights, right? talking about are open weights, right? And and it's just not software, right? And and it's just not software, right? And and it's just not software, right? Like a an AI model is not software at Like a an AI model is not software at Like a an AI model is not software at the end of the day. the end of the day. the end of the day. Um Um Um And so open source software used to be And so open source software used to be And so open source software used to be supported by people donating their time supported by people donating their time supported by people donating their time or big companies kind of authorizing or big companies kind of authorizing or big companies kind of authorizing their employees to donate their time. So their employees to donate their time. So their employees to donate their time. So it was sort of like a bulk in kind, you it was sort of like a bulk in kind, you it was sort of like a bulk in kind, you know, donation of people's time. know, donation of people's time. know, donation of people's time. It really doesn't work in AI, right? It really doesn't work in AI, right? It really doesn't work in AI, right? Like like I can't just like go home at Like like I can't just like go home at Like like I can't just like go home at night and like train a frontier open night and like train a frontier open night and like train a frontier open source model with friends for fun. Like source model with friends for fun. Like source model with friends for fun. Like we need millions or billions of dollars we need millions or billions of dollars we need millions or billions of dollars of computing resources of computing resources of computing resources in order to do it. So So I think it does in order to do it. So So I think it does in order to do it. So So I think it does support your point that like obviously support your point that like obviously support your point that like obviously there need to be economic incentives and there need to be economic incentives and there need to be economic incentives and there need to be funding mechanisms in there need to be funding mechanisms in there need to be funding mechanisms in place. Frankly, I think even more so place. Frankly, I think even more so place. Frankly, I think even more so with with Chinese with with Chinese with with Chinese models than than with domestically models than than with domestically models than than with domestically produced models, right? If if if there's produced models, right? If if if there's produced models, right? If if if there's no source of economic if there's no no source of economic if there's no no source of economic if there's no source of funding for for Moonshot to source of funding for for Moonshot to source of funding for for Moonshot to continue to train models, like we know continue to train models, like we know continue to train models, like we know where the funding will come from where the funding will come from where the funding will come from instead, and it's not like something we instead, and it's not like something we instead, and it's not like something we right, you know, it's it's government right, you know, it's it's government right, you know, it's it's government and and things that like are actually and and things that like are actually and and things that like are actually worse for us, I think. So like I I I worse for us, I think. So like I I I worse for us, I think. So like I I I think I think you raise sort of an think I think you raise sort of an think I think you raise sort of an interesting point that that this is an interesting point that that this is an interesting point that that this is an important economic structure and like I important economic structure and like I important economic structure and like I I I think this means we'll see more of I I think this means we'll see more of I I think this means we'll see more of this in the future. Would you agree with this in the future. Would you agree with this in the future. Would you agree with that? that? that? >> Yeah, it's really about sustainability >> Yeah, it's really about sustainability >> Yeah, it's really about sustainability and then is about how do you make sure and then is about how do you make sure and then is about how do you make sure that all this initial capex almost to that all this initial capex almost to that all this initial capex almost to train the model, fail again and again train the model, fail again and again train the model, fail again and again and train the model again. Like how do and train the model again. Like how do and train the model again. Like how do you really pay it back and how do you you really pay it back and how do you you really pay it back and how do you make sure that there's enough confidence make sure that there's enough confidence make sure that there's enough confidence and funding and funding and funding from everybody involved to go to do the from everybody involved to go to do the from everybody involved to go to do the next one, right? And I [clears throat] next one, right? And I [clears throat] next one, right? And I [clears throat] recently heard someone recently heard someone recently heard someone I recently heard someone making analogy I recently heard someone making analogy I recently heard someone making analogy to this to the pharmaceutical industry. to this to the pharmaceutical industry. to this to the pharmaceutical industry. It's almost like how do you make sure It's almost like how do you make sure It's almost like how do you make sure that the R&D process of new drugs are that the R&D process of new drugs are that the R&D process of new drugs are properly funded and there's proper properly funded and there's proper properly funded and there's proper sustainable sustainable sustainable method to making sure that people are method to making sure that people are method to making sure that people are willing to take big risk, big bet to go willing to take big risk, big bet to go willing to take big risk, big bet to go to do research for new drugs and then to do research for new drugs and then to do research for new drugs and then later because they know there's a later because they know there's a later because they know there's a economic incentive in the end when the economic incentive in the end when the economic incentive in the end when the new drug release to the market, a new drug release to the market, a new drug release to the market, a portion of those of course like besides portion of those of course like besides portion of those of course like besides the just distribution channels, right? A the just distribution channels, right? A the just distribution channels, right? A portion of those revenue will flow back portion of those revenue will flow back portion of those revenue will flow back to continue to fund the next R&D effort. to continue to fund the next R&D effort. to continue to fund the next R&D effort. And this is what we're kind of seeing And this is what we're kind of seeing And this is what we're kind of seeing similar to the model development now, similar to the model development now, similar to the model development now, yeah. yeah. yeah. >> That's a really interesting analogy cuz >> That's a really interesting analogy cuz >> That's a really interesting analogy cuz it's like it's like it's like what what once a once a drug, a molecule what what once a once a drug, a molecule what what once a once a drug, a molecule is released, you have the strongest is released, you have the strongest is released, you have the strongest possible control which is nobody else possible control which is nobody else possible control which is nobody else can manufacture it at all, right? It's can manufacture it at all, right? It's can manufacture it at all, right? It's like the close most closed possible like the close most closed possible like the close most closed possible source, right? Like it's like a secret. source, right? Like it's like a secret. source, right? Like it's like a secret. Um Um Um Uh but but yeah, like in the case of Uh but but yeah, like in the case of Uh but but yeah, like in the case of models, especially open source models, models, especially open source models, models, especially open source models, you know, once it's out there anybody you know, once it's out there anybody you know, once it's out there anybody can take it, use it, extend it, etc. So can take it, use it, extend it, etc. So can take it, use it, extend it, etc. So So having yeah, so having some economics So having yeah, so having some economics So having yeah, so having some economics attached to it probably does make sense. attached to it probably does make sense. attached to it probably does make sense. >> Yeah. >> Yeah. >> Yeah. >> Well, actually that that raises a >> Well, actually that that raises a >> Well, actually that that raises a question for me too which Matt, you were question for me too which Matt, you were question for me too which Matt, you were alluding to this earlier about how alluding to this earlier about how alluding to this earlier about how different open source models are from different open source models are from different open source models are from the dynamics of open source software the dynamics of open source software the dynamics of open source software maintenance. When it comes to open maintenance. When it comes to open maintenance. When it comes to open source AI, what actually needs to be source AI, what actually needs to be source AI, what actually needs to be maintained? Is it the infrastructure maintained? Is it the infrastructure maintained? Is it the infrastructure around it? Do the models themselves need around it? Do the models themselves need around it? Do the models themselves need maintenance at all? Just kind of what maintenance at all? Just kind of what maintenance at all? Just kind of what what are those dynamics because I think what are those dynamics because I think what are those dynamics because I think even the the developer behavior around even the the developer behavior around even the the developer behavior around it is is pretty different. it is is pretty different. it is is pretty different. >> Yeah, I mean Simon sort of said this >> Yeah, I mean Simon sort of said this >> Yeah, I mean Simon sort of said this already, but maybe maybe I'll just already, but maybe maybe I'll just already, but maybe maybe I'll just expand it a bit which is um expand it a bit which is um expand it a bit which is um you know, you you see the results of of you know, you you see the results of of you know, you you see the results of of a big training run, you know, we're a big training run, you know, we're a big training run, you know, we're training now mean it's pre-training and training now mean it's pre-training and training now mean it's pre-training and then and then, you know, RL kind of um then and then, you know, RL kind of um then and then, you know, RL kind of um I know mid-training or sort of I know mid-training or sort of I know mid-training or sort of post-training on these thing or you post-training on these thing or you post-training on these thing or you know, know, know, >> pre-training SFT or RL right like it's a >> pre-training SFT or RL right like it's a >> pre-training SFT or RL right like it's a very complicated pipeline. We see only very complicated pipeline. We see only very complicated pipeline. We see only the result the result the result >> Yeah. >> Yeah. >> Yeah. >> of this at the end and the numbers are >> of this at the end and the numbers are >> of this at the end and the numbers are big like, oh, you know, this was a big like, oh, you know, this was a big like, oh, you know, this was a hundred million dollar training run, but hundred million dollar training run, but hundred million dollar training run, but what you often forget is like there may what you often forget is like there may what you often forget is like there may have been five failed training run you have been five failed training run you have been five failed training run you know, large-scale failed training runs know, large-scale failed training runs know, large-scale failed training runs before you even get to that point. before you even get to that point. before you even get to that point. >> a lot of sweat and tears >> a lot of sweat and tears >> a lot of sweat and tears >> Yeah, yeah. >> Yeah, yeah. >> Yeah, yeah. >> Yeah, exactly. What one of my favorite >> Yeah, exactly. What one of my favorite >> Yeah, exactly. What one of my favorite artifacts, maybe we could even track artifacts, maybe we could even track artifacts, maybe we could even track down the link is one of the early Llama down the link is one of the early Llama down the link is one of the early Llama models. models. models. >> Yeah. >> Yeah. >> Yeah. >> They published the whole like >> They published the whole like >> They published the whole like conversation log between the or between conversation log between the or between conversation log between the or between the people who are baby sitting the the people who are baby sitting the the people who are baby sitting the training clusters like while the models training clusters like while the models training clusters like while the models were training. were training. were training. >> Yeah. >> Yeah. >> Yeah. >> And it's it's so funny. It's just like, >> And it's it's so funny. It's just like, >> And it's it's so funny. It's just like, oh no, everything's gone wrong like oh no, everything's gone wrong like oh no, everything's gone wrong like chaos like panic. And then like then the chaos like panic. And then like then the chaos like panic. And then like then the next comment is like, okay, we solved next comment is like, okay, we solved next comment is like, okay, we solved it. Everything's okay. You know, it. Everything's okay. You know, it. Everything's okay. You know, clusters up, losses going down. So clusters up, losses going down. So clusters up, losses going down. So there's there's a lot a lot a lot that there's there's a lot a lot a lot that there's there's a lot a lot a lot that goes in behind the scenes before these goes in behind the scenes before these goes in behind the scenes before these models get released. You know, once once models get released. You know, once once models get released. You know, once once they're out there, I guess it's a little they're out there, I guess it's a little they're out there, I guess it's a little bit more in your in your zone, you know, bit more in your in your zone, you know, bit more in your in your zone, you know, to to kind of make sure it's like to to kind of make sure it's like to to kind of make sure it's like operationalized. operationalized. operationalized. >> Oh, yeah. But but this is also a very >> Oh, yeah. But but this is also a very >> Oh, yeah. But but this is also a very interesting point once it's out there is interesting point once it's out there is interesting point once it's out there is a whole community effort trying to a whole community effort trying to a whole community effort trying to optimize in this model because the model optimize in this model because the model optimize in this model because the model is trained on a given type of hardware is trained on a given type of hardware is trained on a given type of hardware on given type of architecture and but on given type of architecture and but on given type of architecture and but when it's out in the wild, everybody has when it's out in the wild, everybody has when it's out in the wild, everybody has different cluster topology and use cases different cluster topology and use cases different cluster topology and use cases and it's about how do you turn like a and it's about how do you turn like a and it's about how do you turn like a use case of one now to a use case of use case of one now to a use case of use case of one now to a use case of almost infinity. Now you have people almost infinity. Now you have people almost infinity. Now you have people trying to adapt it to the edge devices trying to adapt it to the edge devices trying to adapt it to the edge devices and people trying to run it at largest and people trying to run it at largest and people trying to run it at largest scale ever. scale ever. scale ever. Adapt it and making sure it runs for Adapt it and making sure it runs for Adapt it and making sure it runs for voice agent but also for coding agent voice agent but also for coding agent voice agent but also for coding agent which are entirely different kind of use which are entirely different kind of use which are entirely different kind of use cases. So, this is a whole community cases. So, this is a whole community cases. So, this is a whole community effort trying to further optimize, effort trying to further optimize, effort trying to further optimize, specialize, and making sure the running specialize, and making sure the running specialize, and making sure the running of it is reliable and continue to be of it is reliable and continue to be of it is reliable and continue to be able to optimize against it. So, that's able to optimize against it. So, that's able to optimize against it. So, that's a whole village later a whole village later a whole village later throughout the open source to make sure throughout the open source to make sure throughout the open source to make sure it's improved. it's improved. it's improved. >> Yeah, and that's cool cuz that really is >> Yeah, and that's cool cuz that really is >> Yeah, and that's cool cuz that really is like open source software. I mean, and like open source software. I mean, and like open source software. I mean, and this is what you do, but you know, like this is what you do, but you know, like this is what you do, but you know, like this is like anybody can contribute and this is like anybody can contribute and this is like anybody can contribute and and like make these better. Um and like make these better. Um and like make these better. Um My the fun thought experiment is if My the fun thought experiment is if My the fun thought experiment is if GPUs, you know, dropped in price by 99% GPUs, you know, dropped in price by 99% GPUs, you know, dropped in price by 99% right? Like like if if if GPU-based right? Like like if if if GPU-based right? Like like if if if GPU-based compute actually became, you know, kind compute actually became, you know, kind compute actually became, you know, kind of cheap and widely available like then of cheap and widely available like then of cheap and widely available like then do we get back to a kind of a real open do we get back to a kind of a real open do we get back to a kind of a real open source world where, you know, one person source world where, you know, one person source world where, you know, one person sitting in their basement or 100 people sitting in their basement or 100 people sitting in their basement or 100 people working in their free time can like come working in their free time can like come working in their free time can like come up with something new, try many of these up with something new, try many of these up with something new, try many of these sort of model training paths that, you sort of model training paths that, you sort of model training paths that, you know, that like are in the queue know, that like are in the queue know, that like are in the queue somewhere at one of the big companies somewhere at one of the big companies somewhere at one of the big companies but and you know, kind of kind of see, but and you know, kind of kind of see, but and you know, kind of kind of see, you know, really expand and advance the you know, really expand and advance the you know, really expand and advance the field collectively. field collectively. field collectively. >> Yeah. Yeah, well, I mean, this relates >> Yeah. Yeah, well, I mean, this relates >> Yeah. Yeah, well, I mean, this relates to what you were talking about at the to what you were talking about at the to what you were talking about at the beginning. It's like, you know, at the beginning. It's like, you know, at the beginning. It's like, you know, at the beginning the amount of compute you beginning the amount of compute you beginning the amount of compute you needed to be at the quote-unquote needed to be at the quote-unquote needed to be at the quote-unquote frontier was negligible and now it's frontier was negligible and now it's frontier was negligible and now it's just like it's it's enormous and how do just like it's it's enormous and how do just like it's it's enormous and how do you how do you get that, you know, back you how do you get that, you know, back you how do you get that, you know, back to consumer parity again? to consumer parity again? to consumer parity again? >> I've I've said this on the podcast >> I've I've said this on the podcast >> I've I've said this on the podcast before but I'll keep saying it. AlexNet before but I'll keep saying it. AlexNet before but I'll keep saying it. AlexNet first, you know, kind of like neural first, you know, kind of like neural first, you know, kind of like neural network to run on on GPUs that we care network to run on on GPUs that we care network to run on on GPUs that we care about ran on two GPUs. And that's not about ran on two GPUs. And that's not about ran on two GPUs. And that's not that's not like there are no missing that's not like there are no missing that's not like there are no missing decimal points or commas in there. decimal points or commas in there. decimal points or commas in there. Literally two. Literally two. Literally two. >> Yeah. How that would get [laughter] you >> Yeah. How that would get [laughter] you >> Yeah. How that would get [laughter] you literally nowhere. literally nowhere. literally nowhere. >> Yeah. Yeah. Um so, I guess this this >> Yeah. Yeah. Um so, I guess this this >> Yeah. Yeah. Um so, I guess this this relates to to another question that relates to to another question that relates to to another question that we've had which is which is inference we've had which is which is inference we've had which is which is inference has gotten harder over the past 18 has gotten harder over the past 18 has gotten harder over the past 18 months months months because of this combination of scale and because of this combination of scale and because of this combination of scale and diversity in the kinds of models that we diversity in the kinds of models that we diversity in the kinds of models that we have and of course like agents that are have and of course like agents that are have and of course like agents that are doing increasingly long-running tasks. doing increasingly long-running tasks. doing increasingly long-running tasks. So, So, So, what makes what makes what makes open source in this world and in this open source in this world and in this open source in this world and in this scenario not just like nice and nice scenario not just like nice and nice scenario not just like nice and nice thing to have but absolutely necessary. thing to have but absolutely necessary. thing to have but absolutely necessary. >> Yeah, so scale comes from a few points. >> Yeah, so scale comes from a few points. >> Yeah, so scale comes from a few points. It comes from It comes from It comes from whether or not you can run this gigantic whether or not you can run this gigantic whether or not you can run this gigantic model on a data center rack. And but model on a data center rack. And but model on a data center rack. And but also it comes from can you optimize it also it comes from can you optimize it also it comes from can you optimize it to the extreme point to the speed of to the extreme point to the speed of to the extreme point to the speed of light so that you are getting the most light so that you are getting the most light so that you are getting the most value out of it. But also you need a value out of it. But also you need a value out of it. But also you need a whole community and collaboration and whole community and collaboration and whole community and collaboration and partners of effort to validate it and partners of effort to validate it and partners of effort to validate it and making sure that there's no last making sure that there's no last making sure that there's no last remaining bugs that appear like 0.0001% remaining bugs that appear like 0.0001% remaining bugs that appear like 0.0001% of the time. So this is where scale of the time. So this is where scale of the time. So this is where scale scaling up the deployment and then scaling up the deployment and then scaling up the deployment and then making sure more people are running it making sure more people are running it making sure more people are running it at scale at scale at scale like largest footprint possible will like largest footprint possible will like largest footprint possible will ensure everybody's experience of running ensure everybody's experience of running ensure everybody's experience of running this model is better. And this is where this model is better. And this is where this model is better. And this is where kind of this is why open-source kind of this is why open-source kind of this is why open-source inference is a current leading way right inference is a current leading way right inference is a current leading way right now instead of closed-source inference now instead of closed-source inference now instead of closed-source inference engine. engine. engine. And frankly right all the a lot of the And frankly right all the a lot of the And frankly right all the a lot of the a lot of the open I'm sorry. A lot of a lot of the open I'm sorry. A lot of a lot of the open I'm sorry. A lot of the inference cloud and API as a service the inference cloud and API as a service the inference cloud and API as a service today do use and leverage open-source today do use and leverage open-source today do use and leverage open-source inference engine under the hood. And the inference engine under the hood. And the inference engine under the hood. And the reason to do that is just there's just reason to do that is just there's just reason to do that is just there's just so much battle so much battle so much battle tested and learned recipes and things tested and learned recipes and things tested and learned recipes and things that they can build on top of. And this that they can build on top of. And this that they can build on top of. And this is where open-source is the current is where open-source is the current is where open-source is the current leading way to run models. leading way to run models. leading way to run models. >> Something that I find kind of funny and >> Something that I find kind of funny and >> Something that I find kind of funny and like looking back at not just the like looking back at not just the like looking back at not just the history of LM and Infract but also history of LM and Infract but also history of LM and Infract but also company like Open Router or even Ollama company like Open Router or even Ollama company like Open Router or even Ollama all of these different teams kind of got all of these different teams kind of got all of these different teams kind of got started around 2022 and 2023. started around 2022 and 2023. started around 2022 and 2023. Some of them even before ChatGPT in your Some of them even before ChatGPT in your Some of them even before ChatGPT in your case case case built around open models. And what kind built around open models. And what kind built around open models. And what kind of special thing do you think was of special thing do you think was of special thing do you think was happening at that time just like in the happening at that time just like in the happening at that time just like in the world of research and in the world of AI world of research and in the world of AI world of research and in the world of AI where we see these companies now when we where we see these companies now when we where we see these companies now when we look at them and you think of Y L M is look at them and you think of Y L M is look at them and you think of Y L M is mission-critical or open router is mission-critical or open router is mission-critical or open router is mission-critical. mission-critical. mission-critical. Um, and what do you What do you think Um, and what do you What do you think Um, and what do you What do you think was going on at that time where even was going on at that time where even was going on at that time where even before we had, you know, a big consumer before we had, you know, a big consumer before we had, you know, a big consumer use case, we had, you know, teams like use case, we had, you know, teams like use case, we had, you know, teams like yours that were were building out these yours that were were building out these yours that were were building out these things and and kind of how did you guys things and and kind of how did you guys things and and kind of how did you guys know, I guess, is is the question. know, I guess, is is the question. know, I guess, is is the question. >> I think there's two parts to this. Our >> I think there's two parts to this. Our >> I think there's two parts to this. Our team always have a open source kind of team always have a open source kind of team always have a open source kind of angle where you find UC Berkeley a long angle where you find UC Berkeley a long angle where you find UC Berkeley a long tradition of open source software and tradition of open source software and tradition of open source software and our system research principles. our system research principles. our system research principles. Um, but also just curiosity overall. Um, but also just curiosity overall. Um, but also just curiosity overall. Everybody was so curious about how are Everybody was so curious about how are Everybody was so curious about how are these AI models going benefit the these AI models going benefit the these AI models going benefit the humanity and how can we use it better? humanity and how can we use it better? humanity and how can we use it better? And this is where open source is where And this is where open source is where And this is where open source is where you meet the most mission-aligned people you meet the most mission-aligned people you meet the most mission-aligned people together. Like right, open router, together. Like right, open router, together. Like right, open router, Ollama as you mentioned, they're all Ollama as you mentioned, they're all Ollama as you mentioned, they're all great partners and friends in the great partners and friends in the great partners and friends in the ecosystem because we're there to ecosystem because we're there to ecosystem because we're there to understand how everybody else can understand how everybody else can understand how everybody else can leverage leverage leverage AI model better and making sure they AI model better and making sure they AI model better and making sure they have a good use of it. have a good use of it. have a good use of it. >> And speaking of, you know, leveraging >> And speaking of, you know, leveraging >> And speaking of, you know, leveraging open we open source models really well, open we open source models really well, open we open source models really well, so I think another thing that was in the so I think another thing that was in the so I think another thing that was in the news really recently, um, was Hugging news really recently, um, was Hugging news really recently, um, was Hugging Face using a Chinese open source model Face using a Chinese open source model Face using a Chinese open source model to help contain a cyber attack uh to help contain a cyber attack uh to help contain a cyber attack uh carried out by a rogue unsandboxed open carried out by a rogue unsandboxed open carried out by a rogue unsandboxed open AI model that was being tested. Um, so AI model that was being tested. Um, so AI model that was being tested. Um, so like what can we take away from like what can we take away from like what can we take away from something like that? And can you just something like that? And can you just something like that? And can you just first, I mean, walk us through your first, I mean, walk us through your first, I mean, walk us through your impressions and understanding of what impressions and understanding of what impressions and understanding of what happened and then kind of what what you happened and then kind of what what you happened and then kind of what what you think the the takeaways there are. think the the takeaways there are. think the the takeaways there are. >> Yeah, like this kind of goes back to our >> Yeah, like this kind of goes back to our >> Yeah, like this kind of goes back to our previous point about controls. So for previous point about controls. So for previous point about controls. So for the Hugging Face incident, they break it the Hugging Face incident, they break it the Hugging Face incident, they break it down pretty down pretty down pretty pretty pretty well on their website and pretty pretty well on their website and pretty pretty well on their website and blogs. So really thank for their blogs. So really thank for their blogs. So really thank for their transparency there. transparency there. transparency there. In the end is about all of the closed In the end is about all of the closed In the end is about all of the closed proprietary model APIs, their guardrails proprietary model APIs, their guardrails proprietary model APIs, their guardrails are a little bit arbitrary, but also are a little bit arbitrary, but also are a little bit arbitrary, but also very difficult to enforce. That means very difficult to enforce. That means very difficult to enforce. That means they have so much false positive in the they have so much false positive in the they have so much false positive in the guardrails that will have legitimate use guardrails that will have legitimate use guardrails that will have legitimate use cases just being blocked. And this is cases just being blocked. And this is cases just being blocked. And this is like a evergreen problem even in the like a evergreen problem even in the like a evergreen problem even in the social media days. Like how do you social media days. Like how do you social media days. Like how do you design content filters correctly and design content filters correctly and design content filters correctly and like be able to do the moderation like be able to do the moderation like be able to do the moderation correctly? So if moderation is never correctly? So if moderation is never correctly? So if moderation is never solved, which is going to be very very solved, which is going to be very very solved, which is going to be very very hard, then there's always a place where hard, then there's always a place where hard, then there's always a place where you have a model where you know and you have a model where you know and you have a model where you know and trust that you are publishing to and be trust that you are publishing to and be trust that you are publishing to and be able to use from. So Hugging Face, they able to use from. So Hugging Face, they able to use from. So Hugging Face, they have to use open way model. But in the have to use open way model. But in the have to use open way model. But in the future also see for the trusted use future also see for the trusted use future also see for the trusted use case, people will go to open way by case, people will go to open way by case, people will go to open way by default because that is where you know default because that is where you know default because that is where you know for sure that the guardrail is for sure that the guardrail is for sure that the guardrail is lessened or you can control your lessened or you can control your lessened or you can control your guardrail for trusted use cases. guardrail for trusted use cases. guardrail for trusted use cases. Even for us, like this is actually Even for us, like this is actually Even for us, like this is actually applied to us today where applied to us today where applied to us today where a lot of the Anthropic model do are a lot of the Anthropic model do are a lot of the Anthropic model do are banning Frontier AI research and then banning Frontier AI research and then banning Frontier AI research and then when we're studying GPU kernels even as when we're studying GPU kernels even as when we're studying GPU kernels even as a invalid memory access error, we are a invalid memory access error, we are a invalid memory access error, we are triggering the the red line. And so a triggering the the red line. And so a triggering the the red line. And so a lot of our developer within Inflection lot of our developer within Inflection lot of our developer within Inflection for VLAM and are like retreating from for VLAM and are like retreating from for VLAM and are like retreating from using Fable 5 because you have a 2-hour using Fable 5 because you have a 2-hour using Fable 5 because you have a 2-hour job and you trigger the the red line, job and you trigger the the red line, job and you trigger the the red line, which is false positive, and then you which is false positive, and then you which is false positive, and then you have to lose all of your work and so a have to lose all of your work and so a have to lose all of your work and so a lot of our developer are using like Kimi lot of our developer are using like Kimi lot of our developer are using like Kimi case for today even just making sure case for today even just making sure case for today even just making sure because it's a similar quality and it because it's a similar quality and it because it's a similar quality and it has a has a has a guardrail that makes sense to us. Zero guardrail that makes sense to us. Zero guardrail that makes sense to us. Zero goes while using it. goes while using it. goes while using it. >> I think your social media analogy is a >> I think your social media analogy is a >> I think your social media analogy is a really apt one. really apt one. really apt one. >> Yeah. >> Yeah. >> Yeah. >> Cuz like in both in both cases what's >> Cuz like in both in both cases what's >> Cuz like in both in both cases what's kind of happened is you've taken like kind of happened is you've taken like kind of happened is you've taken like distributed human activity and kind of distributed human activity and kind of distributed human activity and kind of centralized in in place. Like like centralized in in place. Like like centralized in in place. Like like before social media, people talked to before social media, people talked to before social media, people talked to each other and published articles and each other and published articles and each other and published articles and like messaged each other and had message like messaged each other and had message like messaged each other and had message boards. But then you centralize it all boards. But then you centralize it all boards. But then you centralize it all under one kind of like profit-seeking under one kind of like profit-seeking under one kind of like profit-seeking enterprise, and all of a sudden the enterprise, and all of a sudden the enterprise, and all of a sudden the incentives change like a lot, right? And incentives change like a lot, right? And incentives change like a lot, right? And we we have a specific carve-out for we we have a specific carve-out for we we have a specific carve-out for social media, which is you're not social media, which is you're not social media, which is you're not responsible for what people say on your responsible for what people say on your responsible for what people say on your platform. And that allowed the platform. And that allowed the platform. And that allowed the moderation problem to be tractable, I moderation problem to be tractable, I moderation problem to be tractable, I think, right? It's like, "Okay, we're think, right? It's like, "Okay, we're think, right? It's like, "Okay, we're going to do our best to eliminate going to do our best to eliminate going to do our best to eliminate obviously illegal things, um you know, obviously illegal things, um you know, obviously illegal things, um you know, like things that are just going to like things that are just going to like things that are just going to degrade experience on the site, you degrade experience on the site, you degrade experience on the site, you know, you know, in in in in really bad know, you know, in in in in really bad know, you know, in in in in really bad ways. But like, you know, if it's on the ways. But like, you know, if it's on the ways. But like, you know, if it's on the edge, you know, it's something that's edge, you know, it's something that's edge, you know, it's something that's maybe out of fashion to say, but not maybe out of fashion to say, but not maybe out of fashion to say, but not illegal. illegal. illegal. >> Yeah. >> Yeah. >> Yeah. >> Like we're just we we we just can't >> Like we're just we we we just can't >> Like we're just we we we just can't police everything. We can't be the world police everything. We can't be the world police everything. We can't be the world police of like all human communications. police of like all human communications. police of like all human communications. I think something similar is sort of I think something similar is sort of I think something similar is sort of happening in AI, right? Where a lot of happening in AI, right? Where a lot of happening in AI, right? Where a lot of work, not just talking, but work, is work, not just talking, but work, is work, not just talking, but work, is kind of being consolidated in in one kind of being consolidated in in one kind of being consolidated in in one place. If I need to place. If I need to place. If I need to write code or create a spreadsheet or write code or create a spreadsheet or write code or create a spreadsheet or get advice, you know, health advice or get advice, you know, health advice or get advice, you know, health advice or like like anything, it's all happening like like anything, it's all happening like like anything, it's all happening in like one or two sites. in like one or two sites. in like one or two sites. >> Yeah. >> Yeah. >> Yeah. >> And they don't have that carve-out, >> And they don't have that carve-out, >> And they don't have that carve-out, right? Like they don't have that right? Like they don't have that right? Like they don't have that exemption of of like, "Hey, we're not exemption of of like, "Hey, we're not exemption of of like, "Hey, we're not responsible for what actually happens on responsible for what actually happens on responsible for what actually happens on the site." And like some of them, the site." And like some of them, the site." And like some of them, especially, you know, Anthropic, like is especially, you know, Anthropic, like is especially, you know, Anthropic, like is is kind of going further than even what is kind of going further than even what is kind of going further than even what would be sort of like legally required, would be sort of like legally required, would be sort of like legally required, and they're sort of taking ethical and they're sort of taking ethical and they're sort of taking ethical stances on these things, which may be stances on these things, which may be stances on these things, which may be right or may be wrong, right? That's right or may be wrong, right? That's right or may be wrong, right? That's sort of their their decisions. And sort of their their decisions. And sort of their their decisions. And how do you actually do this, right? Like how do you actually do this, right? Like how do you actually do this, right? Like the problem is actually bigger than just the problem is actually bigger than just the problem is actually bigger than just communication, cuz it's like actual work communication, cuz it's like actual work communication, cuz it's like actual work in action, and you don't have this in action, and you don't have this in action, and you don't have this exemption, and you have this sort of exemption, and you have this sort of exemption, and you have this sort of layer of like ethical stuff on top. So I layer of like ethical stuff on top. So I layer of like ethical stuff on top. So I I yeah, I'm I'm just totally I yeah, I'm I'm just totally I yeah, I'm I'm just totally It's I'm just totally with you. It's It's I'm just totally with you. It's It's I'm just totally with you. It's like and and it seems like like and and it seems like like and and it seems like some of these com- you know, companies some of these com- you know, companies some of these com- you know, companies have gone like erred on the side of have gone like erred on the side of have gone like erred on the side of caution, which is which is probably a caution, which is which is probably a caution, which is which is probably a reasonable thing to do, but very very reasonable thing to do, but very very reasonable thing to do, but very very frustrating as as a user, right? Like I frustrating as as a user, right? Like I frustrating as as a user, right? Like I you know, I saw online some translation you know, I saw online some translation you know, I saw online some translation attempts are blocked because they think attempts are blocked because they think attempts are blocked because they think the particular combination of like the particular combination of like the particular combination of like language and content is somehow like language and content is somehow like language and content is somehow like triggering like politically sensitive triggering like politically sensitive triggering like politically sensitive topics and you know things like this. topics and you know things like this. topics and you know things like this. >> Um Simon, we're we're nearing the close >> Um Simon, we're we're nearing the close >> Um Simon, we're we're nearing the close of the conversation of the conversation of the conversation and I just wanted to take a step back a and I just wanted to take a step back a and I just wanted to take a step back a bit and ask just about Infract and you bit and ask just about Infract and you bit and ask just about Infract and you know running the company and I know that know running the company and I know that know running the company and I know that Ion Stoica of Databricks is an advisor Ion Stoica of Databricks is an advisor Ion Stoica of Databricks is an advisor and a co-founder of Infract and I'm just and a co-founder of Infract and I'm just and a co-founder of Infract and I'm just curious what you've learned from him in curious what you've learned from him in curious what you've learned from him in terms of taking open-source project like terms of taking open-source project like terms of taking open-source project like Llama 2 LM and building a company. Llama 2 LM and building a company. Llama 2 LM and building a company. >> Yeah, Ion as the co-founder he has >> Yeah, Ion as the co-founder he has >> Yeah, Ion as the co-founder he has always been thinking about open-source always been thinking about open-source always been thinking about open-source and where how do you support open-source and where how do you support open-source and where how do you support open-source better? And then now with experience better? And then now with experience better? And then now with experience from Databricks and Anyscale at and even from Databricks and Anyscale at and even from Databricks and Anyscale at and even Arena which is a public platform and now Arena which is a public platform and now Arena which is a public platform and now at Infract his focus really stem from at Infract his focus really stem from at Infract his focus really stem from how do we build such a how do we build such a how do we build such a you know world where this software is you know world where this software is you know world where this software is being used by so so so many being used by so so so many being used by so so so many people on such a critical project. How people on such a critical project. How people on such a critical project. How do you make sure they're getting the do you make sure they're getting the do you make sure they're getting the best quality and where does the value best quality and where does the value best quality and where does the value come from? Right? If the open-source come from? Right? If the open-source come from? Right? If the open-source movement and open-source inference movement and open-source inference movement and open-source inference engine is a starting point, then what engine is a starting point, then what engine is a starting point, then what are the gaps that we can fill as Infract are the gaps that we can fill as Infract are the gaps that we can fill as Infract and then what are the gaps that we and then what are the gaps that we and then what are the gaps that we should absolutely put in? Ion has always should absolutely put in? Ion has always should absolutely put in? Ion has always been open-source first. So for whatever been open-source first. So for whatever been open-source first. So for whatever we need to build, we would like to build we need to build, we would like to build we need to build, we would like to build them in open-source and then for them in open-source and then for them in open-source and then for whatever value we can deliver we want to whatever value we can deliver we want to whatever value we can deliver we want to really close the last mile and making really close the last mile and making really close the last mile and making sure that customer and partners are sure that customer and partners are sure that customer and partners are getting the best out of it. getting the best out of it. getting the best out of it. >> Just I guess to close this out 5 years >> Just I guess to close this out 5 years >> Just I guess to close this out 5 years from now, from now, from now, do you think open weight open-source AI do you think open weight open-source AI do you think open weight open-source AI models have they closed the gap with models have they closed the gap with models have they closed the gap with frontier models completely? Are frontier frontier models completely? Are frontier frontier models completely? Are frontier models always one step ahead? Kind of models always one step ahead? Kind of models always one step ahead? Kind of how how do you see that shaking out? how how do you see that shaking out? how how do you see that shaking out? >> 5 years >> 5 years >> 5 years >> All right. All right. 1 year. One year. >> All right. All right. 1 year. One year. >> All right. All right. 1 year. One year. You got five years wrong. You got five years wrong. You got five years wrong. >> [laughter] >> [laughter] >> [laughter] >> Who knows? >> Who knows? >> Who knows? >> Or you know, we're all going to be like >> Or you know, we're all going to be like >> Or you know, we're all going to be like just floating around in our Wall-E pods just floating around in our Wall-E pods just floating around in our Wall-E pods and our spaceships. [laughter] So yeah. and our spaceships. [laughter] So yeah. and our spaceships. [laughter] So yeah. >> Exactly. >> Exactly. >> Exactly. >> Progress, right? >> Progress, right? >> Progress, right? >> Yeah. Yeah. >> Yeah. Yeah. >> Yeah. Yeah. >> Uh for for me really at this point >> Uh for for me really at this point >> Uh for for me really at this point there's kind of a there's kind of a there's kind of a point we haven't talked too much about point we haven't talked too much about point we haven't talked too much about is what really differentiate opens open is what really differentiate opens open is what really differentiate opens open way model from closed way model, right? way model from closed way model, right? way model from closed way model, right? In the end there's not much In the end there's not much In the end there's not much differentiation. It's a more about the differentiation. It's a more about the differentiation. It's a more about the distribution strategy and go-to-market distribution strategy and go-to-market distribution strategy and go-to-market strategy. And the capability-wise, strategy. And the capability-wise, strategy. And the capability-wise, I don't really see a big gap, not even I don't really see a big gap, not even I don't really see a big gap, not even today, because today, because today, because for how these model are coming to be, for how these model are coming to be, for how these model are coming to be, they're really starting from the first they're really starting from the first they're really starting from the first principle, right? You have a computer principle, right? You have a computer principle, right? You have a computer cluster, you have training data, and you cluster, you have training data, and you cluster, you have training data, and you have brilliant researchers uh that group have brilliant researchers uh that group have brilliant researchers uh that group together and really to build this together and really to build this together and really to build this amazing artifact that is this model amazing artifact that is this model amazing artifact that is this model pre-trained model and then later are old pre-trained model and then later are old pre-trained model and then later are old uh post-trained model and that's a world uh post-trained model and that's a world uh post-trained model and that's a world can use. But if you look at the can use. But if you look at the can use. But if you look at the ingredients, right? The one of the most ingredients, right? The one of the most ingredients, right? The one of the most important part is the data. It's about important part is the data. It's about important part is the data. It's about who gets what data and then what are the who gets what data and then what are the who gets what data and then what are the environment you are building to that's a environment you are building to that's a environment you are building to that's a model improve on itself and make better, model improve on itself and make better, model improve on itself and make better, right? One of the very useful uh right? One of the very useful uh right? One of the very useful uh benchmark that we have on Arena for uh benchmark that we have on Arena for uh benchmark that we have on Arena for uh for for case three has been front-end for for case three has been front-end for for case three has been front-end coding, right? That means for coding, right? That means for coding, right? That means for Moonshot, they have built some of the Moonshot, they have built some of the Moonshot, they have built some of the best environment for front-end coding, best environment for front-end coding, best environment for front-end coding, right? They have published amazing demo right? They have published amazing demo right? They have published amazing demo on the ability for this model to code on the ability for this model to code on the ability for this model to code and then see what the rendered is and and then see what the rendered is and and then see what the rendered is and then kind of continue looping on this then kind of continue looping on this then kind of continue looping on this iterative process. Now, this is about iterative process. Now, this is about iterative process. Now, this is about their environment to improve the model. their environment to improve the model. their environment to improve the model. It's not about just source data, it's It's not about just source data, it's It's not about just source data, it's not about where they get the data from, not about where they get the data from, not about where they get the data from, rather is who can build the best rather is who can build the best rather is who can build the best environment and who can make the most environment and who can make the most environment and who can make the most sort of sort of sort of uh optimization and algorithmic choices uh optimization and algorithmic choices uh optimization and algorithmic choices to leverage all those learning from this to leverage all those learning from this to leverage all those learning from this environment. So, the next year is all environment. So, the next year is all environment. So, the next year is all going to be about that. It's about going to be about that. It's about going to be about that. It's about how open we model labs are how open we model labs are how open we model labs are differentiating and really getting the differentiating and really getting the differentiating and really getting the model to meet the real world and have model to meet the real world and have model to meet the real world and have this kind of what people are popular this kind of what people are popular this kind of what people are popular today like recursive self-improvement today like recursive self-improvement today like recursive self-improvement almost to really improve the model almost to really improve the model almost to really improve the model overall. overall. overall. And so, really project out a year, And so, really project out a year, And so, really project out a year, there's not going to be any difference. there's not going to be any difference. there's not going to be any difference. >> Yeah. >> Yeah. >> Yeah. >> Yeah. And and >> Yeah. And and >> Yeah. And and you've used this term brilliant you've used this term brilliant you've used this term brilliant researchers a few times. Um researchers a few times. Um researchers a few times. Um there are brilliant researchers there are brilliant researchers there are brilliant researchers everywhere in the world, clearly. Um everywhere in the world, clearly. Um everywhere in the world, clearly. Um what why do you think what why do you think what why do you think you know, in the US all the smart you know, in the US all the smart you know, in the US all the smart researchers are working on closed models researchers are working on closed models researchers are working on closed models and in and in China all the smart and in and in China all the smart and in and in China all the smart researchers are working on open models? researchers are working on open models? researchers are working on open models? >> I mean, from my point of view, they are >> I mean, from my point of view, they are >> I mean, from my point of view, they are attracted to interesting problems, not attracted to interesting problems, not attracted to interesting problems, not necessarily on the open or closed necessarily on the open or closed necessarily on the open or closed stance. But rather but however, open way stance. But rather but however, open way stance. But rather but however, open way model does give people a really, really model does give people a really, really model does give people a really, really good boost on the impact of such models. good boost on the impact of such models. good boost on the impact of such models. So, that is like a plus. So, that is like a plus. So, that is like a plus. >> Mhm. >> Mhm. >> Mhm. >> And I think all the brilliant >> And I think all the brilliant >> And I think all the brilliant researchers are attracted to researchers are attracted to researchers are attracted to how to improve the model overall, right? how to improve the model overall, right? how to improve the model overall, right? I actually one interesting point about I actually one interesting point about I actually one interesting point about this um this um this um maybe fairly technical for for this Kimi maybe fairly technical for for this Kimi maybe fairly technical for for this Kimi case three model is they removed a case three model is they removed a case three model is they removed a rotary positional embedding. So, Rope rotary positional embedding. So, Rope rotary positional embedding. So, Rope has always been there for a lot of the has always been there for a lot of the has always been there for a lot of the Transformers model. Transformers model. Transformers model. And guess who removed it? Is the And guess who removed it? Is the And guess who removed it? Is the inventor of Rope, who are inventor of Rope, who are inventor of Rope, who are >> Oh, that's cool. >> Oh, that's cool. >> Oh, that's cool. >> like the the >> like the the >> like the the Jenning, he he wrote the first paper Jenning, he he wrote the first paper Jenning, he he wrote the first paper introducing Rope as a concept. And then introducing Rope as a concept. And then introducing Rope as a concept. And then he now also wrote the explanation of why he now also wrote the explanation of why he now also wrote the explanation of why you don't need it you don't need it you don't need it as part of the technical report in this as part of the technical report in this as part of the technical report in this Kimi case three model. So, like when we Kimi case three model. So, like when we Kimi case three model. So, like when we read it, it's like really come full read it, it's like really come full read it, it's like really come full circle is you have all these brilliant, circle is you have all these brilliant, circle is you have all these brilliant, uh uh uh researchers that are able to really researchers that are able to really researchers that are able to really study how this work and really study the study how this work and really study the study how this work and really study the secret of training and pre-training and secret of training and pre-training and secret of training and pre-training and share it across the whole world and share it across the whole world and share it across the whole world and recognizing and iterating on their past, recognizing and iterating on their past, recognizing and iterating on their past, right? So, really a a miracle, I would right? So, really a a miracle, I would right? So, really a a miracle, I would say, for this model to come alive. say, for this model to come alive. say, for this model to come alive. >> It's so It's so interesting like >> It's so It's so interesting like >> It's so It's so interesting like you know, AI is in this funny zone where you know, AI is in this funny zone where you know, AI is in this funny zone where empirically it works incredibly well. empirically it works incredibly well. empirically it works incredibly well. >> Yeah. >> Yeah. >> Yeah. >> But then you go ask the theorists and >> But then you go ask the theorists and >> But then you go ask the theorists and they have like no idea what's going on. they have like no idea what's going on. they have like no idea what's going on. >> Right. It's so like >> Right. It's so like >> Right. It's so like >> You have these kind of iterative things >> You have these kind of iterative things >> You have these kind of iterative things where when you go read the, you know, where when you go read the, you know, where when you go read the, you know, primer on transformers, primer on transformers, primer on transformers, >> Yeah. >> Yeah. >> Yeah. >> you read about positional embeddings and >> you read about positional embeddings and >> you read about positional embeddings and why are all positional codings and why why are all positional codings and why why are all positional codings and why it's so important because otherwise you it's so important because otherwise you it's so important because otherwise you can't sort of like understand meaning, can't sort of like understand meaning, can't sort of like understand meaning, >> Yeah. >> Yeah. >> Yeah. >> And then it turns out once you >> And then it turns out once you >> And then it turns out once you understand one level deeper because understand one level deeper because understand one level deeper because we've been doing this for a few years we've been doing this for a few years we've been doing this for a few years and you have all these smart people and you have all these smart people and you have all these smart people like, "Oh, actually you don't need like, "Oh, actually you don't need like, "Oh, actually you don't need >> Exactly. Exactly. Yeah. >> Exactly. Exactly. Yeah. >> Exactly. Exactly. Yeah. >> simpler actually is better. Um >> simpler actually is better. Um >> simpler actually is better. Um We We didn't um We We didn't um We We didn't um We didn't talk about distillation much We didn't talk about distillation much We didn't talk about distillation much so far in this conversation, but I think so far in this conversation, but I think so far in this conversation, but I think it's very relevant to this like it's very relevant to this like it's very relevant to this like >> Yeah. >> Yeah. >> Yeah. >> I I I have just one question which is >> I I I have just one question which is >> I I I have just one question which is like I'm I'm not going to ask like is like I'm I'm not going to ask like is like I'm I'm not going to ask like is distillation happening? I think this is distillation happening? I think this is distillation happening? I think this is kind of speculation on the part of kind of speculation on the part of kind of speculation on the part of everybody, you know, in the world, but everybody, you know, in the world, but everybody, you know, in the world, but like like like you work a lot with these Chinese labs. you work a lot with these Chinese labs. you work a lot with these Chinese labs. Do you think Do you think Do you think distillation like is a critical distillation like is a critical distillation like is a critical component of what they they component of what they they component of what they they do or or or like are they kind of just do or or or like are they kind of just do or or or like are they kind of just doing good work and and you know doing good work and and you know doing good work and and you know distillation if if it's done is sort of distillation if if it's done is sort of distillation if if it's done is sort of an incidental part of it? an incidental part of it? an incidental part of it? >> I I will lean to the latter part >> I I will lean to the latter part >> I I will lean to the latter part specifically, as I mentioned previously, specifically, as I mentioned previously, specifically, as I mentioned previously, environment matters so much today. So, environment matters so much today. So, environment matters so much today. So, these are our own environments, right? these are our own environments, right? these are our own environments, right? These cannot be distilled. Like you These cannot be distilled. Like you These cannot be distilled. Like you don't have other people's environment to don't have other people's environment to don't have other people's environment to really distill a copy from. It's about really distill a copy from. It's about really distill a copy from. It's about constructing it, understanding also constructing it, understanding also constructing it, understanding also understanding the learning process. You understanding the learning process. You understanding the learning process. You cannot distill how the model learns with cannot distill how the model learns with cannot distill how the model learns with the environment. A lot of these are just the environment. A lot of these are just the environment. A lot of these are just not doable today. Um there are things not doable today. Um there are things not doable today. Um there are things potentially you can do with rewriting potentially you can do with rewriting potentially you can do with rewriting the data sets, right? Making better the data sets, right? Making better the data sets, right? Making better pre-training data, but again, you can do pre-training data, but again, you can do pre-training data, but again, you can do it with any models. Any models are are it with any models. Any models are are it with any models. Any models are are going to follow instructions, are going going to follow instructions, are going going to follow instructions, are going to be useful in terms of utility there. to be useful in terms of utility there. to be useful in terms of utility there. So, I really don't think from currently So, I really don't think from currently So, I really don't think from currently what we're seeing, what we're seeing, what we're seeing, this is a big cornerstone of what's this is a big cornerstone of what's this is a big cornerstone of what's powering the progress today. In the end, powering the progress today. In the end, powering the progress today. In the end, what's powering the progress is still what's powering the progress is still what's powering the progress is still just just just really smart people with very really smart people with very really smart people with very interesting algorithms, data interesting algorithms, data interesting algorithms, data environment, and they will produce, of environment, and they will produce, of environment, and they will produce, of course, compute. They will produce the course, compute. They will produce the course, compute. They will produce the models. models. models. >> I think it has really interesting policy >> I think it has really interesting policy >> I think it has really interesting policy implications. I I tend to agree with implications. I I tend to agree with implications. I I tend to agree with you, by the way, that that, you know, we you, by the way, that that, you know, we you, by the way, that that, you know, we have smart people everywhere working on have smart people everywhere working on have smart people everywhere working on a bunch of smart things, and it's not a bunch of smart things, and it's not a bunch of smart things, and it's not about, you know, distilling data from about, you know, distilling data from about, you know, distilling data from any one place. Um it has really any one place. Um it has really any one place. Um it has really interesting policy implications, right? interesting policy implications, right? interesting policy implications, right? Because it doesn't, you know, it's Because it doesn't, you know, it's Because it doesn't, you know, it's almost tempting if you're if you're sort almost tempting if you're if you're sort almost tempting if you're if you're sort of, you know, in the White House to say, of, you know, in the White House to say, of, you know, in the White House to say, "Oh, sure, we'll just we'll just turn "Oh, sure, we'll just we'll just turn "Oh, sure, we'll just we'll just turn off distillation, all our problems will off distillation, all our problems will off distillation, all our problems will be solved." But like, you know, I think be solved." But like, you know, I think be solved." But like, you know, I think it's more the case that they're just, it's more the case that they're just, it's more the case that they're just, you know, smart people doing interesting you know, smart people doing interesting you know, smart people doing interesting things. And so, it's so it's like, how things. And so, it's so it's like, how things. And so, it's so it's like, how do we how do we kind of like adapt adapt do we how do we kind of like adapt adapt do we how do we kind of like adapt adapt to that? I think it's to that? I think it's to that? I think it's >> Yeah, and creative innovation. Straight, >> Yeah, and creative innovation. Straight, >> Yeah, and creative innovation. Straight, like one part in my essay will kind of like one part in my essay will kind of like one part in my essay will kind of mention that open source and open way mention that open source and open way mention that open source and open way really helps innovation because it set really helps innovation because it set really helps innovation because it set out this racetrack where everybody can out this racetrack where everybody can out this racetrack where everybody can learn from each other and see where each learn from each other and see where each learn from each other and see where each person, like, every each player is in person, like, every each player is in person, like, every each player is in this racetrack, and then you're able to this racetrack, and then you're able to this racetrack, and then you're able to improve and stand on shoulder of each improve and stand on shoulder of each improve and stand on shoulder of each other kind of to improve yourself. So, other kind of to improve yourself. So, other kind of to improve yourself. So, uh it's that is where everybody can move uh it's that is where everybody can move uh it's that is where everybody can move forward faster. forward faster. forward faster. >> Yeah, and one thing we're looking for a >> Yeah, and one thing we're looking for a >> Yeah, and one thing we're looking for a lot from an investment standpoint is um lot from an investment standpoint is um lot from an investment standpoint is um is people doing more open source model is people doing more open source model is people doing more open source model training all all over the world, not training all all over the world, not training all all over the world, not just in China, you know, not just in the just in China, you know, not just in the just in China, you know, not just in the United States, but all over the Right, United States, but all over the Right, United States, but all over the Right, because you know, you get that sort of because you know, you get that sort of because you know, you get that sort of magic of collaboration when everybody's magic of collaboration when everybody's magic of collaboration when everybody's doing it. And achieve global harmony and doing it. And achieve global harmony and doing it. And achieve global harmony and all all the all all the all all the >> Well, I think that's a good note to end >> Well, I think that's a good note to end >> Well, I think that's a good note to end on. on. on. Um Matt, Simon, thank you so much for Um Matt, Simon, thank you so much for Um Matt, Simon, thank you so much for joining us. Thanks for tuning in. joining us. Thanks for tuning in. joining us. Thanks for tuning in. >> Thanks so much, Cole. Thanks, Shen.