Stanford CS153 Frontier Systems | Teaching AI to Touch Atoms — Transcript

This one's going to be good. Um, I'm so excited to welcome today Liam Ferris and Doge Chub. Thanks for coming, guys. Very nice. Uh, Liam and Doge are the co-founders of a company called Periodic Labs. How many people here are aware of Periodic Labs? The kids are paying attention. Love that. >> Let's just start with between when so you know you were part of the you helped set up the post training team at at uh OpenAI and were there for the chat GPT moment uh which you know arguably you and the team there kind of helped shape that continuous post training loop and then doge on your side you had worked on genome over at deep mine. So these two parallel trees were like progressing and then periodic came together a year ago.

We're almost at one year in now, right? No, it's uh it's May now. Okay. So we're like 11 months in. >> Yeah. Yeah. >> We'll call it like June. We kind of started. >> Okay. So if you had to assess what updates what priors have you had to update most aggressively over those the last 11 months relative to the priors you had before starting periodic. >> I have one. Right. >> Um, so when we were founding periodic, we knew that building up these labs would take some time, especially like a high throughput autonomous lab. So very much in the spirit of AI and machine learning, we're following just scaling these things up. Uh, we think new opportunities in science will come at scale. But we had this kind of cartoon version of periodic where it's like, okay, well, the first year obviously it's going to take some time to build up the physical infrastructure. So our first year is going to be in silica.

we're going to be doing design of new materials just strictly computationally. Then year one we walk up to the high throughput lab, flip it on and start doing science. And that's not at all how it went. So instead it's like creating these small labs semi-manual semi-autonomous uh allowed us to direct the research program, understand like what kind of equipment we want to scale up to the big lab and just you know to your point earlier close the feedback loop as quickly as possible. Uh so that was that was a huge update.

>> Any you do at Doge do any prior you had to update? >> I think we were really excited about kind of having these LLMs not just do coding but impact real atoms. So we were expecting that we'd see crazy results but I think the results are crazier than we expected such as >> like there's something amazing happens when you had automated intelligence affecting not just Python but the actual atoms that surround you. Um, so we've been able to um synthesize materials much better than before. We've been able to make computational progress much better than before. Uh, and it's just starting. So, it's been really exciting.

I I realize it might be helpful to I know it's a cartoon version that you described we had uh a year ago, but it might be helpful to simplify just so everybody understands what the pipeline is. Let me let me try explaining and you guys can red team it. Um there's a 30,000 square foot facility in Menlo Park. I >> think 40 >> 40,000. Okay, we got a 40,000 square foot facility in Menlo Park. Half the team is former machine learning uh folks from OpenAI, Deep Mind and so on. And the other half uh is on the floor is um physicists, chemists and so on from Stanford, MIT, Caltech and so on. And then just you know across the so that that's the if you had to think about the layout of the lab half of the square you know footage is is people sitting from two different disciplines and then the other half is a automated facility where there's robots there's there's AI um there's an AI system that you guys have been training which is the end result of

the pre-training mid-training post-training pipeline. Those AIs then predict new materials, ideally searching for high temperature superconductor candidates. Then a robot or a set of robots synthesizes those materials into certain forms, powder or whatever it might be. Then there are machines that verify whether those materials have the properties that the AI said it would and then you pipe that feedback that verification back into the training loop over and over again. Is that roughly correct?

>> That's right. And maybe one thing that's not as intuitive is the AI's job isn't to just make these like very uh grand predictions about what's a good superconductor, but there are lots of small things that you have to do to make progress in science. Like very minor things like how do you mix powders correctly or like how do you make sure this impurity isn't there or like we often have sample mixups? How do you detect the sample mixup and correct it before it affects results? So AI is actually extremely helpful on those mundane things as well. And to be honest, scientific progress is like a bunch of mundane things attached to each other. So, um, it's been really good for that as well. Yeah. Yeah. I mean, I think the task is much easier than coming up with some candidates doing this full loop and you're like, okay, do I have a new hightemp superconductor or not? There's a lot of uh local error correction. So, you can say, how good is my system at characterizing some materials? So, materials don't come out of an oven with like labels on them. you have to figure out what you actually made. So you shoot like one process is you shoot X-rays at it. You look at the

diffraction pattern and making sense of these patterns uh can be difficult. So that's a system that like we focus on building. Um but then when you kind of get all these little pieces together then that constitutes the untan loop. Um I'm going to connect the dots between this lecture and the previous one. We had Robiachi from DeepMind. Um and Rob's been part of the pre-training team on Nana Banana and their world models and so on. And one of the things if you guys remember Rob talked about was when advising on how to uh execute on their projects. The final project for the class is the oneperson frontier lab. He said you know the the key is to start when even if you're trying to build a very general capability and in your guys's case it's a um general reasoning engine about the physical world like physics and chemistry.

uh his advice was start with an eval that's specific get to the state-of-the-art on that domain or that eval and then go from there. In your guys's case that eval has ended up being or that domain has ended up being semiconductors uh as a result of superconductors. Can you talk a little bit about why semiconductors is the right place to measure evals or am I even describing it correctly? >> Yeah, I mean I think um maybe I'll I'll I'll kind of separate the question. So one is do you start your ML campaign with an eval? And one thing that we've seen is science has like a lot of messiness. It's decision-m under uncertainty. And we've seen huge progress for reasoning models when there's this great verifiability, this objectiveness to it. So a task like what is 2 plus two? You know the ground truth is four. Uh the labs have become excellent at throwing a huge amount of computation against these things. Uh but in science it's often kind of murkier.

It's like well we have some evidence of um from you know this instrument. We have some other evidence from adjacent experiments. How do we make sense of that? How do we decide what to do next? Um, and I think we've learned at periodic that it's actually necessary to spend a bit more time learning the shape of the problem uh before just like here's my like eval and just like blinders on hill climb that um and then maybe to the point about semiconductors um periodic has an extremely deep understanding of like atomistic physics and everyone knows semiconductors are shrinking and there's uh a really good opportunity to start modeling and sort alleviating some of these big bottlenecks and every AI company is scooping up as many chips as possible.

There's just incredible demand. Every new generation of like logic and memory is running into materials engineering problems and so we want to alleviate that. So we want to accelerate the progress even further. Uh computation is physical and we want to help accelerate it. >> I mean yeah and from the physics side as well. So um what governs superc conductivity and what governs semiconductor properties interfaces are pretty similar. Uh you know we have this hypothesis that AGI won't be this magical thing that generalizes to everything. If that's true then you want to pick a direction that you think really matters and then push AI in that direction. So we felt like interaction between atoms and electrons is where we want to be. Um, and it's super fun and that works really well for superc conductivity because there's one form of superc conductivity that's just uh phonons and electrons interacting and there's one form of superc conductivity that we don't understand yet but we feel like it's related to phonons, electrons and maybe magnetic uh properties.

>> We should do like a show of hands of how many people understood that state. >> Yeah, as I was saying that I was looking at uh at least one. Yeah. Um so you know as a quick show how many people are physics majors okay so at least five people understood that okay >> like taking a step back right of course we're not doing polymers in the lab right now so we don't expect that what we're doing necessarily extends to polymers at automistic scale but superconductors semiconductors is like a very deep part of human technology and there's a sense in which Mo's law is the most impressive thing humanity has done um so we really want to keep pushing on that and um we've learned so much for example we're pushing on super conduct video but we learned so much about synthesizability of materials how to synize them better because again like they are governed by similar things like thermodynamics automistic interactions I I think given that feedback we're going to have to spend like two minutes just giving people context on what is a superconductor >> right >> why is it useful and what it's what is its link to semiconductors can we just

for just for uh exhaustiveness start there >> so as we You know, physics is endless. It's boundless. And there are different energy scales. So, you know, you can go really deep and start studying string theory or quarks. Um, but those studies don't affect our day-to-day life very much. The OZX the the the energy scale that affects our life is usually, you know, a few EVs and this governs chemistry, biology, but also batteries, semiconductors. In that energy scale, the thing that really matters is uh quantum mechanics. So it's not even necessarily quantum field theory, but you might benefit from that. But it's just quantum mechanics. And in that regime, you just have atoms and electrons. It's ions and electrons and they're interacting with each other. So this is like if you've taken a introductory solid state physics class, this is what you would learn. Uh that's the level at which we're studying. We're not doing any quarks. We're not doing any strings. Well, for the non-physicists, let's even though this actually, you know, this is systems class, we have mastery over different um

frameworks. For a second, let's reason by analogy. How about we say a superconductor is a pipe through which things need to flow. Can you build on that analogy for a sec? >> Yeah. So, um one thing that we've learned about materials in the last century or two is that, uh when you pass current through a material, you have resistance. um depending on what the material is that resistance might be driven by electrons colliding with ions and every time there's scattering you're basically losing uh some uh to energy to heat >> so on average today in materials like on a in semiconductor on a chip what percentage of energy is lost >> yeah so like for a copper wire it will all be basically heat to resistance right um so like a major amount so if you're putting a lot of energy into these chips In data centers, you're also suffering from uh heat loss. You also have to then dissipate that heat.

>> So would be fair, roughly on average at least 50% of all energy that goes into data center is wasted because of materials bottlenecks. >> I mean we've been told that Moors law might be like a combination of just materials improvements uh stitched on top of each other. Um what's what was amazing is back in 1910 around um there was the first time we could uh liquefy helium and we realized that there are certain materials that don't have any resistivity. So uh current can just blow through it without any uh dissipation >> and that's an incredible observation.

One of the most amazing things we learned about um physics but we still haven't really like taken that technology and apply it to our life enough. Uh we're still using very old superconductors and MR machines. you're using liquid helium and if you think about the exciting future we can imagine like fusion energy uh quantum computers uh mag levs uh lossless transmission these all require superconductors to some extent >> well what I've learned uh interact with dos is if I say something like at least 50% he doesn't answer that means that's a no point >> number exactly let's say no >> double digit percentages are lost in a in a chip because of energy transmission loss is would that be accurate?

>> Yeah, seems to be not. >> Okay, let's start there. Um, and then the idea is by using an AI system to discover new materials that reduces that loss, we can unlock extraordinary gains in efficiency in energy transmission. Roughly that would be the EVA, right? That would be one. You can also do communication with these um and you can do exciting physics like quantum computing, crazy magnetic fields for fusion. Um I think it's also important to add that there's a a huge generality. So our ability to understand and engineer matter uh matters for every single advanced industry. So you know there's applications in semiconductors but aerospace, auto energy everywhere.

Um and there's kind of other areas in uh semiconductor manufacturing that we're looking as well. Just a there's materials interesting physics and materials engineering challenges uh throughout the whole stack. >> Are we ready to talk about the name of the agent that you guys have been working on? What no you should go ahead and talk about why you named it that. I think it's a interesting metaphor for what what's going on. >> One of our incredible researchers, manage has named it Anest. And there are two reasons. Anest was the one who whose lab liquefied helium and discovered superc conductivity. But also because Anes was one of the first people who said scientific research should be done at industrial scale. So he was kind of strange for his time like back in 1908 where he would hire professional engineers, professional technicians and run kind of like an industrial scale lab to do scientific research and that really you know is inspiring to us because we feel like science is very important and it should be done very seriously at industrial scale with a big sense of urgency.

Honest as in O N E S. >> Yeah. >> Which a few people have told me they thought meant honest, but like sounded like honest, but um how should like could you contrast Liam the honest system relative to what was a system the system you were working on at uh OpenAI when you co-created chat GPT? >> Uh much deeper understanding of just the physical world around us. Um >> as measured by Eval's capability to uh engineer new systems, predict uh that it's going to be stable, um be able to predict what is it that you made. Um so like maybe an example is in order to discover a new material, you need to be able to synthesize it, you need to be able to make it and the prediction of that can be really challenging. So that's a a huge focus for ourselves. Um, and I think there's just like this basic premise that machine learning models are

good on the data you train them on on the tasks you train them to do. Uh, but absent this data, there's some generalization, but it's not infinite. Uh, we don't think that you can just start thinking your way to a new roomtemp superconductor by reading a textbook, shutting it, and thinking super hard. Uh, you have to make contact with reality. You have to carry out experiments. Um and so basically our systems have through this new data through our computational predictions through the data produced from our labs just have a much deeper understanding of how to do these types of things. Um so we're not really spending much time thinking about how do we improve coding like that's continuing to improve uh through the existing efforts but really focusing our efforts towards you know the construction of the physical world.

>> Okay. Should we transition to questions? >> Sure that sounds great. The question is, how do you guys think about combining different types of models to push the frontier of scientific discovery >> or like optimization approaches? >> Yeah. Yeah. >> Yeah. So, great question. >> Thank you for keeping me on. >> So, there's a trend where a lot of physicists move into other fields. One of the fields they've been moving into a lot is machine learning. And uh a common thing is the physicists will first fall in love with Beijian optimization because there's more theory in it and there's a sense of um uncertainty. Um but like one thing to keep in mind is if your machine learning model has generalization problems which every model will have right every model will only be trained on a subset of the universe. um it's uncertainty about uncertainty will be even bigger like generalization issues for point prediction is smaller than generalization issues for uncertainty so what we see practically I think is basian optimization is probably not that useful uh on the other hand active

learning is the opposite I think so if you're working on an academic data set like you take one academic data set you do uniform split train and test uh active learning won't help so you know when we were doing deep learning research back at Google there were a lot of people trying active learning to improve imageet accuracy and it would never work. Um, so then that gives people the false impression that active learning doesn't work. But in reality, I think it's the opposite. In real life, like something like imagine you're training Whimo. You have to do active learning because Whimo is good at some things to a point and then not other things and you have to now kind of push into the other things. So this is very relevant to us because when we do science, our models understand some of the science really well and some of the science not at all. And active learning means we can push into the part we don't know yet slowly and then expand generalization bit by bit. So we do active learning all the time like every day we're running new experiments and usually those samples are could be considered active learning because they're are predicted by our existing model for the future. Um yeah that's

kind of how I view maybe the different approaches. you want to add? >> Yeah, I think like on LM side, we use standard optimization techniques. Um, there's a lot of carryover in that and I think we finding those to be effective. But I think big plus one to this notion of like active learning and effectively what you're doing every day is pushing the frontier of like what we understand about these systems. And it kind of goes back to my comment earlier where it's insufficient just to read a textbook, close it, and think really hard. you have to carry these things out. And as part of doing our discovery process, we intend to produce things and we routinely make the thing that we intended to make, but sometimes we just find anomalies and we have to track those down. Um, and so that's a really fun part of the scientific process and we're like, hey, this peak is just not accounted for. Um, and that material doesn't exist. So it's like of course you and then that provides data for the

next version of the system. So >> it's very core. >> The question is what's the difference between an AI scientist and an AI agent? >> So it's a model that is like a large language model. It's uh thinking it's orchestrating tool calls. The tools might be um standard tools or the tools might be other neural nets that we train. So the a neural net's invoking other neural nets. Uh but you can just kind of map it to the the same thing. question is what is the definition of new material discovery at periodic >> that's a great question uh so there are different dimensions to it right one dimension could be the structure so uh we like if you look at ICST today there are about 250,000 crystals in organic um and they have a certain subset of prototypes and you know very rarely scientists find new structures in the sense that it's a crystal structure that has never been seen before uh so that could be a discovery another one could be a material with properties that didn't exist before. So for example,

today the ambient pressure superc conductivity state-of-the-art is something like 133 Kelvin. Um if you find something at 160 Kelvin, that could also be um discovery. We also really care about discovery of synthesis recipes. So even if a material has been known before, if we can make it more efficiently or make it with slightly optimized properties, we're really also happy. Um it's just about you know pushing the boundary of what humans can do for organizing the atoms around us like you know like since since you're students here maybe I can uh chat with you about this a bit like material science is often not very exciting but it's very hard to understand because it's basically physics of atoms that we interact with. Um so I think it's very exciting if you want to if you want to try it. Um and M's discovery basically in my mind is our control over how we set up the atoms around us. Um, which is obviously very exciting, right?

>> Your question was what led us to have conviction in >> Yeah. Like what? Yeah. Why do you have conviction in period? I think >> how about that? Can we abstract to that? Is that the question? These guys I mean you guys can tell like look I I I think you've had a chance to see over the course of the class, right? Um just a really wide area of leaders, right? whether it's people at the where's Jensen at the chip level or um Scott Nolan at the energy stack part of the stack and so on and so forth. Um, I just think, you know, for a long time people have talked about AI actually being useful in in the physical world and uh I I met with a lot of teams over the last few years, especially when I was at A16Z, who were making attempts at that. Um, but that just the progress just wasn't showing up as much as the

marketing was making it seem like. And so I was pretty frustrated with that. And then uh I had a chance to actually after last year's class um started spending some time at the applied physics group on campus um the connect working on eval models on their ability to reason about condensed matter physics data and you know thank you ZX for the opportunity for that um and we came up with uh a draft paper um for submission to Nurips that kind like benchmarked a bunch of last year's generation frontier models and they were just terrible.

Um I think then Jason Quan at OpenAI who I sent the paper to Jason is the chief strategy offici said hey would you know these results are kind of terrible. Um I just want to make sure we're not making any mistakes in the methodology like who would be a good who could give us peer review on this paper cuz that's the scientific tradition right you like measure stuff if it looks off get some peer review then we got introduced that way and I think separately because Jason was like you should definitely talk to Lee and Dors they've been thinking about the same thing and I think you you had just left open doors you were wind like rolling off deep mind and there was just a meeting of the minds and it just became clear to me that um It it was I don't think it was very it is one of those things where the opportunity was quite clear. The need for the world to have make more progress on this domain was like the the the value creation is extraordinary, right?

Like we humanity is generally the history of humanity can be separated into material errors, right? Stone age, bronze age, copper age, silicon age. Like how do we get to flying cars? I mean really as a civilization one of the bottlenecks is materials. Um and I think uh we had kind of converged on a similar theory of that opportunity and everybody knew it it was super valuable but somebody had to just go do it and the guys knew how to do it given their experience at Deepmind and OpenAI and so I don't I think because like from from me when we first met to like term sheet and starting work together was four days five days >> fast yeah >> I don't know how yeah that's that that was my answer but you guys should you know talk about the feel free to answer.

>> Yeah, I mean well I mean I can't answer for why you're excited about it, but I mean I think one reason why I'm excited and I think you know both of us is just the ability to engineer the world is just so profound. Um so much of the world has kind of been accidentally discovered or just you know trial and error and we see just how quickly the digital world is changing. But if we can make these things smarter about our reality, um we think that's kind of like what pulls forward this like sci-fi future. And yeah, I think we're we're very optimistic on the course of the technology, but also kind of cognizant that you need to make this this leap into the physical world. It's not going to come just from like, you know, again, reading a textbook. You need to make that connection. Um and so I think the implications of the system is just really profound across so many things.

That's that's what gets us excited. >> Actually, can I ask you a question? >> Yes. >> Are the students feeling anxiety over like their education and career given what's happening with LMS and AGI? >> Yeah. >> Raise hands. >> Okay. Wow. The reason I asked is um I recently gave a physics call next door and I feel like the biggest concern I heard from physicists were like what are we doing here if AGI is going to do all physics? Like in my mind this is kind of confusing to me because the LMS are out there but there are only few things they really revolutionize. Um like in my field I would say the thing they revolutionize is force fields. You know we approximate quantum mechanics with machine learning these days and graph neural networks have done a great job.

Whimo is incredible. You know Tesla they can do self-driving. uh translation coding seems really good but there's so much more to improve and like you know if you're interested in something just use lens as an excuse and go improve it um >> so if you were in the class today and you were working on your senior project what would you work on or your final project sorry >> ah so when I was finishing my undergrad I worked on using machine learning to design analog circuits so I would do that um like whatever you know the students really passionate about interested in it's probably not revolutionized by LMS it. Somebody has to do it. The LMS won't do it by themselves. So, um I think it's a very exciting time, you know, like golden eras in human history is usually short, but it's really good to be in the golden era because that's when most impact happens. So, this is this is, you know, your your chance to be part of it. How do you go about solving problems and predicting automistic properties? Do you use uh tools like density functional theory? And how do you for example model some natalysis? Um this is a technical

question so I can answer it. >> Please go ahead. Um so you know I should say I feel like um simulation tools are good for some things and not good for other things. Um if you look at the theory of density functional theory it's clearly good for ground state properties. It's literally the con Honenburg theorem. So we find it to be really good at predicting um formation enthalpy of ground state but we don't think it's good for predicting band gaps or excited states. Um catalysis is really hard because catalysis requires you to know the automistic structure and we don't actually know what's going on.

It's very messy. There are steps like atoms form steps. There are defects. Sometimes individual defect makes a difference that you can't model. So we haven't been modeling catalysis as much. Maybe that's a more empirical direction. Um we have been benefiting so as I mentioned earlier machine learning really revolutionized force fields. So I don't know if you know this but I think one of Einstein's first papers was could be considered a force field paper. He was I think studying empirical methods to approximate the interaction between like a surface and a um molecule kind of thing. Um and force fields have de developed tremendously lots of big improvements from Stanford. Uh but when machine learning came about it really changed how we do force fields and in our company we have really good machine learning expertise not just in LLNS but also in graph neural networks. So we have come up with new architectures, new capabilities but it's still I mean it still cannot do everything uh and can't do catalysis. So yeah, you want to start?

>> Sure. I mean yes there's definitely chicken and egg problem but taking a step back that's not just a problem for our company or for machine learning right the chicken and egg problem of scientific discovery is applies to everyone. I mean you might even think if there's an alien civilization somewhere doing science they'll also have the same chicken and egg problem. The chicken and egg problem is the science we know we understand well. The science we want to discover we don't yet understand well. So you always have to iterate towards it. That's why the earlier question about active learning was a good one because the only way you can iterate towards it is basically do something like active learning. Um there are some data sets that's like imageet that's helpful to us for example force field data sets. So you might have seen um Meta open source this UMA data set that's like 100 million uh density functional theory calculations. So it's a bit like imagenet where you can train on it and the model you train is useful for other tasks. Um but in general I feel like there'll never be this comprehensive data set for science because if there was that would stop being science there would be like

textbook that would be education and then you'd immediately train on all that data and try to discover new science. And this is partly why Lim and I are so excited about this company because there's no end to it. You know, if you automate accountants, at some point there's only so much accounting to do. But if you can automate science, there's no end to it. You can keep discovering more and more interesting science. You can keep developing better and better technology. So it's nice to be in a field where there's no end. >> Yeah. And maybe some more comments, too. Um I think the sample efficiency of these algorithms is is quite high. Uh so we're able to really quickly get going in some chemical spaces or some search spaces with a relatively limited amount of data. And from an ML perspective, one of the core things we think a lot about at periodic is how do we push the frontier of sample efficiency especially in like reinforcement learning. Um so again kind of going back to how a lot of frontier labs uh create models, you typically have uh the ability to do uh for your policy many many rollouts. So

you're asking a math question. You could do a thousand rollouts, a million rollouts. You can kind of scale that really quickly. Whereas in the physical world, it's not plausible to just expand that um arbitrarily high. Uh we're scaling it up a lot at periodic, but we can't just like increase it by a factor of 10 or 100 on a whim. Uh so anyways, we're we're focused a lot on sample efficiency here and how to make better use of the data. And a big part of that is like model based reinforcement learning. And so I think we've been happy with the sample efficiency so far. The question was what are some open problems in AI for science that uh people in academic lab should focus on and the other question was going back to your 2022 paper uh you had this active learning pipeline for discovering zero kelvin stable materials. Are you still using these algorithms? Good. Um, so I mean one thing we really care about is being able to intentionally synthesize materials, you know, like so if you if you go into a lab, I think we're really good at making materials if they've been

made before. We're also pretty good at making materials if something similar has been made before. But if I bring you a completely new material, it's actually pretty hard to figure out how to make it. And it's a lot of trial and error. It's a lot of like just brute force trying things. uh if if I were in a lab today, I would really try to figure out how to make the synthesis approach more intentional. And of course, this is very different for different fields. We're focusing on one kind of inorganic crystal space, but um like I'm sure this is also a case in different fields. I heard in biology some things are easy to synite, some things aren't. Uh I'm sure that's but the reason I think since this kind of sounds boring, but it's actually extremely important is that's how we build the things around us. like once we can get good at making things intentionally now we can construct an amazing technology. Um in terms of whether we still use those tools I mean we still use active learning we still definitely use density functional theory. Um one point of that paper was if you can make your zero Kelvin convex hole bigger it will make it will give better synizability predictions and

that's definitely correct. So you can actually observe that the more densely sampled your convex hole is is sorry very technical again the the better your predictions will be. So we actually use the biggest possible convex hole uh including our work other work that's publicly available. >> Yeah. And I'd say maybe answering a different uh abstraction uh I think making sense of noisy data. So sort of like decision-m under uncertainty uh consistency of results.

So um you know the internet's uh full of results, some correct, some not. Uh and so I think like a system that can be more effective uh at kind of parsing these things and saying at least like flagging like okay these pieces of evidence are consistent, this is inconsistent. Um I think those are useful technologies. Sample efficiency again um another really key aspect. So I think these are like some things that would apply for many different scientific domains. >> Yes. Something else that like Liam's answer reminded me we really care about and I think others can also improve on is automating characterization analysis.

So you know in every scientific lab there's a lot of different characterization instruments and they usually uh give you the measurements in a way that's not immediately interpretable by a human and you have to spend a lot of time analyzing those results. But we've seen that you can actually make tremendous progress if you give an LLM the kind of tools that human scientists use and then automatically analyze those results. Yeah. I mean maybe even like a frontier problem of how do you come up with a good hypothesis. Um so it's it's quite easy to grade whether or not you got that map answer correct or whether your code compiles. uh but correct hypo or like a you know interesting hypothesis um I think is really challenging and um that's I think really a frontier for this field. Um another aspect too is we often get the sense that the knowledge is in the model. It's like it's in the weight somewhere but it doesn't really

um surface it at the right times. Um, so we would hope for some intelligent reasoning, some use of that knowledge on a scientific problem and it won't do it necessarily. Uh, but if you ask it directly about all those different skills or um, pieces of knowledge, it does have it. You can reliably surface it. And so these models aren't really combining the information in uh, as intelligent a way as a human had all of that in their brain. So I think there's a lot of deep work to be done there. you can basically timestamp the universe.

So, okay, like here's the the set of um known uh like this is the known data we have experimentally computationally at some time stamp t and uh produce a world modeling task where you're trying to then predict outcomes of future things. Uh so again AI scientists AI agent all the same here that LLM that agent can then use some of these computational tools in the service of predicting these things. Um so that's one example where the agent is invoking these things in order to predict uh the future things.

So like the world model is not just in uh weights it's not just like inference time but it's using these tools in order to do that. The question is uh you guys are tackling a lot of difficult problems. How do you stay motivated when things aren't working? Um I think it's I mean I think there's like a huge amount of internal motivation. So if we had tied all of our motivation to some like extrinsic goal of like you know the roomtemp superconductor then we would have like lost like motivation because we haven't discovered the room temp superconductor yet. Um we don't know if it's physically possible. We we hope to be, but um if it if it's physically possible, we're going to aim for it. Um but I think there's very clear progression that you can track. So um you can say, are we making progress uh technically on the different pieces we want to see and is that consistently getting better? And you know like again this whole end to end discovery loop is

incredibly difficult. But we haven't seen uh any very compelling uh instance of it yet. Um we hope to soon. But there's very clear attribution to all the little pieces and are we making progress on those little things and that gets me Doge and the team very excited and also you know like I feel like we have this responsibility to take this LLM technology the AI technology and lead it to something positive and um like I think part of what motivates us is this is one path of making these technologies more positive been useful for humanity. There are many other paths but uh yeah there's a lot to do and it's very motivating to try to work on like a very revolutionary technology and try to steer it towards something positive. I can't think of like something more motivating than that. So >> the question is around uh barriers to um doing work in the physical world um and

are those barriers expected to decrease over time and yeah so I think the obvious thing is like you have to build the thing that's difficult um I would say that our expectation would be the barriers should lower um as AI systems and as robotic systems get better we hope that the creation of new autonomous labs or building new infrastructure gets easier and easier. So hopefully the labs we're putting together now are among some of the hardest today. Uh yeah, so that how would that insert it?

I think also you can pick your domain wisely. So if you're saying like okay actually I'm not going to deal with material science. I want to go one energy scale deeper that Doge was talking about. I'm going to revolutionize high energy physics. Um in that case you would be uh beholden to start constructing particle accelerators and that's a much more difficult uh experimental loop. Uh these are tens of billions of dollars takes thousands of people. So whereas there's like other areas in the physical domain that uh can be much faster to iterate on lower capital requirements as kind of a really good test bed for kind of making the connection of AI into the physical world. Uh also I feel like the the the bar isn't too high, you know? So what is the state-ofthe-art in frontier labs right now, right? Like math, combinatorics, I think maybe they'll go into like very theoretical physics. So even if you do slightly more physical things like I don't know, grab a camera and like play with this output or like

point a telescope in this like there's the the bar is so low in terms of how physical and relevant you have to make AI to make progress. you probably can just be slightly above it. And then there's like what we're doing which is like a big investment into material science and then there's what Liam is saying with like particle accelerators or something. But yeah, I think the bar is pretty low right now. But it may not stay like that. In a in a few years I think a lot of young excited people will realize LLM can do way more than coding and then uh it'll be very fun. Yeah. question is around other kind of areas uh where there's a high degree of verifiability that might be really exciting for these types of systems as well as uh areas that periodic might be interested in in the future. Um I think predicting asset prices is interesting.

So finance um I think that's you know highly verifiable. Uh can you predict the future state of you know this stock or this bond uh this yield? Uh so I think there's interesting things to be done there. Um from a technical perspective uh there's like latency considerations. Um and that is something that periodic is not interested >> the asset prices or the latency. >> No no latency. Absolutely. Okay. Good. Uh but yeah pivot into finance. I don't think so.

>> And one one more thing is um you know currently these verifiable rewards aren't just verifiable but they also have full context present to the model. like if you're doing coding right the whole code base the compiler um but in real life most things aren't like that like when we do experiments one thing we notice is not all the context can be recorded um this is my suspicion is this is correct and true for almost everything in the world where you can never uh record all the aspects of what happened and then the question is how can you improve these models in that uncertainty um that's very exciting and it doesn't just apply to physics right it applies to everything including asset prices.

Yeah. >> Thanks, guys. Yeah.