Machine Learning

A Day in the Half Life is a podcast from Lawrence Berkeley National Laboratory (Berkeley Lab) about the incredible and often unexpected ways that science evolves over time, as told by the researchers who led it into its current state and those who are going to bring it into the future.

In our very first episode, we discuss machine learning. First developed about 80 years ago, machine learning (ML) is a type of artificial intelligence centered on programs – called algorithms – that can teach themselves different ways of processing data after they are trained on sample datasets.

In the early days of ML, the technology was used for simple tasks such as voice recognition or identifying a specific type of object in images, and was only found in high-end academic, government, or military devices. But now, advanced ML algorithms are everywhere, powering everything from our cars to our voice assistants to the ads appearing on our news feeds. 

And, in addition to making everyday life easier, ML algorithms are beginning to improve and expedite scientific and medical research in truly dramatic ways. In fact, the range of potential applications is so huge that the question has shifted from “Can we use machine learning to solve this?” to “Do we understand the way these algorithms work well enough to feel comfortable using ML for this?” 

Our two ML expert guests are:

John Dagdelen, a materials science graduate student researcher at Berkeley Lab and UC Berkeley. John is part of several scientific teams using ML to discover new materials and material properties, as well as using ML to make discoveries in COVID-19 research.

Prabhat, the former leader of the Data and Analytics Services group at NERSC, Berkeley Lab’s world-renown supercomputing center. Prabhat has been using and developing ML for decades, including for use in climate research. He is now at Microsoft.

Transcript

Aliyah:

Welcome to a Day in the Half Life, a podcast about the incredible and often unexpected ways that science evolves as told by the researchers who led it into its current state and those who are going to bring it into the future. Why focus on this and launch yet another science podcast? Well, when we look around us at the latest technology and medicines available, or read news about advances in our understanding of everything from black holes to gut bacteria, we rarely stop and think about the fact that these amazing things all began with fundamental discoveries in a lab. And those Eureka moments grew into what we see around us today because of curious people who never stopped saying what if, as you can imagine, those people have some stories to tell, and we have a lot of those people. I'm Aliyah Kovner, a science reporter at Lawrence Berkeley National Laboratory, an institution dedicated to scientific research and technology development that has potential to address global challenges. Today on our very first episode, we're going to talk about machine learning because it's a perfect example of how a field or concept can branch out and grow far beyond our imagination.

Aliyah:

First developed about 80 years ago, the technology has really exploded in recent years and is now a crucial part of devices and services that most of us rely on. Chances are you're right next to one of those devices, or have one in your pocket, at this moment. Our experts today are two scientists who currently use machine learning in their research: Prabhat: who leads the Data and Analytics Services group at NERSC, Berkeley Lab's super-computing center, has been part of machine learning research for decades, and is well connected in the field. John Dagdelen is fresher to the machine learning realm. He is a graduate student researcher in materials science at Berkeley Lab and UC Berkeley.

Prabhat:

All right. Hi, this is Prabhat

Aliyah:

Hi! Nice to talk to you. Hey John, how are you?

John:

Great, thank you.

Aliyah :

Thank you both so much for being here. So, Prabhat, you've been in this field and using machine learning as a tool for a long time now, but what did machine learning start out as?

Prabhat (02:17):

That's a good question. And I think maybe you want to say at the very outset that there are a few terms that are used interchangeably in this area. So there's artificial intelligence (AI), machine learning, deep learning, and statistics. And I think maybe early on, it's useful to just understand how they relate to each other. So just very broadly, the goal of creating machines that are intelligent is called AI. And that's the Holy grail problem that computer scientists have been now working on for a long time. Within AI, there are a class of techniques called machine learning, and I guess one can think of machine learning as really enabling machines or computers or software, to learn by themselves, as opposed to us programming them. But within machine learning there is a class of methods called deep learning.

Prabhat:

And frankly, perhaps the reason we are having this podcast today is because of the success, the recent success of deep learning. And finally the fundamentals of all of these classes of techniques is really all in statistics. You know, we've had statistics with us for a few hundred years now. Statistical approaches were really quite powerful when we had very little data and we really needed models that we understood well. And I think three things happened in the last 40, 50 years that, you know, our data sets have become larger and larger. Our machines, thanks to Moore's Law, have become exponentially more powerful. So we have big compute available to us on our laptops or desktops and our favorites of computing centers. And then finally, these algorithms, powered by both faster machines and access to large data sets, these algorithms, in particular deep learning… We've come to appreciate that these deep networks are certainly very, very, capable.

John (04:07):

I think that's a pretty fantastic overview. I often get the question, what's the difference between machine learning and AI? And often we, as practitioners, will sometimes use the terms interchangeably when we're not speaking very carefully. But it is important to note that for a system to be intelligent, it has to have knowledge somehow, right? And if it, if it doesn't know anything, it can't make decisions. So in the past, we tried just explicitly programming in this knowledge, but it's much more efficient to, to build algorithms that can teach themselves from data. So that's really the goal.

Aliyah:

So I've read that these self-teaching deep learning networks were actually modeled on how the human brain works and how we learn. Is that correct?

Prabhat:

That's right. So there's a, I guess, a romantic notion that, in some ways what we have achieved with deep learning is how the human brain works. Computational neuroscientists and computer scientists have tried to approximate what, what a neuron does, uh, in simple terms. And, uh, those concepts certainly underlie what neural networks are able to do today. But, uh, I would say that, and I think this is also acknowledged across the board that really aren't under any romantic notion again of, oh, you know, what a neural network does is exactly what the brain does. I think far from it. It's a long ways from what human intelligence is all about.

Aliyah :

So what is the most fundamental difference between the two?

Prabhat (05:38):

So I guess we can maybe draw some distinction between how a deep network neuron or a deep network is different from a human brain in that if you look at how a child learns to distinguish between apples and oranges and the, you know, fluffy toy in a chair -- you can give only one example to a child and by rotating the object, and playing with it, and touching, and even just pushing it around, a child does build a fairly powerful mental model of what an apple is. And tomorrow, if you perhaps show the child a different apple, that might be a slightly different color or size, chances are that the child is going to be able to generalize and infer that it's also an apple. Deep networks, right now, don't quite have that, that, that capability. So essentially you need to literally provide thousands, tens of thousands of images, of different kinds of apples and oranges and, uh, hope that the network is going to do the right thing.

John:

Yeah. I think also one of the key differences that has made machine learning so successful recently is that you can scale it. Like, um, you can take a model that you've trained to do something and then put it on a thousand computers in the cloud and then have them all running the same thing, um, which you can't do with, uh, with a living person. So, one of the, the goals is to scale the intelligence in a way that we can take advantage of the automation that computers enable, but the intelligence that, you know, we observe in people.

John:

And also I think, uh, there's some really interesting ways that that studies, uh, on neuroscience have influenced machine learning. For example, some of the early research onto how animals see, the visual processing that goes on in their brains, um, scientists were trying to figure out where in the brain is vision concentrated. If you show a cat, an image of a mouse, where in its brain is lighting up, right? So they actually put electrodes down into the brains of cats and then showed them images. And, were trying to identify what parts of their brain were lighting up to identify where's vision, right? This was in the early days, our understanding of the brain. And accidentally, they found that it was distributed; like vision wasn't concentrated in any one specific neuron or something like that. It was kind of distributed around certain areas, and different parts of lit up when you showed them images. But they were changing the slides between the images they were showing these cats, and they found that, uh, they had an electrode in one very specific portion of the brain. And when they were changing the slide and the, the vertical line between the slide slid past the cat's vision and lit up a part of their brain.

John (08:20):

And so from that, they discovered that there were these, these sort of visual filters that were going on inside of the brain, that process things like vertical edges and differences between light and darkness in a piece of a picture. And so that inspired a lot of the computer vision research, where they started building these filters into their models, using them as visual features, and these models were able to perform this task much better when they were seeing in a way that was more similar to how we understood the brain in animals was seeing.

Aliyah:

Wow, that's really cool. I know that in the history of different applications of machine learning, it's been used for image and pattern recognition for a really long time -- that was one of the first things that scientists said, oh, we can apply this, and it will make our lives way easier. Are there any ways in which it processes images, you know, that it [ML algorithms] taught itself to do that are completely different than the way we would process an image?

John:

That's interesting. I'm, I'm not very up to date on the neuroscience of vision, um, so I don't, I don't quite know if we know enough about vision to make that statement, um, because there's a lot of complicated stuff going on in our brains. But I think that there are certainly examples where machine learning algorithms have learned how to do something in a way that is not conventional and not how we would think the human brain would solve that [problem]. One of the examples was AlphaGo and AlphaGo Zero. Um, so AlphaGo was trained on Go games played by humans. Um, but AlphaGo zero only played itself. It never played, it never saw any information from a human game. And the strategies that it learned, um, were very alien to people who are experienced Go players. They actually described the experience of playing against it, like playing an alien, because it would do things that didn't make any sense. And they've never seen anything like it before. And they thought, "wow, this is a really terrible move. "This algorithm is pretty bad." And then, you know, a few dozen moves later they'd realize it was genius and so strange that it would go that direction, but they see the inspiration there. And so then they started playing these algorithms to learn how to improve their game and dive into new areas that they hadn't explored before, because then it gave them an advantage over other human players. They learned how to play in this alien way.

John (10:35):

So, I think there's a lot of examples where machine learning algorithms are developing, they're developing understandings or ways of looking at data that isn't natural to humans. But often, a lot of the hardest problems to solve with machine learning are the ones that are just trivial for humans. So we can easily put words together as speakers of English. English is a dynamic language, right, and so now people are using words like "cancel culture." Between, you know, five years ago that that term wouldn't have made a lot of sense to people, but having been in the world we live in right now, we just understand what that term means. Having not been explicitly told what it means before. Right. So that's something that's extremely difficult for machine learning algorithms. Uh, as soon as you change the context that words are being used in, if they didn't see that in their training, they won't understand that, that new context. And so that is something that is, um, is always a challenge... Is trying to do the easy things actually.

Aliyah:

So, you know, we've, we've talked about a couple of examples now of applications of machine learning that are really cool. And, you know, maybe you encounter them in day-to-day life, like Google image recognition, or, you know, a virtual opponent for a game. But I just wanted to go back a little bit and talk about, you know, how did it get to this point? Like, what were the first applications of machine that a consumer or non-scientist might've encountered and how long ago were those kind of breaking into the world?

Prabhat (12:00):

Uh, you know, pretty much since the, the 80s, computer vision researchers have really taken the lead on putting out challenges for the community to solve. And those challenges have revolved around, uh, putting up images of the, of the real world and then your task, uh, the machine learning algorithms' task is to find objects in those images. "Is there a cat in this image? Where is the cat?" So there are, there have been really a host of such datasets and problems available in the computer vision community. Pretty much since the eighties, I would say. Similarly, if you could think of a robotic system and you propose a complex task to the robot such as, you know, "go and pick up this piece of paper," or "close the door," or "go from point a to point B overcoming some obstacles," uh, those kinds of problems, you know, have been, pitched over the years. They've had more complex versions of these problems around self-driving cars. Um, so to that, I think those kinds of problems, you know, have been looked at by researchers for many years. Now. I think your question was maybe a little bit more on the consumer facing side. So, you know, obviously machine learning, technology being deployed, that you might even not recognize, behind the scenes. Certainly, uh, I think of Google, and what you do when you enter words in the search bar. You know Google has to index and search billions sometimes even, uh, tens of billions, hundreds of billions of documents or files. So to find which of those top 10 documents are the most relevant to you, uh, that's not an easy task. There are machine learning algorithms behind the scenes that solve that problem for you, you know, even your iPhone.

Prabhat:

I mean, if you look at all of the kinds of, uh, wizardry really that's there and built into the electronics of a device, you have a camera that automatically focuses, automatically determines what the aperture needs to be, whether I should use the flash or not, who I should be focused on, and so forth. So there's a lot of image processing. That's powered by machine learning in your iPhone camera. Uh, iPhone's feature speech recognition now. So really I think more and more really, I think that the eventual success of machine learning and AI technologies is going to be when you don't even realize it's there, it's just behind the scenes and it does the right thing. It does the most intuitive thing that you would expect it to do. So I think we are starting to see that already. We're starting to see that in Apple's products. We are starting to see that in the cloud providers like Google and Microsoft.

John (14:36):

Yeah, there's a lot of public research institutions that are investing in super computing resources that go beyond the typical use cases of doing simulations on climate or quantum chemistry calculations, that kind of thing. They're starting to push into building the infrastructure to be able to do this large scale machine learning research. Um, so in this, in those cases, you need hundreds of GPU's all working synchronously to process data and learn from it.

Aliyah:

What's a GPU? Can you define that?

John:

Oh yeah, sure. So GPU is, uh, most people, uh, if they've ever come across a GPU, it was in a personal computer used for playing video games. Um, and so it means graphical processing unit -- you may have heard them being called video cards. So that's roughly how it started. Um, I sorta think of GPUs as like the tunnels under Disneyland that make the whole thing, all the magic work. But roughly, a GPU is a piece of specialized computing hardware that can do a certain type of calculation really fast. And that calculation is just processing these arrays of numbers. Um, and then some machine learning researchers, uh, I think this was in the early 2000s probably, Prabhat you can correct me if I'm wrong, but, uh, I think it was Andrew Ing's group, right.

Prabhat:

That's right.

John (15:51):

They started showing that, I think this was at Stanford at the time, um, they showed that you could run these deep learning models on these computer game graphics cards much faster than on a conventional, you know, top of the line CPU, which is just a central processing unit, or just the general processor that you have in your computer. And then that basically changed the game for how machine learning research and machine learning training is done. Yeah.

Aliyah:

Pun intended. [Laughing]

John:

Haha, yeah. Exactly. [Laughing]

Aliyah (16:17):

So, yeah, I mean, it's just, it's interesting to hear about ways that it's creeping in more and more into public life, but, you know, it's also coming into so many more kinds of science. Just at Berkeley lab, now there's, there are machine learning people who are in the bio-sciences and then there's scientists like yourself, John, who are doing it for materials science. So what are some cool examples that have developed recently?

Prabhat:

Sure. You know, I start to think of science at different scales. So, we start at the largest scale of cosmology, um, people who are looking at, um, replacing expansive cosmological simulations of the universe with, uh, deep learning part surrogate models. Uh, I personally worked on a project that looked at predicting cosmological constants from large-scale simulations, so that can be done more accurately with deep learning. Now in astronomy, when we are looking at telescopes that's, you know, scan the nighttime sky many times over, 365 days a year, uh, deep learning algorithms have more potential for more accurately finding and localizing a star or a galaxy or a quasar or a supernova. If you look at the earth sciences, I personally worked on [a project] looking at earth simulation data, so climate simulation data and, finding extreme weather patterns and quantifying how climate change might affect how extreme weather impacts us, where we live. Going down a few scales, if you look at genomics, we are starting to see, uh, machine learning powered algorithms, uh, predict a genomic sequence more accurately. And in some cases predict the function of a genome; essentially going from a genome to function.

Prabhat (18:00):

I think that's [use of ML algorithms] starting to happen if you go all the way down to subatomic physics and you look at exquisite instruments, like the large Hadron Collider, essentially being able to remove signal from noise. So, removing uninteresting particles from new and interesting particles, that's a task that now deep-learning powered algorithms seem to be really good at attacking. So really I think across all scales in science, we are starting to see people systematically exploring deep learning methods. I did leave out material science for John's comment, on. So, please John.

John:

Yeah! So, there's a lot of really interesting applications of machine learning in material science. Um, to preface what is material science? So material science is the study of the stuff that stuff is made out of. So engineers are infinitely creative, they will, they will design you an elevator to the moon. The problem is that we don't have the materials that can withstand the forces or have the right properties to make those designs work. Um, and so material science is the fight against the laws of physics to try to make things stronger, make them more conductive, make them stretch farther without breaking. All of these, all these really important properties that, that then go into enabling new designs for, for things like iPhones or that make your battery last two days instead of one day, and enable your car to drive a thousand miles with a single charge. 

All of this will be enabled by materials advances. So, traditionally materials development has been a pretty slow process. It's very difficult to, um, find new materials that have the properties that we're interested in. So that's an important application of machine learning right now is can we predict the properties of materials instead of the lengthy process that it takes to actually test them in a laboratory. So then instead of testing -- Thomas Edison style --10 materials, every month, you can test 10,000 in an hour and then go and look at the most promising ones in the lab. So, there's a lot of ways of doing that. I come from a background of computational materials science, where we do quantum mechanical simulations to, uh, it's, it's a prediction of the properties, but based in the physics, the machine learning is kind of another level on top of that, where we will use those simulations that are faster than experiment to train these machine learning models, which are faster than those simulations.

John (20:26):

So it's kind of a, uh, an accelerator in that way. Another thing that we're interested in doing with machine learning is predicting the, the materials themselves. So, so coming up with new ways of mixing the elements, um, in different compositions and structures that imbue them with the properties we're interested in, we also are using it for a lot of automation tasks. This is kind of where my research lives: [in asking] "can we take the huge body of existing literature and gain insights from that data that we may have missed?" So that's, that's roughly what our most recent research has been doing, is we've been applying these, uh, these natural language processing machine learning models. So these are, these are machine learning models that deal with human language. Uh, we've been feeding them scientific papers and then asking them for guidance on what we should look at next, or what might be promising materials. And we found that it's actually a lot more successful than we expected, and we were pleasantly surprised that it's able to have really big impacts on our research.

Aliyah:

So in an example is like this, where you're using it for materials, either discovering new materials or kind of helping you find connections between discoveries that have been made, what are you putting into these, into these algorithms so that you get something helpful back from it?

John:

Yeah, so that's, that's actually the key problem, um, for developing machine learning algorithms for science. These algorithms, they, they live in a land of numbers. So machine learning models typically, uh, their input is going to be a vector, which is a, just a stack of numbers. You can, you can think of it as sort of the list of all the different features you might be interested in. Um, and then the model takes in that and then processes it and gives you an output, which is also a number or some kind or a collection of numbers. So translating these, these things that exist in the real world, into these mathematical representations that then machine learning models can deal with is extremely difficult. And that is where the domain expertise comes in. That's why it's not machine learning researchers that are coming up with these advances. It's the scientists, because they need to understand what the problem they're trying to solve is. Then they can give it to these deep learning models once it's translated into the language. They're kind of acting like scientific translators. So they're, they're taking things from the real world into this numbers. Right?

Aliyah (22:37):

Interesting. When you are using these deep learning models to process materials science questions, has it ever come up with something where, similar to the example of the Go game strategy, at first glance you've just thought, no, what is this?! This is so bizarre. But then found that it's actually genius. Has that happened yet?

John:

Yes. So that, that has happened actually. Um, so, so specifically for an example, we wanted to, we wanted to discover thermoelectric materials, and thermoelectrics turn heat into electricity. So they're really useful for things like recovering waste heat from a car's exhaust or from, uh, industrial process waste heat. They're used to power the, the Mars Rover: you wrap a piece of nuclear material that decays and produces heat with a thermoelectric, and now you have, you have electricity forever. But [for this project] we were asking our model, "can you just rank the materials that we created these representations for, with how similar they are to the word [the concept] thermoelectric?" And then we would compare those with the list of the known thermoelectric materials that we, uh, that we just know as materials scientists. But we would, we would notice that some materials were coming in that just didn't, they didn't, they weren't thermoelectrics. As in, we had never read of them as a thermoelectric and they were pretty high up in the list, like [ranked by the algorithm] in the two hundreds or so, um, of all materials, right?

John:

The top 200 materials or something like that. So that was pretty high up on the list considering. Um, and so we thought, okay, maybe our model is getting confused. There's probably something wrong with it. Um, but we have ways of investigating whether a material is thermoelectric through simulations, these, these quantum mechanical simulations that I was talking about. Um, and so we did those simulations, but then we found these materials were thermoelectrics! The simulations were saying, these should be thermoelectrics, our model had been telling us that it thinks something is a thermoelectric, even though there's no paper out there that ever said it was,

Aliyah:

Wow! That was just from you doing a process that was meant to improve the model, not even necessarily actively discover new materials?

John:

Yeah. Yeah. We didn't, we didn't start building that model with the intention of predicting new thermoelectric materials, but we quickly discovered that it was capable of doing things like that. And then we started, we, we shifted our research focus to what can we use this model to do that for other things? You know, what materials are could come out of it? How, how well does it do this process? And so that's kind of our, our continuing research directions on those projects. Can we develop models that are even better at that than the original?

Aliyah (25:01):

Nice. So, you know, we've talked a lot about how it has gone into different fields of research, but one thing we haven't gotten to yet, that's so topical and is the reason why this is happening over Zoom, as opposed to in person is, you know, medical events, uh, researching diseases. And machine learning has, for years, been used for things like diagnosing diseases based on medical scans. And now it is being used to help develop knowledge and treatments for COVID-19. John you're working on that. Can you tell us about it?

John:

Yeah. So the, the project that, um, that you're referring to, it's called COVID scholar. You can find it at covidscholar.org. And, uh, we started out as a search engine for the COVID-19 literature. We have a lot of experience in collecting a lot of scientific, uh, text data from different places online and putting it all together for analysis. So we started just scraping every source of COVID-19 papers that we could find. And we started expanding into applying some of the same analysis that we were doing on the materials text data to the COVID-19 literature. And that is, uh, still in its early phases, as far as the natural language processing algorithms go. But we're finding that we can accelerate the research of COVID-19 researchers pretty significantly by one, helping them find what they're looking for and to trying to suggest what they should be looking for, but they may not be aware of. 

So, we're using similar methods that we did in the material science projects that we've, we've done in the past. We're repurposing them for, for COVID-19. Um, so some of this has not been released yet. We're still working out some of the details on how this is going to look for people, but you can, you can already find the search, the search, the search engine, and some of the preliminary beta stuff on the word embeddings on our website. And we also are collaborating with a lot of groups that are doing COVID-19 research, and we're using our infrastructure that we've built to make their work go more smoothly or help influence or inform, um, some of the directions. So one of the collaborations is actually with the new rapid reviews COVID-19, uh, journal, which was started by UC Berkeley and MIT press.

John:

That journal, it's intention is to get peer reviews very quickly for these pre-prints that are coming out so that, um, and make all of this public, um, so that the process of science can just be accelerated a little bit, um, in this area. And so we're, we're providing them with a lot of tools to identify new pre-prints as they come out, ones that may be especially important or especially of interest to a certain types of research. Um, and then they're using our infrastructure to actually do the process of the peer review. Um, we're also collaborating with a group, that just released the first iteration of their, their research called, Knowledge Graph COVID-19 or KG-COVID-19. And that's a project from LBNL, where they are linking different sources of COVID 19 data into a knowledge graph.

So a Knowledge Graph is, is sort of one of those detective walls with the yarn and pictures and everything linking it together. It's that, but at a massive scale, across different types of data. Um, and so the project is also applying machine learning on that data that they've collected and built, um, to identify links between different concepts that may be relevant to COVID-19. So things like "does this gene play a role in the expression of this protein that we think is involved in COVID-19?" those kinds of things. And so we're, from our side going from, can we take this massive amount of text data and process it down in a way that links up with that Knowledge Graph? So then you can take advantage of all this Knowledge Graph data and the data from our text database. So that is gonna, that's, that's a very exciting direction for us. Um, it's, it's in its early stages right now, but we're really excited where it could go.

Aliyah (28:55):

I think that that's such a great example of, you know, where I see machine learning going just because this text-based learning or these text-based learning models, they're kind of doing what human scientists do, just so much more efficiently. Because when humans conduct science, it's all about learning from what other people have discovered and finding connections and building upon it until you can then create something new or find something that was there always, but you didn't get to see it beofre. And so this is doing that for you, but so much more efficiently. And when you have a crisis like, COVID-19, you know, we just need it done fast. And it's not about credit. It's not about who gets there first. It's just about getting there quickly, right?

Aliyah:

So is this gonna kind of tool going to be common for every type of research out there, moving forward?

John:

Yeah, I hope it will be. Um, we're, we've been thinking about how can we generalize this beyond just one domain. We've done it for materials, we're trying to do it for COVID-19. Can we make it a little bit more general? I kind of think of machine learning, uh, these tools as… So I was a big fan of power Rangers as a kid. I don't know if you, if you were, but in power Rangers, the episodes are very formulaic. So these, these group of teenagers would just be, like, doing their life, and then this monster would come and they would fight the monster and defeat it, but then the monster would come back as just a giant building-size monster version of itself. And then they would have to all get in this robot that would fight the monster. And I sort of think of machine learning is becoming like a robot for scientists that you can get in and fight bigger problems than you could tackle by yourself. Um, and often involves the collaboration of a lot of people working together. 

For example, an entire community of people contributing to open source software that makes machine learning frameworks as powerful as they are. Uh, that, that is really the key thing. We're able to take, we're able to leverage so much more than we ever would have been able to alone, um, using these tools. And I think that's, what's going to lead to some pretty significant scientific advancements in the future, especially as science becomes more integrated with, uh, technology. 

So pretty much everybody I'm thinking, you know, I'm gonna make a little bit of a longer-term prediction, but I think scientists are going to have sort of like an AI assistant that they use to, uh, help get potential hypotheses or analyze data, or it's gonna make their lives a lot easier. They're just going to be able to ask machine learning algorithms to do a lot of the things that they were doing by hand or answer questions that are just difficult for people to, to suppose. So for example, um, has anybody done research that's similar to mine before? That's a question that you'd have to pretty much have read every paper to be able to answer that question definitively, but machine learning algorithms can actually do that. And so that's kind of a, that's a really interesting new direction that we're all going in.

Prabhat (31:48):

Yeah. I really liked John's analogy in that if you go back to Berkeley Lab and team science, right? So we are known for team science and I guess the conflation there being that any, any modern endeavor in science comprises of theory, experiments, simulation, and there might be a data-driven component to it. You might need engineers on your teams to support it. But maybe adding on an AI agent that truly is a master at mining literature, or is intimately familiar with [processing] a lot of data and is able to respond with accurate, um, justification. So if you ask it a question and it responds with an answer and it backs, backs up the answer with data. I think would just be a phenomenal addition.

Aliyah:

Yeah. I mean, it definitely seems like this is an area that is really just going to continue grow.

John:

Yeah.

Aliyah (32:41):

So my final question is can current, machine learning models beat the verify you're not a robot test that appears when you're trying to buy something online, can they beat that?

Prabhat:

Yes, definitely [laughs]

Aliyah:

So why does that exist! [Laughs] And how can we build one that is truly robot proof?

Prabhat:

Yeah, you know, I have a friend in Oxford who basically designed a captcha breaker. So, you know, both, entering a right response to a warped text word, or even these, you know, identify which images have crosswalk paths... I mean, those are all easy problems. So really, I think, you know, the question is actually much deeper than that. Now that we are seeing superhuman performance for a range of tasks, I think we as a community, as even humans, I think we are going to be pushed into a corner where we might need to think about, "What are we very good at? What does it mean to be human?" We thought that extremely complex games that involve in ingenuity and creativity, like chess or Go are uniquely human skills. That's not the case anymore. Um, we've seen machine learning powered systems, create pieces of art that look amazing and sound amazing. So I think many of the attributes that we felt were uniquely human, uh, I think machines are becoming better at that. So, certainly I think overcoming you know, what is called the Turing test or some other tests, I think it really is just a matter of time before, a custom machine suitably trained will learn to overcome these challenges.

John:

Yeah.

Prabhat (34:29):

So, uh, yeah, I, I do think, the window of opportunity or rather, the areas in which humans are really, really excellent and machines are not as, as good, I think there are very few remaining problems of that type. So I would say that, yeah, I think, you know, solving a captcha or a log-in problem - those are easy problems.

John:

I think, uh, yeah, that, that points to a good, I think a lot of people that work in, in this field would just say it's a matter of time for any problem that we're posing. Even the things that you wouldn't expect. Um, so it's just -- the more we learn about what the capabilities of these algorithms are basically are going to be approximating a human brain in terms of what its capabilities are. Um, this is probably long-term like hundreds of years, but yeah, eventually there's going to be pretty much no limits on, uh, what, what people can do, and it can't. Um, so that's going to be, that's why it's really important to think of the philosophical questions now, early-on in the process, and understand what is this going to do to society? Certainly, I think it'll make society better. Um, but we have to make sure that the, the interim phase where things are developing is, is still tolerable for everybody to live in.

Prabhat (35:48):

Oh, so, Aliyah, I think I do want to maybe caution here that, uh, you know, because this is a science lab and, um, you know, I think as John mentioned towards the end that this might take some time to pan out. You mentioned a hundred years, you know, I mean, you can change the, the order of magnitude of the estimate -- it could be 10 years, could be 20, but I think that there's probably a reason for caution as well. In that, uh, you know, it's going to take a little bit of time and deliberation before the scientific community truly accepts machine learning as a part of their established toolbox. So if, for example, we are not able to explain what these machine learning systems do. If we are not able to characterize how they fail, then, uh, that, that is actually gonna hold back the adoption of these methods by the science community. So right now, I think, you know, we are in a phase where it seems to be working, but we don't seem to have a handle on the theory yet. So until that happens, I think, people are going to be justifiably cautious,

Aliyah (36:46):

Right. Science is all about, you know, show me your methods, right? Because no one wants to just be told, Oh, this is, this works. And, you know, have the hand-waving because that's not, it also doesn't build trust in what you're, what you're giving to other people and what you're giving to the public. So it's interesting that this, uh, you know, really does bring up so many philosophical questions while solving so many problems. And how do you balance that?

John:

Yeah. Like science is about understanding this vast, mystical, crazy complex world that we live in and distilling that down into understandable truths about the universe, but machine learning is almost the opposite. It just re-projects it back into this super unintelligible, complicated thing. And so that's not going to be, that's not going to be sustainable for the long-term, that's not, that's not what science is all about. So we're, we've gotta solve that problem.

Aliyah (37:37):

Yeah, that's, that's interesting that the theory has to catch up with the use.

Prabhat:

I think that's the point that, you know, I think this analogy is drawn repeatedly by the founders of deep learning that often, uh, experimental science moves fast and shows that this can be done. And the theory takes a little bit of time to catch up. Sometimes there's a gap of a decade, sometimes, you know, a couple of decades. So I think we're in the same place with deep learning that we see -- experimentally, empirically -- that it's working, but the theory is not there. So the question is, you know, do you wait for the theory to catch up? Companies like Google and Facebook and Amazon, they're not going to wait. They're just going to deploy it right away. Uh, but I think, you know, if you take the example of a self-driving car, you know, once in three months, perhaps it's okay, if a pedestrian is hit by a self-driving car, but if you know, every week you see someone getting killed by a self-driving car, then that's not going to be acceptable.

Prabhat:

You can't say at that point that my model has, you know, 10 billion parameters, and I can't explain to you, sorry, DMV, but I can't explain to you what happened. So basically I think legal compliance requirements are coming soon. But for us, you know, in the scientific world where we do care about, uh, underlying mechanisms, theory, um, I think we owe it to the rest of our colleagues. I mean, I've been in this role wherein I've, I've said that, you know, you've used imperfect heuristics [trial and error-based methods] for the last 40 years -- I'm going to show you near-perfect results with deep learning. But these are black box results, and I'm not going to be able to explain why they work so well.

Aliyah (39:13):

So, so the algorithms that are built by these networks, they're, um, I mean, you can look at them, right? But it's just, it's not necessarily easy to interpret?

Prabhat:

Right, that's right.

Aliyah:

Do you think that scientists will be able to kind of decode what's going on in the black box? Or do you think that a lot of it, that a lot of progress will have to come from the scientific community embracing a little bit of uncertainty?

Prabhat:

So there's probably a middle ground where, um, we can almost, in some ways, hardwire or constrain the network to be interpretable. You know, we, our deep learning models are totally science agnostic at this point in time, but obviously there is a body of scientific literature. We know governing laws, we know basic principles and so on and so forth. So you could also bake-in or constrain these networks with this knowledge. So constrain this network from fully general to do anything, to subscribe to some of of the underlying physics or principles that we know that the system subscribes to.

John (40:18):

Yeah, I, I don't know if I can do any better than, than that. I mean, that's, I think that's going to be the direction that we go. It's, it's a term called physics-based machine learning that I've heard thrown around, PBML. Where you like, like, Prabhat said, you constrain your models to have to be physically correct. Um, so they, they're not going to output things that are physically impossible that would do a lot to assuage the fears, I think of scientists, um, when using these tools, um, and then also build in some of this interpretability. Um, I also think that you could, there is there's classes of problems that you can solve without that being a problem. For example, people don't worry if Google is scientifically correct when it produces search results for papers, you just, you can make that judgment for yourself. Once you read the abstracts as they come up in the search results.

Aliyah (41:03):

Right. But if you were trying to test the rate of the universe's expansion and it came up with some number, you want to know why! You want to know how, how it got to that number. [Laughing] Great. Well, thank you both so much.

John:

Thank you.

Prabhat:

Thanks, Aliyah. John. Nice to meet you.

John:

Nice to meet you too.

Aliyah (41:25):

Thanks for listening to the pilot episode of a Day in the Half Life, and I hope you follow us as we bring you more insider science stories.