Transcript for #73 – Andrew Ng: Deep Learning, Education, and Real-World AI

Growing up in Hong Kong Singapore, I started learning to code when I was five or six years old. At that time, I was learning the basic programming language, and it would take these books, and they'll tell you type this program into your computer, so type that program to my computer. And as a result of all that typing, I would get to play these very simple, shoot them up games that I had implemented on my little computer. So I thought it's fascinating as a young kid that I could write this code that's really just copying code from a book into my computer to then play these cool little video games. Another moment for me was when I was a teenager and my father, Chris Doctor, was reading about ex-bit systems and about neural networks. So he got me recently these books and I thought it was really cool. He could write a computer that started to exhibit intelligence. Then I remember doing internship was in high school, this was in Singapore, where I remember doing a lot of photocopying and office assistants. And the highlight of my job was when I got to use this shredder. So the teenager me, remember thinking, boy, this is a lot of photocopying. If only you could write software, build a robot, something to automate this. Maybe I could do something else. So I think a lot of my work since then has centered on the theme of automation. Even the way I think about machine learning today, we're very good at writing learning algorithms that can automate things that people can do. Or even launching the first MOOCs method from online courses that later led to a Coursera, I was trying to automate. what could be automatable in how I was teaching on campus.

Yeah, I felt, you know, teaching a standard teaching machine learning to about 400 students a year at the time. And I found myself filming the exact same video every year, telling the same jokes, the same room. And I thought, why am I doing this? Why would you take last year's video? And then I can spend my time building a deeper relationship with students. So that process of thinking through how to do that, that led to the first first moves that we launched.

You know, teaching online, what not many people know was that a lot of those videos were shot between the hours of 10 p.m. and 3 a.m. A lot of times, launching the first one was outstanding. We were very now so close, but a hundred thousand people had signed up. We just started to write the code and we had not yet actually filmed the videos. So we have a lot of pressure, a hundred thousand people waiting for us to produce a content. So many Friday, Saturdays, I would go out, have dinner my friends, and then I would think, okay, do you want to go home now or do you want to go to the office to film videos? And the start to be able to help 100,000 people potentially learn machine learning. Fortunately, that made me think, okay, I'm going to go to my office, goes to my time-ealer recording studio. I would adjust my logic, where can I adjust my, you know, waggoms have to make sure my little mic was on. And then I would start recording, often until 2 a.m. I think I'm fortunate that it doesn't show that it was recorded that late at night, but it was really inspiring the thought that we could create content to help. So many people learn about machine learning.

I think it's humbling and I wasn't thinking about what I was feeling. I think one thing we've done proud to say we got right from the early days was I taught my whole team back then that the number one priority is to do what's best for the learners, do what's best for students. And so when I went to the recording studio, the only thing on my mind was, what can I say, how can I design my size? Well, I need to draw right to make these concepts as clear as possible for learners. I think you know, I've seen sometimes instructors is tempting to, hey, let's talk about my work, maybe if I teach you about my research, some will cite my papers a couple more times. And I think one thing we got right launched the first few moves and later building closerer was putting a place that bedrock principle of let's just do what's best for learners and figure about everything else. And I think that that is a guiding principle turned out to be really important to the rise of the movement.

I really want to help anyone that had an interest in machine learning to break into fields. And I think sometimes, I think people ask me, hey, why are you spending so much time explaining and grading to send? And my answer was, If I look at what I think to learn the needs and what benefit from, I felt that having that occurred on the standing of the foundations, coming back to the basics, would put them in a better state to then build on a long-term career. So try to consistently make decisions on that principle.

I think the numbers grow over time. I think a lot of those things that maybe it feels like it came out of nowhere, but it's an inside that building it took years. There's not those overnight successes that took years to get there. My first forery into this type of online education was when we were filming my Stanford calls and sticking the videos on YouTube. And then some of the things we uploaded the whole work and so on. But you know, basically the one hour or 15 minute video that we put on YouTube. And then we had four or five other versions of websites that had built, most of which you would never have heard of because they reached small audiences, but that allowed me to iterate, allow my team and me to iterate to learn what the ideas that work and what doesn't. For example, one of the features I was really excited about in really proud of was, both this website where multiple people could be logged into the website at the same time. So today, if you go to website, you know, if you all logged in and then I want to log in, you need to log out. There's the same browser, the same computer. But I thought, well, what if two people say you and me were watching a video together in front of a computer? What if a website could have you type your name and password, have me type my name and password? And then now the computer knows both of us are watching together and it gives both of us credit for anything we do as a group. Inferences feature, wrote it out in a school in San Francisco. We had about 20 something users. Where's the teacher there? Sacred Heart Cathedral, perhaps. It's just great. I mean, guess what? Zero people use this feature. It turns out people studying online. They want to watch the videos by themselves. So you can play back, pause at your own speed, rather than in groups. So that was one example of a timely lesson learned. All the many that allow us to hone into the set of features.

And I think with how machine learning movements as well, I think it didn't come out of nowhere. Instead, what happened was, as more people learn about machine learning, they will tell their friends, and their friends will see how the people will to their work, and then the community kept on growing. And I think what's still growing, you know, I don't know in the future what percentage of all developers will be AI developers. I could easily see it being more for 50%, right? Because so many AI developers broadly constitute not just people doing the machine learning modeling, but the infrastructure, the pipelines, all the software surrounding the core machine learning model. Maybe it's even bigger. I feel like today, almost every software engineer has some understanding of the cloud, not all. Maybe this is my microcontroller developer, it doesn't need to do the cloud. But I feel like the vast majority of software engineers today are sort of having the patience to cloud. I think the future maybe will approach nearly a hundred percent of all developers being, you know, in some way, an AI developer, or at least have an appreciation of machine learning.

Yeah, I think once upon a time, only a small part of humanity was literate, you know, could read and write. And maybe you thought, maybe not everyone needs to learn to read and write. You know, you just go listen to a few months, right, read to you and maybe that was enough. Or maybe we just need a few handful of authors to write the best solos. And then no one else needs to write. But what we found was that by giving as many people, you know, in some countries, almost everyone, basically literacy, it dramatically enhanced human-to-human communications. And we can now write for an audience of one, such as if I sent you an email, or you sent me an email. I think in computing we're still in that phase where so few people know how the code, that the code is mostly have the code for relatively large audiences. But the very one, well, most people became Developers at some level, similar to how most people in developing economies are somewhat literate. I would love to see the owners of a modern pop store be able to write a little bit of code to customize the TV display for their special this week. I think it will enhance human-to-computer communications, which is becoming more and more important today as well.

I could see society getting there. There's one interesting thing, you know, if I go talk to the mom and pop store, if I talk to a lot of people in their daily professions, I previously didn't have a good story for why they should learn to code. You know, we could give them some reasons. But what I found with the rise of machine learning and data sizes is that I think the number of people with a concrete use for data science in their daily lives, in their jobs, maybe even larger than the number of people with a concrete use for software engineering. For example, if you run a small mom and pop store, I think if you can analyze the data about your sales, your customers, I think there's actually a real value there. Maybe even more than traditional software engineering. So I find that for a lot of my friends and various professions, be it recruiters or countings or people that work in factories, which I do with more and more these days. I feel if they were data scientists at some level, they could immediately use that in their work. So I think that data science and machine learning maybe even easier on-tray into the developer world for a lot of people than the software engineering.

I think it depends on the concepts you will explain. For mathematical concepts, it's nice to build up the equation one piece of the time. And the whiteboard marker or the penistarles is a very easy way to build up the equation. Build up a complex concept one piece of the time while you're talking about it and sometimes that enhances understand the ability. The downside of writing is that it's slow, and so if you want to long sentence, it's very hard to write that. So I think they're frozen cons, and sometimes I use slides, and sometimes I use a whiteboard or a stylus.

Yeah, I was really fortunate to have at Peter at the US, my first PhD student, and I think even my long-term professional success builds on early foundations or early work that Peter was so critical to, so it was really grateful to him for working with me. You know, what not a lot of people know is just how hard research was and stories. Peter's PhD thesis was using reinforcement learning to fly helicopters. And so, you know, even today, the website heli.stanford.edu, HEOi.stanford.edu is still up in watch videos of us using reinforcement learning to make a helicopter fly upside down fly loose rolls. It's cool.

So it was over 10 years old.

And what not many people see is how hard it was. So Peter and Adam codes and Morgan quickly and I were working on various versions of the helicopter and a lot of things did not work. For example, in terms of one of the hardest problems we had was when the helicopter is lying around upside down doing stunts How do you figure out the position? How do you localize the whole factor? So we want to try and also also things, having one GPS unit doesn't work because you fly outside down, GPS unit is facing down, so you can't see the satellites. So we try to experiment to try to have two GPS units, one facing up, one facing down, so if you flip over, that didn't work because the downward facing one couldn't synchronize if you're flipping. quickly, Morgan quickly was exploring this crazy complicated configuration of specialized hardware to interpret GPS signals. Looking at FPGs, completely insane. It's spent about a year working on that. Didn't work. So I remember Peter, a great guy, him and me sitting down in my office, looking at some of the latest things we had tried that didn't work and saying, you know, done it like what now because because we tried so many things and it just didn't work. In the end, what we did in Adam Cole was crucial to this was put cameras on the ground and use cameras on the ground to localize the helicopter and that solved the localization problem so that we could then focus on the reinforcement learning and investment for reinforcement learning techniques to didn't actually make the helicopter fly. And you know, and remind it, when it was doing this work at Stanford, around that time, there was a lot of reinforcement learning theoretical papers, but not a lot of practical applications. So the Autonomous helicopter for flying helicopters was one of the few practical applications of reinforcement learning at the time, which caused it to become pretty well-known. I feel like we might have almost come full circle with today that's so much, but so much hype, so much excitement about reinforcing their own thing, but the game we're hunting for more applications of all of these great ideas that the Vietnamese come up with.

You know, I like theory, but when I work on theory myself, and this is personal taste, I'm not saying anyone else should do what I do. But when I work on theory, I proceed and enjoy more. If I feel that the work I do will influence people, have positive impact, will help someone. I remember when many years ago, I was speaking with a mathematics professor. And it kind of just said, hey, why do you do what you do? And then he said, he had stars in his eyes when he answered, and this mathematician, not from Stanford, different university, he said, I do what I do because it helps me to discover truth and beauty in the universe. He had stars in his eyes and he said, yeah. And I thought that's great. I don't want to do that. I think it's great that someone does that fully support the people that do it. A lot of respect for people that, but I am almost evaded when I can see a line to how the work that my teams and I are doing helps people. The world needs also so people. I'm just one type. I don't think everyone should do things the same way as I do. But when I delve into either theory or practice, if I press the have conviction that here's a path for it to help people, I find that more satisfying. It's half that conviction.

Yeah, I can tell you the thing we got wrong or the thing we got right. The thing we really got wrong was the importance of the early importance of unsupervised learning. So early days of Google Brain, we put a lot of effort into unsupervised learning rather than supervised learning. And this is arguing. I think it was around 2005 after Europe's, at that time caught nips, but now in Europe said ended. And Jeffington and I were sitting on the cafeteria outside, you know, the conference, we had lunch with this chatting. And Jeff pulled up this napkin. He started sketching this argument on an napkin. It was very compelling, the hour of repeated. Human brain has about 100 trillion, so there's 10 to the 14 synaptic connections. You will live about 10 to nine seconds, that's 30 years. You actually live for two by 10 to nine, maybe three by 10 to nine seconds. So let's say 10 to nine. So if each synaptic connection, each weight in your brain's neural network has just a one bit parameter that's 10 to the 14 bits you need to learn in up to 10 to nine seconds of your life. So, via this simple argument, which is a lot of problems, it's very simplified. That's 10 to the 5 bits per second you need to learn in your life. And I have a 1-year-old daughter. I am not pointing out 10 to the 5 bits per second of labels to her. So, and I think I'm a very loving parent, but I'm just not going to do that. So, from this very crude, definitely problematic argument, there's just no way that most of what we know is through supervised learning. But why can you get so many bits of information from sucking an image as audio experiences in the world? And so that argument, and there are a lot of no forces argument, you know, going to really convince me that there's a lot of power to unsupervised learning. So that was the part that we actually maybe got wrong. I still think unsupervised learning is really important, but we, but in the early days, you know, 10, 15 years ago, a lot of us thought that was to pop forward.

For the time, that was the part we got wrong. The part we got right was the importance of scale. So Adam Coats, another wonderful person, fortunate to work with him. He was in my group as Stanford at the time, and Adam had run these experiments as Stanford, showing that the bigger we train a learning algorithm, the better it's performance. And it was based on that, it was a graph that Adam generated, you know, where the X axis Y axis lies going up into the right. So big, big, big, big, big, distinct, the better performance accuracy is of earth practices. So it's really based on that chart that Adam generated, that gave me the conviction that you could scale these models way bigger than what we could on the few CPUs, which you would have to stand for, that we could get. even better results. And it was really based on that one figure that Adam generated, that gave me the conviction, a deco of Sebastian Thrun to pitch, you know, starting a project at Google, which became the Google Brain Crunch Brain.

It was controversial at the time. Some of my well-meaning friends, you'll see many people in the machine learning community, I won't name, but people with some of whom we know. My well-meaning friends came and were trying to give me friends, like, hey, Andrew, why are you doing this? It's crazy. It's in the near and that architecture. Look at these architectures of building. You're just going to go for scale. There's a bad career move. So my well-meaning friends, we're trying to, some of them were trying to talk me out of it. But I find that if you want to make a breakthrough, you sometimes have to have conviction and do something before it's popular, since that lets you have a bigger impact.

I think both are important. And it's also problem-dependent. So for a few datasets, we may be approaching, you know, base error rate or approaching or surpassing human level performance and then there is that theoretical ceiling that we will never surpass or base error rate. But then I think there are plenty of problems where we're still quite far from either human-level performance or from Bayes or Array. And Baker Data says, with neural networks, without further outlook, innovation will be sufficient to take us further. But on the flip side, if we look at the recent breakthroughs using, you know, transforming networks on language models, it was a combination of novel architecture, but also scale had a lot to do with it. We look at what happened with new GPU and birds. I think scale was a large part of the story.

I find it today, we have maturing processes to managing code, things like git, version control. It talks a long time to evolve the good processes. I remember when my friends and I were emailing each other, C++ files in email. But then we had CVS a version git, maybe something else in the future. We're very maturing in terms of two managing data and think of how the clean data and how the software, I'm very hot messy data problems. I think there's a lot of innovation there. to be has still.

I'll give you one example. When we work with manufacturing companies, it's not at all an comment for there to be multiple labels that disagree with each other, right? And so we were doing the work in vision inspection. We will You'll take, say, a plastic part, and show it to one inspector. And the inspector sometimes very opinionated to go clearly, that's the defect, the scratch, unacceptable, go and reject this part. Take the same part to different inspector, different, very opinionated, clearly the scratch is small. It's fine. Don't throw the way you're going to make us the old. And then sometimes you take the same plastic part, show it to the same inspector in the afternoon in this post in the morning, and very affinity goes in the morning to say, clearly it's okay, and the afternoon, equally confident, clearly this is a defect. And so what does the new I team suppose to do if sometimes even one person doesn't agree with himself or herself in the span of a day? So I think these are the types of very practical, very messy data problems that my team's wrestle with. In the case of large consumer internet companies where you have a billion users, you have a lot of data, you don't worry about it, just take the average, it kind of works. But in a case of other industry settings, we don't have Big data, if you're a small data, very small data says, maybe you're in the 100 defective parts or 100 examples of a defect. If you have only 100 examples, these little labeling errors, you know, if 10 of your 100 labels are wrong, that actually is 10% of your data is there as a big impact. So how do you clean this up? What are you supposed to do? This is an example of the types of things that my teams did. This is a land AI example, a wrestling with to deal with small data. When it comes to all the time, once you're outside, consumer internet.

Yeah. I find it actually when I'm teaching at Stanford, I increasingly encourage students at Stanford to try to find their own project for the end of term project rather than just downloading someone else's nicely clean data set. It's actually much harder if you need to go and define your own problem and find your own data set rather than you go to one of the several good websites, very good websites with clean sculpt data sets that you could just work on.

Deep Learning AI is working to create causes to help people break into AI. So my machine learning course that I taught through Stanford means one of the most popular causes on Coursera.

No, yeah, thank you. I was once reading a news article. I think it was tech review and I'm going to mess up this statistic. But I remember reading article that said something like one third of our programmers are self-taught. I may have the number one third around me as two thirds. But remember that article, I thought, this doesn't make sense. Everyone is self-taught. So because you teach yourself, I don't teach people. I just don't.

So the deep-lying specialization of a by-demand AI is, I think, it was called Serious Specialization. It might still be. So it's a very popular way for people to take that specialization, to learn about everything from neural networks, to how to tune in your network. So what is the confident to what is a RNA in our sequence model or what is an attention model? the different specialization steps everyone through those algorithms so you deeply understand it and can implement it and use it for whatever.

Yeah, need to understand basic programming since there are programming exercises in Python. And the math prerec is quite basic. So no calculus is needed. If you know calculus is great, you get better intuitions. But they're really trying to teach that specialization without requiring calculus. So I think on high school math would be sufficient. If you know how to multiply two matrices, I think that that that that that that's great.

basically in the algebra, even very, very, basically in the algebra and some programming. I think that people that have done the machine learning calls will find a deep learning specialization a bit easier, but it's also possible to jump into the deep learning specialization directly, but it'll be a little bit harder since we tend to go over faster concepts like how this gradient is going to send the work and what is an objective function, which covers more slowly in the machine learning calls.

So if you take the deep learning session, you learn the foundations of what is in your network. How do you build up a neural network from a single literacy unit to stack of layers to different activation functions? You don't have the trained in your networks. One thing I'm very proud of in that specialization is we go through a lot of practical know-how of how to actually make these things work. So what are the differences between different optimization algorithms? So what do you do of the algorithm over fit? So how do you tell the algorithm is over fitting? When do you collect more data? When should you not bother to collect more data? I find that Even today unfortunately there are engineers that will spend six months trying to pursue a political direction such as collect more data because we heard more data is valuable. But sometimes you could run some tests and could have figured out six months earlier that for this particular problem collecting more data isn't going to cut it. So just don't spend six months collecting more data. spend your time modifying the architecture or trying something else. So go through a lot of the practical know-how so that when someone, when you take the device specialization, you have those skills to be very efficient in how you build these networks.

Yes, and also the systematic frame rest of thinking for how to go about building practical machine learning. Maybe to make an analogy, when we learn to code, we have to learn the syntax of some programming language, right, be it Python or C++ or Octivo or whatever. But that equally important or maybe even more important part of coding is to understand how to string together these lines of code and to code here and things. So when should you put something in the function column? When should you not? How do you think about abstraction? So those frameworks are what makes the program efficient even more than understanding the syntax. I remember when I was an undergrad at Carnegie Mellon, one of my friends would debug their code by first trying to compile it and then it was C++ code. And then every line that is syntax error, they want to get rid of syntax errors as quickly as possible. So how do you do that? Well, they would delete every single line of code with syntax error. So really efficient for getting rid of syntax errors were horrible debugging errors. So I think, so we learn how to debug. And I think in machine learning, the way you debug, machine learning program is very different than the way you You know, I do buy new research or whatever. Use a debugger, I trace the code, and then traditional software engineering. So, as an evolving discipline, but I find that the people that are really good at debugging machine learning algorithms are easily 10x, maybe 100x faster at getting some of the work.

Yeah, the often the question is, why doesn't it work yet? Or can I expect it to eventually work? And what are the things I could try? Change the architecture, more data, more regularization, different optimization algorithms, different types of data, so to answer those questions systematically, so that you don't heading down, so you don't spend six months heading down the blind alley before someone comes and says, why is she spend six months doing this?

similar to learning mathematics, I think one of the challenges of deep learning is that there are a lot of concepts that build on top of each other. If you ask me what's hard about mathematics, I have a hard time pinpointing one thing, is it addition, subtraction, is it carry, is it multiplication law stuff? I think one of the challenges of learning math and learning certain technical fields is that a lot of concepts, and if you miss a concept, then you're kind of missing the prerequisite for something that comes later. So in the deep learning specialization, try to break down the concepts to maximize the art of each component being understandable. So when you move on, to the more advanced thing. We learn your confidence, hopefully you have enough intuitions from the earlier sections, to then understand why we structure confidence in a certain way and then eventually why we built RNNs and LSTMs or attention model in a certain way, building on top of the earlier concepts. Are you doing a lot of teaching as well? Do you have a favorite? This is the hard concept moment in your teaching.

I still teach reinforcement for slowly learning in one of my scientific classes, and my PhD thesis is learning for some things, right? I find that if I'm trying to teach students the most useful techniques for them to use today, I end up shrinking the amount of time I talk about reinforcement learning. It's not what's working today. Now, our work changes so fast. Maybe it does be totally different in a couple years. But I think we need a couple more things for reforsal learning to get there. One of my teams is looking to reforsal learning for some robotic control tasks. So I see the applications, but if you look at it as a percentage of all of the impact of the types of things we do, it's at least today outside of playing video games in a few of the games. Actually, that neuros, a bunch of us were standing around saying, hey, what's your best example of an actual deployment force of learning application? And, you know, among like, seeing a machine learning researchers, right? And again, there are some emerging ones, but there are not that many great examples.

That's really cool. And I feel like the world actually needs all sorts. Even within machine learning, I feel like deep learning is so exciting. But AI team shouldn't just use deep learning. I find that my team is just a portfolio of tools. And maybe that's not the exciting thing to say. But some days we use the neural net, some days we use PCA. Actually, the other day I was sitting down with my team, looking at PCA residuals, trying to figure out what's going on with PCA applied to manufacturing problem. Some days we use a proxy draft goal model, some days we use a knowledge draft, which is one of the things that has tremendous industry impact, but the amount of chat about knowledge drafts in academia is really thin compared to the actual raw impact. So I think we're in fourth and learning should be in that portfolio, and then it's about balancing how much we teach all of these things. And the world should have diverse skills, or be sad if everyone just learn one narrow thing.

I think that if my only job was being an academic researcher and having unlimited budget and didn't have to worry about short-term impact and only focus on long-term impact, I've recently spent all my time doing research on non-survivors learning. I still think, unsurprisingly, there's a beautiful idea. At both this past narratives in ICML, I was attending workshops on this interview as talks about self-supervised learning, which is one vertical segment, maybe a sort of unsurvised learning that I'm excited about. Maybe just a summarized idea. I guess you know the idea of describe movie. No, please. So here's an example of self-survised learning. Let's say we grab a lot of unlabeled images after the internet, so we've infinite amounts of this type of data. I'm going to take each image and rotate it by a random multiple of 90 degrees. And then I'm going to train a supervised neural network to predict what was the original orientation. So it has some sort of a rotated 90 degrees, 180 degrees, 170 degrees or zero degrees. So you can generate an infinite amount of label data because you rotate to the image. So you know what's the structure of label. And so various researchers have found that by taking on label data and making up label data sets and training a large neural network on these tasks, you can then take the hidden layer representation and transfer it to a different task very powerful. Learning word embeddings where we take a sentence to lead a word predict the missing word which is how we learn one of the ways we learn word embeddings is another example and I think there's now this portfolio of techniques for generating these made up tools. Another one called jigsaw, would be if you take an image, cut it up into a 3x3 grid, so like a 9, 3x3 puzzle piece, jump out the 9 pieces and have a neural network predict which of the 9 factorial possible permutations it came from. So many groups, including, you know, OpenAI, Peter B has been doing some work on this to Facebook, Google, Brain, I think DeepMine, oh, actually, Aaron then did all of his great work on the CPC objective. So many teams are doing site work, and I think this is a way to generate infinite label data. And I find this a very exciting piece of unsupervised learning.

I don't think there's a hole in Chalada. I think it's just a piece of it. And I think this one piece, self-supervised learning, is starting to get traction. We're very close to it being useful. Well, we're then betting really useful. I think we're getting close and closer to just having a significant real world impact, maybe in computer vision and video. But I think this concept, and I think there'll be other concepts around it. Other unsupervised letting things that that worked on that been excited about. I was really excited about spa's coding and I see a slow feature analysis. I think all of these are ideas that veers of us were working on about a decade ago before we all got distracted by how well supervised learning was doing.

I think there's a lot more work that one could explore around the steam of ideas and other ideas to come with better algorithms.

The official length of the Deavank Socialization is I think 16 weeks, so about four months, but it's going to go at your own pace. So if you subscribe to the Deavank Socialization, there are people that have finished it in less than a month by working more intensely and studying more intensely. So it really depends on the individual. We're created a different socialization. We wanted to make it very accessible and very affordable. And with Coursera and Devon Diya's education mission, one of the things that's really important to me is that if there's someone for whom paying anything is a financial hardship, then just apply for financial aid and get it for free.

I think getting the habit of learning is key and that means regularity. So for example, we send out a weekly newsletter the batch every Wednesday. So people know it's coming Wednesday. You can spend a lot of the time on Wednesday. catching up on the latest news through the batch on Wednesday and for myself I've picked up a habit of spending some time every Saturday and every Sunday reading or studying and so I don't wake up on Saturday and have to make a decision to feel like reading or studying today or not. It's just, it's just what I do. And the fact is, a habit makes it easier. So I think if someone can get into that habit, it's like, you know, just like, we brush our teeth every morning. I don't think about it. If I thought about this a little bit annoying to have to spend two minutes doing that, but it's a habit that it takes no cognitive loads. But there's so much harder to make a decision every morning. So, and then actually that's the reason why we're the same thing every day as well. It's just more or less the decision. I just get out and then we're right for sure. So, but I think if you get that habit, that consistency of studying, then actually fuse easier.

Yeah, and I think it's often not about the birth of sustained ethas and the all-nightas because you could only do that limited number of times. It's the sustained ethas over a long time. I think you're reading two research papers. It's a nice thing to do. But the power is not reading two research papers. This is reading two research papers a week for a year. Then you read a hundred papers and you actually learn a lot when you read a hundred papers.

One thing I still do when I'm trying to study something really deeply is take handwritten notes. If there is, I know there are a lot of people that take the deep learning courses during the commutes or something where it may be more awkward to take notes. So I know it may not work for everyone. But when I'm taking courses on Coursera, you know, and I still take some of my every night then. The most recent I took was a course on clinical trials because those inches about that. I got out of my little, most skinned notebook and I was sitting in my desk and just taking down notes. So what the instructor was saying and that act, we know that that act of taking notes, preferably handwritten notes, increases retention.

Yeah. So there have been a few studies of your search online. You find some of these studies that taking handwritten notes, because handwriting is slower as we're saying just now. It causes you to recode the knowledge in your own words more. And that process of recoding promotes long-term attention. This is as opposed to typing, which is fine. Again, typing is better than nothing or taking a class and not taking a notice that and not taking any class law. But comparing the handwritten notes and typing, you can usually type faster for a lot of people or do you can handwrite notes. And so when people type them all likely to just try and strike verbatim what they heard and that reduces the amount of recalling and that actually results in less long-term retention.

Yeah, and because you can't write as many words, you have to take whatever they said and summarize it into fewer words. And that summarization process requires deeper processing of the meaning, which then results in better attention. That's fascinating. I think, you know, because of course, Sarah has been so precise studying pedagogies. It's actually one of the passions that I really love learning how to more efficiently help others learn. One of the things I do both in creating videos or when we write the batch is I try to think is one minute spent with us going to be a more efficient learning experience than one minute spent anywhere else and we really try to You know, make a time efficient from the learners, because in every one's busy. So when we're editing, I often tell my teams, every word needs to fight for his life. And if you can delete it, where this is deleted, and not wait, that's not wasted in this time.

So most important thing is to get started. Right. And I think in the early parts of the career course work, like the Delegant specialization, it's a very efficient way to master this material. So because, you know, instructors be at me or someone else or you know, Lawrence Moroni teaches our TensorFlow specialization, other things we're working on. spend effort to try to make it time efficient for you to learn new concepts. So coursework is actually a very efficient way for people to learn concepts and the beginning parts of break into new fields. In fact, one thing I see at Stanford, some of my PhD students want to jump in the research right away, and I actually tend to say, look, in your first couple years of PhD students, spend time taking courses because it lays a foundation. It's fine if you're less productive in your first couple years. You'd be better off in the long term. Beyond the certain point, there's materials that doesn't exist in courses, because it's too cutting-edge. The courses we created here, there's some practical experience that we're not yet diagnosed teaching in the course. And I think after exhausting the efficient course, we're then most people need to go on to either ideally work on projects, and then maybe also continue their learning by reading blog posts and research papers and things like that. Doing projects is really important. And again, I think it's important to start spawns, just do something. Today, you read about deep learning, if you say, oh, all these people are doing such exciting things. Whether I'm not building a neural network, they change the world and what's the point? Well, the point is sometimes building that time in your network, you know, be it Mness or upgrade to fashion Mness to whatever. So doing your own fun, Harvey project, that's how you gain the skills to let you do bigger and bigger projects. I find this to be true at the individual level and also at the organizational level. For a company to become good at machine learning, sometimes the right thing to do is not to tackle the giant project is instead to do the small project that lets the organization learn and then build up from there. But this is true both for individuals and for companies.

I think that there are multiple good options of which doing PhD could be one of them. I think that if someone's admitted to a top PhD program, you know, that MIT stands at top schools, I think that's a very good experience. Or if someone gets a job at a top organization at the top AI team, I think that's also a very good experience. There are some things you still need a PhD to do. If someone's aspiration is to be a professor at the top academic university, you just need a PhD to do that. But if it goes to, you know, start a complete, build a complete, do great technical work. I think a PhD is a good experience, but I would look at the different options available to someone, you know, where the place is, where you can get a job, where the place is, you get a PhD program and kind of weigh the pros and cons of those.

I think the thing that affects your experience more is less are you in this company versus that company or academia versus industry. I think the thing that affects your experience most is who are the people you interact with in a daily basis. So even if you look at some of the large companies, The experience of individuals in different teams is very different. And what matters most is not the logo above the door when you walk into the giant building every day. What matters the most is who are the 10 people who are the 30 people you interact with every day? So I actually tend to advise people if you're going to drop from a company. ask who's your manager, who are your peers, who are you actually going to talk to? We're all social creatures. We tend to, you know, become more like the people around us. And if you're working with great people, you will learn faster. Or if you get admitted, if you get a job in a great company or a great university, maybe the logo you walk in is great, but you're actually stuck in some team doing really work that doesn't excite you. And then that's actually really bad experience. So, this is true for universities and for large companies. For small companies, you can kind of figure out who you'd be working quite quickly. And I tend to advise people if a company refuses to tell you who you work with. So, from say, oh, join us. If a rotation system will figure out, I think that that's a worrying answer because it means you may not get sent to, you mean not actually get to team with brave peers and brave people that work with.

Yeah, but I think when someone interviews at the university or the research lab or the launch corporation, it's good to insist on just asking, who are the people, who is my manager? And if you refuse to tell me, I'm going to think, well, maybe that's because you don't have a good answer. It may not be someone I like.

And actually, in my standard cost, CS230, as was an ACN talk, I think I gave our long talk on career advice, including on the job search process and then some of these. So if you can find those videos on the last one, I'll point people to them.

You know, it's the kind of value. A lot of startup failures come from building a product that no one wanted. So when cool technology, but who's going to use it? So I think I tend to be very outcontraught and customer obsess. Ultimately, we don't get to vote if we succeed or fail, is only the customer that the only one that gets a thumbs up or thumbs down votes in the long term. In the short term, you know, there are various people who get various votes, but in the long term, that's why really matters.

I think so. I think startups that are very customer focused, customer says deeply understand the customer and are oriented to serve the customer on more likely to succeed. With a provision that I think all of us should only do things that we think create social good and lose the world for. I personally don't want to build addictive digital products, just a cell of ads. There are things that that could be lucrative that I won't do. But if we can find ways to serve people in meaningful ways, I think those can be, those can be great things to do. Either an academic setting or an corporate setting or a startup setting.

I remember when I was leading the AI group at Baidu, I had two jobs, two parts of my job. One was to build an AI engine to support the existing businesses, and that was running, you know, just ran this performance off. The second part of my job at the time was to try to systematically initiate new lines of businesses using the company's AI capabilities. You know, the self-driving car team came out of my group, the smart speaker team, similar to what is Amazon Echo or Alexa in the US. But we actually announced it before Amazon did. So I do wasn't following Amazon that came out of my group. And I found that to be actually the most fun part of my job. So what I want to do was to build AI Fund as a start of studio to systematically create new startups from scratch. With all of the things we can now do of AI, I think the ability to build new teams to go after this rich space of opportunities is a very important way to very important mechanism to get these projects done that we'll move the world forward. So I've been fortunate to build a few teams that had a meaningful positive impact and I felt that we might be able to do this in a more systematic, repeatable way. So a startup studio is a relatively new concept. There are maybe dozens of startup studios right now. But I feel like all of us, many teams are so trying to figure out how do you systematically build companies with a high success rate. So I think even a lot of my new venture capital friends are seem to be more and more building companies rather than investing in companies. But I find the fascinating thing to do. to figure out the mechanisms by which we could system as create built successful team, successful businesses in areas that we find meaningful.

It's actually a place for us to build standards from scratch. So we often bring in founders and work with them, or maybe even have existing ideas that we match, founders with, and then this launches, hopefully into successful companies.

I think we've been constantly improving and iterating on our processes, but how we do that. So things like how many customer calls do we need to make and only get customer validation. How do you make sure this technology can be built? Quite a lot of our businesses need cutting-edge machine learning algorithm. So it kind of algorithms are developed in the last One or two years and even if it works in a research paper, it turns out taking the production is really hard. There are a lot of issues for making these things work in the real life that are not widely addressed in the academia. So how do you validate that this is actually doable? how the ability to get the specialized domain knowledge, be it an education or healthcare, whatever set you're focusing on. So I think we're actually getting, we've been getting much better at giving the entrepreneurs a high success rate, but I think we're still, I think the whole world is still in the early phases, thinking this out.

Very much so. Starting a company to most entrepreneurs is a really lonely thing and I've seen so many entrepreneurs not know how to make certain decisions like When do you need to, how do you do BDP sales? If you don't know that, this is really hard. Or how do you market this efficiently, other than buying ads, which is really expensive, other more efficient tactics that. For machine learning project, basic decisions can change the course of whether machine learning products or not. And so there are so many hundreds of decisions that entrepreneurs need to make and making mistakes in a couple of key decisions kind of a huge impact on the faith of the company. So I think a standard studio provides a support structure that makes starting a completely much less of a low-end experience. And also, when facing with these key decisions like trying to hire your first VP of engineering was a good selection criteria. How do you solve? Should I hire this person or not? By helping, by having like ecosystem around the entrepreneurs to find this to help, I think we help them at the key moments and hopefully significantly make them more enjoyable and then hire success rates.

And also to hope them recognize what they may not even realize as a key decision point.

I can say one other thing. I think building companies is one thing, but I feel like it's really important that we build companies that move the world forward. For example, within the AFN team, there's one's an idea for a new company that if it had succeeded, whatever resulted in people watching a lot more videos in a certain narrow vertical type of video. I love that the business case was fine. The revenue case was fine, but I looked at it and just said, I don't want to do this. I don't actually just want to have a lot more people watch this type of video. It wasn't educational. It was educational, maybe. And so I code the idea on the basis that I didn't think it would actually help people. So whether building companies or work event prizes or doing personal projects, I think stuff to each of us to figure out what's the difference we want to make in the world.

AI is a general purpose technology and I think it will transform every industry. Our community has already transformed the logic sent to software internet sector, most software internet companies. They are also at the top right, five or six or three or four already have reasonable machine learning capabilities or getting there. It's the room from improvement. But when I look outside the software internet sector, everything from manufacturing, agriculture, healthcare, logistics, transportation, the so many opportunities that very few people are working on. So I think the next way for AI is for us to also transform all of those other industries. There was a McKinsey study estimating $13 trillion of global economic growth. U.S. GDP is $19 trillion, so $13 trillion is a big number, or PWC has been $16 trillion, so whatever number is this large. But the interesting thing to me was a lot of that impact would be outside the software internet sector. So we need more teams to work with these companies to help them adopt AI. And I think this is one thing, so I'll help drive global economic growth and make humanity more powerful.

Frankly, I think it's all of them. Some of the ones I'm spending a lot of time on are manufacturing, agriculture, looking to health care. For example, in manufacturing, we do a lot of work in visual inspection where today there are people standing around using the eye human eye to check if, you know, this plastic part or the smartphone or this thing has a stretch or a gentle something in it. We can use a camera to take a picture, use a algorithm, deep learning, and other things to check if it's defective or not and does how factories improve yield and improve quality and improve throughput. It turns out the practical problems we run into are very different than the ones you might read about in most research papers. The data says it really small, so we face small data problems. The factories keep on changing the environment, so it works well on your test set, but guess what? Something changes in the factory. The lights go on or off. Recently, there was a factory in which a dirt threw through the factory and pooped on something. So that changed stuff. Increasing our revved, make robustness. So all the changes happen in the factory. I find that we run a lot of practical problems that are not as widely discussed in academia. It's really fun being on a cutting-edge solving piece of problems. Maybe before many people are even aware that there is a problem.

I published a document called the AI Transformation Playbook that's online and taught briefly in the AI for everyone, course and course error, about the long-term journey that Counties should take, but the first step is actually to start small. I've seen a lot more companies fail by starting too big than by starting too small. Take even Google. Most people don't realize how hard it was and how controversial it was in the early days. So when I started Google Brain, it was controversial. People thought deep learning, near and near tried it, didn't wear it. Why would you want to do deep learning? So my first internal customer in Google was Google Speech Team, which is not the most lucrative project in Google, but not the most important. It's not web search or advertising. But by starting small, my team helped to speak to him, build a more accurate speech recognition system. And this caused their peers, other teams to start at more faith and deep learning. My second internal customer was the Google Maps team, where we used computer vision to read house numbers from basic street view images to more accurately locate houses with Google Maps, so improve the quality of the geodater. And there's only after those two successes that I then start to the most serious conversation with the Google Ads team.

I think the early small-scale projects, it helps the teams gain faith, but also helps the teams learn what these technologies do. I still remember when our first GPU server, it was a server under some guy's desk. And then that taught us early important lessons about how do you have multiple users share a set of GPUs, which is really not obvious at the time. but those early lessons were important. We learned a lot from that first GPU server that later helped the teams think through how to scale it up to much launch deployments.

I think building and deploying machine learning systems is hard. There's a huge gulf between something that works in a Jupyter notebook on your laptop versus something that runs in a production deployment setting in a factory or a culture plans or whatever. So I see a lot of people get something that works on your laptop and say, wow, look what I've done. That's great. That's hard. That's a very important first step. but a lot of teams underestimate the rest of the steps needed. So for example, I've heard this exact same conversation between a lot of machine learning people and business people. The machine learning person says, look, my algorithm does well on the test set and the clean test set I didn't peak. And the business person says, thank you very much, but your algorithm sucks, it doesn't work. And the machine learning person says, no, wait, I did well on the test set. And I think there is a gulf between what it takes to do well on a test set on your high-drive versus what it takes to work well in a deployment setting. Some common problems, robustness and generalization, you need to find something in the factory, maybe they chop down a tree outside the factory so that treat no longer covers the window and the lighting is different. So the test set changes. And in machine learning, especially in academia, we don't know how the view of test set distributions are dramatically different than the training set distribution. You know that this research, this stuff like domain annotation, transfer learning, you know that the people working on it, but we're really not good at this. So how do you actually get this to work? Because your test set distribution is going to change. And I think also, if you look at the number of lines of code in the software system, the machine learning model is maybe 5% or even fewer relative to the entire software system you need to build. So how do you get all that work done? They make a reliable and systematic.

Yes, and the software system needs to interface with people's workloads. So machine learning is automation theorists. If we take one task out of many tasks that are done in factories. So, actually, there's lots of things. One task is vision spectrum. If we automate that one task, it can be really valuable. But you may need to redesign a lot of other tasks around that one task. For example, Say the machine learning algorithm says this is defective. What is supposed to do? Do you throw away? Do you get a human to double check? Do you want to rework it or fix it? So you need to redesign a lot of tasks around that thing you've now automated. So planning for the change management and making sure that the software you write is consistent with the new workflow. And you take the time to explain to people when it's happened. So I think What land AI has become good at, and then I think we learn by making mistakes and painful experiences or might. What become good at is working with upon this, to think through all the things beyond just the machine learning model around you. You put the notebook, but to build the entire system, manage the change process, and figure out how to deploy this in the way that has an actual impact. The processes that the large software tech companies use for deploying don't work for a lot of other scenarios. For example, when I was leading your large speech teams, if the speech recognition system goes down, what happens? What alarms goes off? And then someone like me would say, hey, you 20 engineers, please fix this in the market. But if you have a system go down the factory, there are not 20 machine learning engineers sitting around. You can pay your duty and have them fix it. So how do you Do you have the maintenance or the death ops or the ML ops or the other aspects of this? So these are concepts that I think land AI and then a few other teams on the cutting-edge are but we don't even have systematic terminology yet to describe some of the stuff we do because I think we're inventing an butterfly.

Let me ask a- The two are not inconsistent.

I would love to get to AGI and I think humanity will, but whether it takes 100 years or 500 or 5000 I find hard to estimate.

I do worry about the long term fate of humanity. I do wonder as well. I do worry about overpopulation on the planet Mars. Just not today. I think there will be a day when maybe maybe someday in the future, Mars would be polluted. There are these children dying and some will look back at this video and say Andrew, how is Andrew so hot that he didn't care about all these children dying on the planet Mars? And I apologize to the future viewer. I do care about the children, but I just don't know how the product of the work on that today.

You know, if you take cell driving cars, the biggest problem with cell driving cars is not that there's some trolley dilemma and you teach this. So, you know, how many times when you're driving your car, did you face this moral dilemma? Is it food that I trash into? So, I think cell driving cars will run the problem. roughly as often as we do when we drive our cars. The biggest problem of self-joint cars is when there's a big white truck across the road and what you should do is break and not crash into it and the self-joint car fails and it crashes into it. So I think we need to solve that problem for us. I think the problem was some of these discussions about HGI alignments, the paperclip problem, is that is a huge distraction from the much harder problems that we actually need to add just today. Some hard problems need to add just today. I think bias is a huge issue. I worry about wealth inequality. The AI and internet are causing the separation of concentration of power because we can now centralized data used air to process it, and so industry after industry, with effect every industry. So the internet industry has a lot of win-and-take modes of win-and-take all dynamics, but with infected all these other industries, so also giving these other industries win-and-take modes of win-and-take all flavors. So look at what Uber and Lyft did to taxi industry. So we're doing this type of thing to log in. So this, so we're creating tremendous wealth, but how do we make sure that the wealth is fairly shared? I think that, and then how do we help people whose jobs are displaced? You know, I think education is part of it. There may be even more that we need to do than education. I think bias is a serious issue. They're at various uses of AI and deep fakes being used for various affairs purposes. So I worry about some teams maybe accidentally and I hope not deliberately making a lot of noise about things that problems in the distant future rather than focusing on some to the much harder problems.

And I think, and then just to the, some companies are a regulator, it comes to you and says, look, your product is messing things up. Fixing it may have a revenue impact. Well, it's much more fun to talk to them about how your problem is not the wife of humanity that's into a face that actually really hard problems we face.

You know, I've made so many mistakes. It feels like every time I discover something, I go, why did I think of this? you know, five years earlier or you're 10 years earlier. And sometimes I read a book and I go, I wish I read this book 10 years ago, my life would have been so different. Although that happened recently and then I was thinking, if only I read this book when we were starting a course era, I could have been so much better. But I discovered that book had not yet been written or it's starting to cost error, so that may be better. But I find that the process of discovery, we keep on finding out things that seem so obvious in hindsight. But it always takes us so much longer than I wish to figure it out.

Well, two answers. One, that's my daughter, Nova. Yes, of course. She's like, no matter how much time I spend for her, I just can't spend in her time with her. Congratulations. Thank you. And then second is the helping other people. I think to me, I think the meaning of life is helping others achieve whatever our dairy dreams. And then also to try to move the world forward by making humanity more powerful as a whole. So the times that I felt most happy and most proud was when I felt someone else allowed me the good fortune of helping them a little bit on the path to their dreams.