Transcript for Tuomas Sandholm: Poker and Game Theory

Yeah, happy to. So, heads up, no limit, Texas Holden has really merged into AI community as a main benchmark for testing these application independent algorithms for imperfect information game solving. And this is a game that's actually played by humans, you don't see that much on TV or casinos because well for various reasons but you do see it in some expert level casinos and you see it in the best poker movies of all time it's actually an event in the world series of poker but mostly it's played online and typically for pretty big sums of money and this is a game that usually only experts play. So if you go to your home game on a Friday night, it probably is not going to be heads up. No limit decks as whole. That might be not limit decks as whole them in some cases, but typically for a big group and it's not as competitive. Well heads up means it's too play or so it's really like me against you and my better or are you better much like chess or or go in that sense but in imperfect information game which makes it much harder because I have to deal with issues of you know and things that I don't know and I know things that you don't know instead of pieces being nice and laid on the board for both of us to see.

Yeah. So as you said, you first get two cards in private each. And then there's a betting rod. Then you get three cards in public on the table, then there's a betting round. Then you get the fourth card in public on the table, there's a betting round. Then you get the fifth card on the table, there's a betting round. So there's a total of four betting rounds and four trances of information revelation if you will. The only the first trance is private. And then it's public from there.

We invited four of the top 10 players with his specialist players in heads-up, no limit takes us all and which is very important because this game is actually quite different than the multiplayer version. We brought him in to Pittsburgh to play at the reverse casino for 20 days. We wanted to get 120,000 hands in. because we wanted to get statistical significance. So it's a lot of hands for humans to play, even for these top pros who play fairly quickly normally. So we couldn't just have one of them play so many hands 20 days. They were playing basically morning to evening and raised 200,000 as a little incentive for them to play. And the setting was so that they didn't all get 50,000. We actually paid them out based on how they did against the AI each, so they had an incentive to play. as hard as they could, whether they're way ahead, the way behind or right at the mark of beating the AI. And you don't make any money, unfortunately. Right, no, we can't make any money. So originally, a couple of years earlier, I actually explored whether we could actually play for money because that would be, of course, interesting as well, to play against the top people for money, but the Pennsylvania Gaming Board said no. So if we couldn't, so this is much like an exhibit like for a musician or a boxer or something like that.

Yes, they were playing much like they normally do. These stop players when they play this game. They play mostly online. So they used to playing through UI. And they did the same thing here. So there was this layout you could imagine, there's a table on a screen, the human sitting there and then there's the AIC thing there and the screen shows everything that's happening, the cards coming out and shows the bits being made and we also had the bedding history for the human. So if the human forgot what had happened in the hands so far, they could actually reference back and so forth.

Well, it didn't really matter that they wouldn't have forgotten anywhere. These are top quality people, but we just wanted to put out there so it's not a question of a human forgetting and the AI somehow trying to get advantage of a better memory.

Yeah, that's great. It's a great question. So for 18 months earlier, I had organized a similar brain versus AI competition with the previous AI called Cloudical, and we couldn't beat the humans. So this time around, it was only 18 months later, and I knew that this new AI liberators was way stronger, but it's hard to say how you'll do against the top humans before you try. So I thought we had about a 50-50 shot. and the international betting sites put us as a 4-1 or 5-1 underdog. So it's kind of interesting that people really believe in people and get over AI. Not just people don't just believe over believing themselves, but they have over confidence in other people as well compared to the performance of AI. And yeah, so we were afforded one to five to one underdog. And even after three days of beating the humans in a row, we were still 50, 50 on the international betting sites.

Yeah. So I'll split it into two parts. So one is wider humans trust humans more than AI and all have over confidence in humans. Yes. I think that's not really related to the telequestion. It's just that they've seen these top players how good they are and they're really fantastic. So it's just hard to believe that the AI could beat them. So I think that's where that comes from. and that's actually maybe a more general lesson about AI that I'm until you've seen it over perform a human, it's hard to believe that it could. But then the tales, a lot of these top players, they're so good at hiding tales that among the top players, it's actually not really worth it for them to invest a lot of effort trying to find tales in each other because they're so good at hiding them. So yes, at the kind of Friday evening game tells that gonna be a huge thing. You can read other people and if you're a good reader, you'll read them like an open book. But at the top level, so poker now. The tells become a lot smaller and smaller aspect of the game as you go to the top levels.

Yeah, so you're exactly right. So when you go from a game three that's 10 to the 161, especially in an imperfect information game, it's way too large to solve directly, even with our fastest equilibrium finding algorithms. So you want to abstract it first, and abstraction in games is much trickier than abstraction in MDPs or other single agent settings. because you have these abstraction pathologies that if I have a finer grain abstraction, The strategy that I can get from that for the real game might actually be worse than the strategy I can get from the coarse-grained abstraction. She have to be very careful.

We're hitting actions. So there's information abstraction, talk about general games. Information abstraction, which is the abstraction of what chance does. And this would be the cards in the case of poker. And then there's action abstraction, which is abstracting the actions of the actual players, which would be bits in the case of poker. Your self and the other players? Yes, yourself and other players. And for information abstraction, We were completely automated. So these are algorithms. We do what we call potential aware abstraction, where we don't just look at the value of the hand, but also how it might materialize in the good or bad hands over time. And it's a certain kind of bottom-up process with integer programming there and clustering and various aspects. How do you build this abstraction? and then in the action abstraction. There, it's largely based on how humans and other AIs have played this game in the past, but in the beginning, we actually used an automated action abstraction technology, which is probably convergent that it finds the optimal combination of bad sizes, but it's not very scalable. So we couldn't use it for the whole game, but we used it for the first couple of betting actions.

So that's why you have to play a lot of hands, so that the role of luck get smaller. So you could otherwise get lucky and get some good hands and then you're going to win the match. Even with thousands of hands, you can get lucky. Because there's so much variance in normally mid-texers hold them because if we both go all in, it's a huge stack of variance. So there are these massive swings in normally mid-texers hold them. So that's why you have to play not just thousands, but over a hundred thousand hands, they'll get statistics of significance.

Oh, that's a good question. There's actually, I heard the story that there's this Norwegian female poker player called Annette Oberstad, who's actually won a tournament by doing exactly that. But that would be extremely rare. So I can't really play well with that.

Yeah, so as you said, Librarios did not use learning methods and played very well without them. Since then, we have actually here. We have a couple of papers on things that do use learning techniques. So, and deep learning in particular, and sort of the way you're talking about where it's learning an evaluation function. but in imperfect information games, unlike let's say in Go or now also in Chess and Shogi. It's not sufficient to learn and evaluation for a state because the value of an information set depends not only on the exact state, but it also depends on both players' beliefs. Like, if I have a bad hand, I'm much better off if the opponent thinks I have a good hand. And vice versa, if I have a good hand, I'm much better off if the opponent believes I have a bad hand. So the value of a state is not just a function of the cards. It depends on if you will the path of play. But only to the extent that it's captured in the belief distributions. So that's why it's not as simple as it is in perfect information games. And I don't want to say it's simple there either. It's of course very complicated computationally there too, but at least conceptually it's very straightforward. There's a state. There's an evaluation function. You can try to learn it. Here. You have to do something more. And what we do is in one of these papers, we're looking at allowing where we allow the opponent to actually take different strategies at the leaf of the search tree as if you will. And that is a different way of doing it and it doesn't assume therefore a particular way that the opponent plays, but it allows the opponent to choose from a set of different continuation strategies. And that forces us to not be too optimistic. in a local head search. And that's one way you can do sound local head search in imperfect information games, which is very difficult. And in you were asking about deep stack, what they did, it was very different than what we do, either in Librarians or in this new work. They were randomly generating various situations in the game. Then they were doing the look ahead from there to the end of the game, as if that was a start of a different game. And then they were using deep learning to learn those values of those states, but the states were not just the physical states, they include belief distributions.

Yes. So we're talking about exactly that. much like you do in alpha beta search or want to crawl to create search but with different techniques. So there's a different search algorithm and then we have to deal with the leaves differently. So if you think about what Libraris did, we didn't have to worry about this because we only did it at the end of the game. So we would always terminate into a real situation and we would know what to pay out this. It didn't do these depth limited local heads, but now in this new paper, which is called depth-limited. I think it's called depth-limited search for imperfect information games. We can actually do sound depth-limited look ahead. So we can actually start to do the look ahead from the beginning of the game on because that's too complicated to do for this whole long game. So in Librarians, we were just doing it for the end.

Yeah, yeah, it is explicitly modeled, but it's not assumed that beliefs are actually output, not input. Of course, the starting beliefs are input, but they just fall from the rules of the game, because we know that the dealer deals uniformly from the deck. So I know that every pair of cards that you might have is equally likely. I know that for a fact, that's just follows from the rules of the game. Of course, except the two cards that I have, I know you don't have those. You have to take that into a context called card removal and that's very important. Is the dealing always coming from a single deck in a heads up, so you can assume single deck that's a card removal, you know that if I have the Ace of Spades, I know you don't have an Ace of Spades.

Well, that's where this beauty of game theory comes. So Nash equilibrium, which John Nash introduced in 1950 introduces what rational play is when you have more than one player. And these are pairs of strategies where strategies are contingency plans, one for each player. So neither player wants to deviate to a different strategy, giving that the other doesn't deviate. But that's a side effect. You get the beliefs from page rule. So Nash equilibrium really isn't just deriving in this imperfect information games. Nash equilibrium doesn't just define strategies. It also defines beliefs for both of us. And it defines beliefs for each state. So the each state, it's called information sets. At each information set in the game, there's a set of different states that we might be in, but I don't know which one we're in. Nashically really tells me exactly what is the probability distribution over those real wall states in my mind.

So why? I'll do a simple example. You know the game rock piece paper scissors. So we can draw it as player one moves first and then player two moves, but of course It's important that player 2 doesn't know what player 1 moved, otherwise player 2 would win every time. So we can draw that as an information set, where player 1 makes 1 or 3 moves first, and then there's an information set for player 2, so player 2 doesn't know which of those nodes we'll see. But once we know the strategy for player one, Nash equilibrium will say that you play one third rock, one third paper, one third scissors. From that I can derive my beliefs on the information said that they're one third, one third, one third.

No, that's the game theory isn't really players specific. So that's also why we don't need any data. We don't need any history how these particular humans played in the past or how any AI or even had played before. It's all about rationality. So we just think that the AI just thinks about what would a rational opponent do. And what would I do if I were, I am Russian, and that's that's the idea of game theory. So it's really a data-free opponent-free approach.

Exactly. There's no opponent-modeling per se. I mean, we've done some work on combining opponent modeling with game theory so you can exploit weak players even more, but that's another strand and in the liberators we didn't turn that on. Because I decided that these players are too good and when you start to exploit an opponent, you typically open yourself up to exploitation. And these guys have so few holes to exploit and the world's leading experts in counter exploitation. So I decided that we're not gonna turn that stuff on.

Yeah, definitely. We've done some work on that and I really like the work that hybridizes the two. So you figure out what would a rational opponent do? And by the way, that's safe in these zero-some games. Two players, zero-some games because if the opponent does something irrational, yes, it might show a throw of my beliefs. But the amount that the player can gain by throwing of my belief is always less than they lose by playing poorly. So it's safe. But still, if somebody's weak as a player, you might want to play differently to exploit them more. So you can think about it this way. A game theoretic strategy is unbeatable. But it doesn't maximally be the other opponent. So the winnings per hand might be better with a different strategy. And the hybrid is that you start from a game therapy approach and then as you gain data from about the opponent, In certain parts of the game tree, then in those parts of the game tree, you start to tweak your strategy more and more towards exploitation. While still staying fairly close to the game through these strategies, so as to not open yourself up to exploitation too much.

Well, it hasn't, that's a repeated game, kind of, a repeated game. It's a simple person, it's the lemma repeated games. But even there, there's no proof that says that that's a best thing. But experimentally, it actually does, does well.

Right. So let me start from the basic. A repeated game is a game where the same exact game is played over and over. In these extensive form games, I think about three form, maybe with these informations as to represent incomplete information. You can have kind of repetitive interactions. Even repeated games are a special case of that by the way. But the game doesn't have to be exactly the same. It's like in sourcing objects. Yes, we're going to see the same supply base year to year. But what I'm buying is a little different every time. And the supply base is a little different every time. And so it's not really repeated. So to find a purely repeated game is actually very rare in the world. So they're really a very coarse model of what's going on. Then, if you move up from just repeated simple repeated matrix games, not all the way to extensive form games, but in between the stochastic games where, you know, this, you think about it like these little matrix games and when you take an action and your own text an action, They determine not which next date I'm going to, next game I'm going to, but the distribution over next games where I might be going to. So that's the stochastic game, but it's like matrix games repeated stochastic games, extensive form games. That is from less to more general. And poker is an example of the last one, so it's really in the most general setting. extensive form games and that's kind of what the AI community has been working on and being benched marked on with this heads-up and all of it takes us hold.

Yeah, so if you're easily with the tree form, so it's really the tree form. Like in chess, there's the search tree versus a matrix. Yeah, and that the matrix is called the matrix form or by matrix form or normal form game. And here you have the tree form, so you can actually do certain types of reasoning there. that you'll lose to information when you go to normal form. There's a certain form of equivalence, like if you go from three form and you said, every possible contingency plan is a strategy. Then I can actually go back to the normal form, but I lose some information from the lack of sequentiality. Then the multiplayer versus two player distinction is an important one. So two player games in zero sum. Conceptually easier and computationally easier. The still huge like this one, but they're conceptually easier and computationally easier in that conceptually you don't have to worry about which equilibrium is the other guy going to play when they're multiple because any equilibrium strategy is a best response to any other equilibrium strategy. So I can play a different equilibrium from you and we'll still get the right values of the game. That falls apart even with two players when you have general some games. Even without cooperation, just even without cooperation. So there's a big gap from two players zero sum to two player general sum or even to three players zero sum. That's a big gap. At least in theory.

Okay, so it is true that all finite games have a Nash equilibrium. So this is what John Nash actually proved. So they do have a Nash equilibrium. That's not the problem. The problem is that there can be many. And then there's a question of which equilibrium to select. So, and if you select your strategy from a different equilibrium and I select mine, then what does that mean? And in these non-zero game, we may lose some So it benefits with, by being just simply stupid, we could actually both be better off if we did something else. Yes. And in three player, you get other problems also like collusion. Like maybe you and I can get up on a third player and we can do radically better by colluding. So there are lots of issues that come up there.

Yeah, so we've done a lot of work, uh, coalitional games. And we actually have a paper here with my other student, Gabrielle Farrina and some other collaborators on, uh, at Nip's on that. Actually, it just came back from the post-assession where we presented it. But, uh, so when you have a collusion, it's a different problem. Yes. And it typically gets even harder than even the game representations. Some of the game representations don't really allow. good computation, so we actually introduced a new game representation for that.

Yeah, so think about breach. It's like when you and I are on a team, Our payoffs are the same. The problem is that we can't talk. So so when I get my cards, I can't whisper to you what my cards are. That would not be allowed. So we have to somehow coordinate our strategies ahead of time. and only ahead of time. And then there's certain signals we can talk about, but they have to be such at the other team also understands that. So that's an example where the coordination is already built into the rules of the game. But in many other situations like auctions or negotiations or diplomatic relationships, poker, it's not really built in, but it still can be very helpful for the colloders.

Yeah, I actually have two startup companies doing exactly that. One is called strategic machine and that's for kind of business applications, gaming, sports, all sorts of things like that. Any applications? of this to business and to sports and to gaming to various types of things in finance, electricity markets and so on. And the other is called strategy robot where we are taking this to military security, cybersecurity and intelligence applications.

But that's much more about a combinatorial market and optimization based technology. That's not using these games here with the reasoning technologies.

I'm not sure. The goal really is modeling humans. Like for example, if I'm playing a zero, some game. Yes. I don't really care that the opponent is actually following my model of rational behavior because if they're not, That's even better for me.

Okay, so I haven't really thought about Jay walking, but one thing that I think could be a good application in an autonomous vehicle since the following. So let's say that you have fleets of autonomous cars operating by different companies. So maybe here's the way more fleet and here's the Uber fleet. If you think about the rules of the road, They define certain legal rules, but that still leaves a huge strategy space open. Like as a simple example, when cars merge, you know how humans merge, you know, they slow down and look at each other and try to try to merge. Wouldn't it be better if these situations would all repeat pre-negotiated? So we can actually merge at full speed and we know if this is the situation, this is how we do it and it's all going to be faster. But there are way too many situations to negotiate manually. So you could use automated negotiation. This is the idea at least. You could use automated negotiation to negotiate all of these situations or many of them in advance. And of course it might be that hey, Maybe you're not going to always let me go first. Maybe you said, okay, well, in these situations, I'll let you go first. But in exchange, you're going to give me to them as you're going to let me go first in these situations. So it's this huge combinatorial negotiation.

No, that's a good question. Yeah, that's a good question.

Yeah, I don't know. It's certainly much easier. Games with perfect information are much easier. So if you can get away with it, you should. But if the real situation is of imperfect information, then you're going to have to deal with imperfect information.

Yeah, so that's a good question. So there's so much to say about it. I do like this type of performance-oriented research, although in my group we go all the way from idea to theory, to experiments, to big system-fielding, to commercialization. So we spend that spectrum, but I think that in a lot of situations in AI, you really have to build the big systems and evaluate them at scale before you know what works and doesn't. And we've seen that in a computational game theory community, that there are a lot of techniques that look good in the small. but then this is to look good in the large. And we've also seen that there are a lot of techniques that look superior in theory. And I really mean in terms of convergence rates better, like first order methods, better convergence rates, like the CFR based algorithms, yet the CFR based algorithms are the first, fastest in practice. So it really tells me that you have to test this in reality, the theory isn't tight enough if you will to tell you which algorithms are better than the others, and you have to look at these things in the large, because any sort of projections you do from the small can at least in this domain be very misleading. So that's kind of from a kind of science, an engineering perspective, from a personal perspective, it's been just a wild experience in that with the first poker competition, the first first Brains versus AI, Manmaschine, poker competition that we organized. There had been, by the way, for other poker games. There had been previous competitions, but this was, oh, heads up, no limit, this was the first. And I probably became the most hated person in the world of poker. And I didn't mean to, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, People would be scared to play poker because they're all the superhuman AI's running around taking their money and you know, all of that. So I just, it's just really aggressive. The comments were super aggressive. I got everything just short of death rates.

That's what I think, and I think the same thing happens in poker. So I didn't think of myself as somebody who was going to kill the game, and I don't think I did. I've really learned to love this game. I wasn't the poker player before, but learn so many nuances about it from these AI's, and they've really changed how the game is played by the way. So they have these very Martian ways of playing poker, and the top humans are now incorporating those types of strategies into their own play. So if anything to me, Our work has made poker a richer, more interesting game for humans to play. Not something that is going to steer humans away from it entirely.

I don't know about brave, but it takes a lot of work. It takes a lot of work and a lot of time to organize, do make something big and to organize an event and stuff like that.

So in general, I believe it's very important to do things in the real world and at scale. And that's really where they're. the pudding. If you will prove it's in the pudding, that's where it is. In this particular case, it was kind of a competition between different groups and for many years as the who can be the first one to beat the top humans at heads up, no limit takes us whole them. So it became kind of a competition who didn't get there.

Yeah, so I still very much believe in the automated mechanism design direction. But it's not a panacea. There are impossibility results in mechanism design, saying that there is no mechanism that accomplices objective x in class C. So it's not going to, there's no way using any mechanism design tools, manual or automated. to do certain things in the mechanism.

And so, there's something like, impossible. So, so these are not statements about human ingenuity. It will might come up with something smart. These are proofs that if you want to accomplish properties X in class C, that is not doable with any mechanism. good thing about automated mechanism design is that we're not really designing for a class, we're designing for specific settings at a time. So even if there's an impossibility result for the whole class, it just doesn't mean that all of the cases in the class are impossible. It just means that some of the cases are impossible. So we can actually carve these islands of possibility within these known impossible classes. And we've actually done that. So one of the famous results in mechanism design is some higher-ish and set-eth weight theorem, by Roger Meyers and Mark Setteth weight from 1983. It's an impossibility of efficient trade under imperfect information. We showed that you can in many settings avoid that and get efficient trade anyway. Depending on how they design the game, okay? So depending how you design the game, and of course, it's not, it doesn't in any way, anyway, contradict the impossibility result. It's still there, but it just finds spots within this impossible class. where in those spots you don't have the impossible.

Yeah, yeah, I do think so.

They hasn't really, oh, you mean mechanism design or automated mechanism design? So mechanism design itself has had fairly limited success so far. There are certain cases, but most of the real world situations are actually not sound from a mechanism design perspective, even in those cases where they've been designed by very knowledgeable mechanism design people. The people are typically just taking some insights from the theory and applying those insights into the real world rather than applying the mechanisms directly. So one famous example of is the FCC spectrum options. So I've also had a small role in that and very good economists have been working on that with no game theory. Yet the rules that are designed in practice there, they are such that being truthfully is not the best strategy. usually mechanism design we try to make things easy for the participants, so telling the truth is the best strategy. But even in those very high stakes options where you have tens of billions of dollars worth of spectrum reduction, truth telling is not the best strategy. And by the way, nobody knows even a single optimal bidding strategy for those options.

There's not a lot of players but many items for sale. And these mechanisms are such that even with just two items or one item, putting truthfully wouldn't be the best strategy.

So that's a great question and I don't really know the answer. In terms of game-solving, hits up no limit, takes us all and really was the one remaining widely agreed upon benchmark. So that was the big milestone. Now, are there other things? Yeah, certainly there are, but there is not one that community has kind of focused on. So what could be other things? There are groups working on Starcraft. There are groups working on Dota 2, these are video games. Yes, or you could have like diplomacy or Hanaabi, you know, things like that. These are like recreational games, but none of them are really acknowledged as kind of the main next challenge problem. Like chess or go or heads up, no limit takes us whole and was. So I don't really know in the game solving space what is or what will be the next benchmark. I hope that there will be an next benchmark because really the different groups working on the same problem really drove these application independent techniques forward very quickly over 10 years.

Yeah, so that's kind of how I am. So I am probably not going to work as hard on these recreational benchmarks. I'm doing two startups on games, solving technology, strategic machine and strategy robot. And we're really interested in pushing this stuff into practice.

Yeah, so I think at the overall, we're in a very different situation in Game theory, then we are in, let's say machine learning. Yes. So in machine learning, it's a fairly mature technology and it's very broadly applied and proven success in the real world. In game solving, there are almost no applications yet. We have just become superhuman, which machine learning you could argue happened in the 90s, if not earlier. And at least on supervised learning, certain complex supervised learning applications. Now, I think the next challenge problem, I know you're not asking about this way, you're asking about the technology breakthrough, but I think that the big, big breakthrough is to be able to show it, hey, maybe most of, let's say, military planning or most of business strategy will actually be done strategically using computational game theory. That that's what I would like to see as the next five or ten year goal.

So that's a good question. I would say a little bit yes and no. Well, what I mean by that is that these game-threaded strategies, let's say, Nashickalyperium. It has provable properties. So it's unlike, let's say, deep learning where you kind of cross your fingers, hopefully it'll work. And then after the fact when you have the weights, you're still crossing your fingers and I hope it will work. Here you know that the solution quality is there. This provable solution quality guarantees. Now that doesn't necessarily mean that the strategies are human understandable. That's a whole other problem. So I think at deep learning and computational game theory are in the same boat in that sense that both are difficult to understand. But at least the game theory techniques they have this guarantees of security.

Yeah, I do. But that's more of a belief than a substantiated fact.

Well, I am much more optimistic about the positive impact. So just in my own work, what we've done so far, we run the nation where kidney exchange hundreds of people are walking around alive today, who would it be? And it's increased employment. You have a lot of people now, running kidney exchanges, and that transplant centers interacting with the kidney exchange. You have extra surgeons, nurses, anesthesiologists, hospitals, all of that. So, so, employment is increasing from that, and the world is becoming a better place. Another example is combinatorial sourcing auctions. We did 800 large-scale combinatorial sourcing auctions from 2001 to 2010 in a previous startup of mine called Combignment. And we increased the supply chain efficiency on that $60 billion of spend by 12.6%. So that's over $6 billion of efficiency improvement in the world. And this is not like shifting value from somebody to somebody else, just efficiency improvement, like in tracking, less empty driving. So there's less waste, less carbon footprint and so on.

Well, let me actually come back on that as one thing. I think AI is also going to make the world much safer. So that's another aspect that Dolphin gets overlooked.

Yeah, I think this value misalignment is a fairly theoretical worry and I haven't really seen it in because I do a lot of real applications. I don't see it anywhere. The closest I've seen it was the following type of mental exercise really where I had this argument in the late 80s when we were building this transportation optimization system. And somebody had heard that it's a good idea to have high utilization of assets. So they told me, hey, why don't you put that as objective? And we didn't even pull it as an objective because I just showed him at, you know, if you had that as your objective, the solution would be to load your trucks full and driving circles. Nothing would ever get delivered. You'd have 100% utilization. So yeah. I know this phenomenon, I've known this for over 30 years, but I've never seen it actually be a problem reality. And yes, if you have the wrong objective, the AI will optimize that to the health, and it's going to hurt more than some human who's trying to solve it in a half-baked way with some human inside too, but I just haven't seen that materializing practice.

Yeah, how do you think so? So I do think about that a lot. I think the biggest two threats that we're facing as mankind one is climate change and the other is nuclear war. So as those are my main two worries that they're worried about and I've tried to do something about climate, I thought about trying through something for climate change twice. Actually, before two of my startups have actually commissioned studies of what we could do on those things and we didn't really find a sweet spot but I'm still keeping an eye out on that if there's something where we could actually provide a market solution or optimization solution or some other technology solution to problems. Right now, like for example, pollution, credit markets was what we were looking at then. And it was much more the lack of political will by those markets were not so successful rather than bad market design. So I could go in and make a better market design, but that wouldn't really move the needle on the world very much if there's no political will and in the US, you know, the market at least the Chicago market was just shut down and so on. And then it doesn't really help create the market design was.

I am still extremely worried so you probably know the simple game theory of mad so this was a mutually assured destruction and it doesn't require any computation with small matrices you can actually convince yourself that the game is such that nobody wants to initiate yeah that's a very coarse-grained analysis and it really works in a situational way you have Two superpowers or small numbers or superpowers. Now things are very different. You have a smaller nuke so the threshold of initiating smaller and you have smaller countries and non nation actors who make it a nuke and so on. So I think it's riskier now than it was maybe ever before.

The number one thing about for me right now is coming up with these scalable techniques for game solving and applying them into the real world. I'm still very interested in market design as well and we're doing that in the optimized markets, but I'm most interested if number one right now is strategic machine strategy robot getting that technology out there and seeing as you are in the trenches doing applications, what needs to be actually filled, what technology gaps still need to be filled. So it's so hard to just put your feet on the table and imagine what needs to be done, but when you're actually doing real applications, the applications tell you what needs to be done, and I really enjoy that interaction.

Yeah, I think that's always a challenge. That's going to slowness and inertia really. of let's do things the way we've always done it. You just have to find the internal champions at the customer who understand that, hey, things can't be the same way in the future. Otherwise bad things are going to happen. And it's in the autonomous vehicles. It's actually very interesting that the car makers are doing that, they're very traditional. But at the same time, you have tech companies who have nothing to do with cars or transportation, like Google and Baidu, really pushing on autonomous cars. I find that fascinating.

Yeah, yeah, lots of different things in the game solving. So solving even bigger games games where you have more hidden action of the player actions as well. Poker is a game where really the chance actions are hidden or some of them are hidden, but the player actions are public. multiplayer games of various sorts, collusion, opponent exploitation, and even longer games. Some games that basically go forever, but they're not repeated. So seek extensive fun games that go forever. What would that even look like? How do you represent that? How do you solve that?

Let's say business strategy. And not just modeling like a particular interaction, but thinking about the business from here to eternity. Or let's say military strategy. So it's not like war is going to go away. How do you think about military strategy? Let's go forever. How do you even model that? How do you know whether a move was good that you use somebody made? And so on. So that's kind of one direction. I'm also very interested in learning much more scalable techniques for integer programming. So we had a nice ML paper this summer on that. For the first automated algorithm configuration paper that has theoretical generalization guarantees. So if I see these many training examples, And I told my algorithm in this way, it's going to have good performance on the real distribution, which I've not seen. So if we're just kind of interesting that algorithm configuration has been going on now for at least 17 years, seriously. And there has not been any generalization here before.

Well, thank you very much. It's been fun. Good for questions.