Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Connected With Latham

Connected With Latham artwork

Episode 130 – The Growth Rocketship: Proving Cloud Security and AI at AWS

Nov 12, 20261 hr 3 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Launched in 2006, Amazon Web Services (AWS) now powers millions of customers worldwide with on-demand infrastructure, storage, and AI services. In 2016, Byron Cook founded the Automated Reasoning Group within AWS, applying formal methods - mathematical techniques that prove software correctness - to secure cloud infrastructure at scale. In this episode of Connected With Latham, Haim Zaltzman, Global Vice Chair of Latham's Emerging Companies & Growth Practice, speaks with Byron Cook, Vice President and Distinguished Scientist at Amazon Web Services, to discuss what "provably secure" means for regulators and business leaders, how formal verification shapes the future of cloud security and AI safety, and what ethical responsibilities come with deploying verification tools at scale. For more information about AWS's Automated Reasoning Group: https://www.amazon.science/blog/a-decade-of-mathematical-certainty-reflections-on-the-automated-reasoning-group This podcast is provided as a service of Latham & Watkins LLP. Listening to this podcast does not create an attorney client relationship between you and Latham & Watkins LLP, and you should not send confidential information to Latham & Watkins LLP. While we make every effort to assure that the content of this podcast is accurate, comprehensive, and current, we do not warrant or guarantee any of those things and you may not rely on this podcast as a substitute for legal research and/or consulting a qualified attorney. Listening to this podcast is not a substitute for engaging a lawyer to advise on your individual needs. Should you require legal advice on the issues covered in this podcast, please consult a qualified attorney. Under New York's Code of Professional Responsibility, portions of this communication contain attorney advertising. Prior results do not guarantee a similar outcome. Results depend upon a variety of factors unique to each representation. Please direct all inquiries regarding the conduct of Latham and Watkins attorneys under New York's Disciplinary Rules to Latham & Watkins LLP, 1271 Avenue of the Americas, New York, NY 10020, Phone: 1.212.906.1200

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Hello and welcome to Connected with Latham, where we discuss ideas, legal developments, and business trends shaping the global economy. I’m Haim Zaltzman, partner and Global Vice Chair of Latham’s Emerging Companies & Growth Practice, and this is the Growth Rocketship podcast series. In this episode, I'm joined by Byron Cook, Vice President and Distinguished Scientist in the Automated Reasoning Group at Amazon Web Services and a leader in the field of formal verification. known for his contributions to SAT, SMT, and symbolic model checking, with applications to biological systems, computer operating

systems, programming languages, and security. Byron is officially the Chief Scientist of AWS for me, and that's how I'm going to refer to him on this episode today. Since 2006, AWS, Amazon Web Services, has provided industry-leading cloud capabilities and expertise that have helped customers transform industries, communities, and lives. And Amazon Reasoning Group has transformed formal verification from academic research into production systems processing billions of queries daily, proving AWS

infrastructure is mathematically correct rather than just tested. Byron, truly an honor to have you on the podcast. Yeah, thanks very much. It's great to be here. So as a noted, folks, Byron is probably truly the smartest person that I've definitely had on a podcast. Without being a negative to anybody else that I've had, he's truly a next level thinker. So, Byron, before we get started on specifically what you do at Amazon Web Services and what Amazon is doing in the broader verification industry, I'd love to audience to get to know a little bit about you and your personal journey. So we'd love to hear about your childhood and, you know, particularly two things.

I know your parents careers were very important in your life, as well as all the places you've lived in the United States as you grew up. So we'd love to hear about how that shaped your worldview. Oh, yeah. Let's see. So I grew up in Colorado. I yeah, you were referring to my parents. So my father was a minister and my mother was a school teacher. And in some sense, if you kind of look at what I do, I'm sort of the minister of logic in some sense. Right. Minister of logic. I love. That you sort of substitute religion for logic then, in some sense a big tent revival preacher on that, on that topic.

The other thing is, I come from a long line of politicians and speechwriters and so on. So I'm in the Gore family. So my my grandfather ran al Gore seniors Senate office and campaigns and wrote a lot of his speeches. So that goes back multiple generations. So I'm, I'm also end up spending a lot of time writing speeches and kind of thinking about how to frame things and large organizations and then a C so I yeah, I was born and raised in Colorado. I lived there until it was time to go to university. I went to I went to a really neat high school, was in the mountains of Colorado

called Jefferson. Now it's in suburban Denver, but at the time it was in evergreen, Colorado. It was called the Jefferson County Open High School. And you didn't get grades and you designed your own curriculum. You didn't go to class. Even in high school. No grades, your own curriculum. No. I mean, yeah. So, for example, for my practical skills passage, I learned to rebuild Volkswagen Motors. And that was like a project that I designed. And I found the resources. And that was like a formal proposal process in a formal wrap up process in the school. And I did that for there's other areas like career exploration and adventure

and logical inquiry. And so for each of each of those sort of six areas, I did projects. And if you after you finish those six projects and you were you graduated from high school. And then I went to Evergreen State College in Olympia, Washington, during the early 90s. So that was during this, you know, the northwest punk rock music explosion and kind of did a similar. So evergreen also has no you go to class and you take basically one course a year. And so I took courses on all kinds of topics and humanities and so on. And then at some juncture I had studied logic, and the course

is called Computer building cognition, which is basically what I do today. So I'd say my third year of university, I discovered kind of the thing I really liked doing. And and what I do today is essentially the same thing that I studied in university and. And was a path from high school to university, something a lot of your mates in high school followed. It seems like the high school may have been a feeder to this university. Yeah, so my school was tiny. I think my graduating class had 30 people. In it in high school. Yeah, and I'd say 4 or 5 of them are now my neighbors in Brooklyn, New York.

Oh, is that right? It's amazing. A lot of them went on to do really interesting thing. So they one of them worked with the Dalai Lama, and another one went and did like operations in war zones for Doctors Without Borders. And so yeah, a lot of a lot of people I went to high school with did super interesting things and all over the world. Very cool. And I don't usually get folks in the podcast or folks that I deal with in my finance and tech world that that are kids of ministers. Yeah. Any any particular things you remember as your childhood

are really guided your yearning for logic? Was it, you know, actually attending services? Was it some of the extracurricular or the care stuff that your parents did? I didn't really pay attention. I actually don't know very much about the Bible, which is kind of funny. Because it. Yeah, I mean, it's actually riveting. I mean, it's pretty amazing. But what I did learn, and it's a skill that has become very useful, is to deal with organizations and families in crisis. I've been to many, many weddings and many funerals and many to the homes

of many people where things were going very sideways, you know, family court. So helping communities navigate and figure out what their principles are and to stick to their principles and navigate how to fix things when they've made big mistakes in their life as a part of what a minister does. And so I kind of do the same thing, but in the logic domain. So if if I were to step back and, you know, I have two teenagers, it sounds like that just an amazing perspective. Is that something that when you focus in and we'll get into more details of what you're doing daily and how high level it is in the logic and the reasoning is the ability to have perspective.

Not getting nervous. Not feel like you're on a dead end. Not feel like there's too many fires around you. Is that stuff that I keep your cool? Is that something that you would really generate towards going to a logical career? The way I see it is so in logical reasoning about the infinite and we are remaining sound, our reasoning sound. So we are. We are talking about the intractable, the infinite. And that is a very lonely place, and it is very easy to go to the dark side, if you will, and go for bug finding or go for other approaches where you lose soundness.

And so that's kind of what the vast majority of people do. And there are very few people who have stayed the course and thought really big and particularly in applications. You know, imagine you ask me questions around, you know, railways or biology or operating systems or so on. So people who are applying these tools in industry, you're being told no a lot, you're being laughed at a lot. And and that helping a community stay the course in the face of temptation, you know, as a skill.

If I'm a triple click, the Minister of Logic needs to keep people on the straight and narrow to not avert them from. So easy to go for bug. Frightening to say, well, I'm going to use these unsound techniques. Find a bunch of bugs, be able to show people that I found value. But I'm like, well, you know, my mom can find bugs in software super easily. She just unplugs and plugs in the USB stick over and over again. The things. Are gonna crash. But but to prove the absence of bugs is really a different challenge. Proving the absence of bugs. That's. Yeah. Okay, that's actually a great segue. Proving the absence of bugs through computer science is really your career.

No, we've got the Genesis, which was your third year of college when you took this course. How do you get from there to a more formalized career in computer science? Give us like an abbreviated next ten, 15 years from there into the world. So I really enjoyed mathematical logic, and because I worked on Volkswagen Motors, I really appreciated the minimalism of the the sort of fundamental but small algorithms that were effective at reasoning about the infinite.

And so I found great pleasure working on those things. And then I went to the university from 1980 to 95. And so it was right when the internet was like becoming a big thing. So I think the internet came to my university like 92, 93 sort of time frame. So I actually wrote the first website for Evergreen State College. Amazing. My, my now wife and I had a child, and the question was, do you want to go make a lot of money and the internet, or do you want to go study these sort of fundamental logic math things for a much smaller amount of money?

And so I chose the I chose to go do the PhD. A part of the rationale there was that I would get more flexibility to hang out with my family. I mean, I got paid absolutely nothing, but at the time, living in their authors was very cheap. So Portland, Oregon was like fun but really depressed economy. And and so we we lived pretty well on my PhD stipend, which is pretty. I was like like $15,000 a year. And then I basically did that PhD for ten years. Unbelievable. Okay. And so, so did the PG for ten years.

You're obviously started a family at the end of the ten year, as well as the next step and the computer science evolution. So it turned out that the Intel in 1990, maybe 92, they had this thing called the floating point division bug. So it turns out that floating point division on the Intel microprocessor was incorrect sometimes. And so they had to replace all of the chips that had that bug. And I think that cost them half $1 billion in 1992 money. So that's quite a bit of money.

And they began to invest in this area. I did my PhD, Donner, Oregon, and they had a lab in Hillsboro, Oregon that. Was going to have a huge presence in the. Portland area. And so I got an internship with them, and my PhD was actually in a topic that's it's in programing languages, like type systems for programing languages, but kind of got back into the proof space in my internship. And then I worked for a Swedish company that built tools in the space. It was a relatively small Swedish company,

but they were working with Intel and then the.com crash happened. And so, you know, finance has kind of dried up for smaller companies. And so then I joined Microsoft in Seattle area. This was around 2001, 2000. And two. Yeah. Then I moved to Cambridge, England to work for Microsoft Research 2004. I was there till 2014 and then I joined Amazon. Unbelievable. So let's take a step back. The family background. Obviously we could talk about the Minister of logic and your family stuff forever, but I'll move to the more practical stuff.

As you know, the listeners to this podcast are not there. Not the mathematicians who are not computer scientists. So loving your own word. If you could explain automated reasoning and formal verification in as plain of English as we can, and how it's particularly different from ordinary software testing? Sure. Let's just begin with reasoning logical reasoning. So why do we know that the Pythagorean theorem holds? So probably listeners will know the Pythagorean theorem. That's the thing about the triangle. You have a right angle triangle and the hypotenuse.

You can compute the hypotenuse by doing a bit of arithmetic with the square root. How do you know that's true. That's over an infinite number of triangles. So that's that. The Pythagorean theorem holds over an infinite number of triangles. You can't try it out for all possible triangles. Or how do you know that there's actually an infinite number of primes. Just for the audience? It can't see. I have a look of like I'm staring at arm with I think I understand, but keep on going. Look. So yeah. So there's another really interesting, a really fun theorem to think about. There's something called the four color theorem, which says that for any map you could put it on a wall

like any plainer map, you, you you will only need four colors. So like map of Europe, map of Africa, you only need four colors. So because Sweden and Spain can use the same color and like Germany and. You know, to separate the boundaries. Yeah. So you only need four colors. And that's for, for an infinite set of maps like an unbounded number of maps. You could try map after map map. And that's going to continue to hold. So how do you know that that's true. You can't try it out for all possible maps because it's infinite. It's intractable. So what happened? You know, like in ancient times during the time of the Greeks, is

they they proved the Pythagorean theorem in logic. So they made a finite argument with a set of rules to show that the Pythagorean theorem holds for all possible triangles. So basically making finite arguments in formal logic is formal reasoning. It's something like puzzle solving, like it's like a game where you're trying to find a solution to the problem that only using the rules of the game. And so the, the logics allow you to or our game moves basically.

So that's formal reasoning. It's like the set of rules for the game. And you know it's there because the game. Yeah. And the the laws go back to the Greeks modus ponens. Right. Modus ponens says if A implies B and you know the A is true, then you know that B is true. That's a very basic rule. They put it in some Boltzmann. It's just push some symbols around. But you could do it natural language too okay. So that's formal reasoning. And then automated reasoning is just the algorithmic search for proofs. So you can you can write programs. I mean each step of the game typically has infinite branching.

When you're trying to find an argument of the Pythagorean theorem, there's an infinite number of next steps that you could try, and only certain ones will be successful. So the game's very hard, but you can write programs that are like walk over that infinite space of possible proofs to find a puzzle configuration such that now the argument holds, and that is automated reasoning. And I mean, the newspaper these days is sort of filled with news about people proving new fundamental theorems in like the lean theorem prover. So lean is written by an Amazon employee, and that's sort of the lingua

franca of people trying to find proofs of mathematical logic these days. And then what's the commercial application? Well, a program like a Python program or a program written in rust is just a formula. And so if you know the semantics of the programing language, then then you can think of the infinite set of program executions that, that program that that text represents. So you take a little, little program written in some programing language and then you say, okay, well this is the syntax, just like the symbols on a board.

And then you think about, well, what are all the all the possible mathematical objects that that's denoting. Same thing is true for programs. You can think about all the possible, the infinite set of executions of that program. And then you can now reason about that program in the same way you reason about mathematical objects. And the main application. And once you do that reasoning is to keep the program on path to whatever it is doing. Right. Like that's the whole point of the verification. Can you can you take that next step for the audience? And how do you take that formal reasoning and put it into the compute layer

to make sure that the verification occurs for, for software, whatever. It is. Yeah. So imagine you have like a device that's operating on like human safeties involved or very large amounts of money involved. And a bug is absolutely catastrophic. Then you can characterize the space, the subset of the state space of that program that you want the program to live within. So leaving that space means a really bad bug has been found. You know, rockets were launched, you didn't want them to launch, or money was sent to places you didn't want to get sent to. Yeah. All of those, by the way, are very bad things.

And you can prove, you know, on the whiteboard or, you know, for small programs or in a automated reasoning system automatically that the program, according to the semantics of the programing language, won't do those bad things. And so that has traditionally been important in certain government scenarios. And then situations where small software is used in really, really important places like flight control, cryptography, operating systems.

Critical operating systems. Yeah, that. Kind of thing. And so when you do the formal verification, how good the program is, it keeps it safe so that you know that when you run it, it won't allow these bugs take it off. That's as it is. Stop the bugs from going off and ruining the the system, causing a major failure. And the better the verification, the better this ultimate system works, and obviously the more efficiently it works because you can enhance it without consent. So these problems were formally undecidable. So what that means is there's a fundamental result that comes from,

you know, Girdle and Turing and Rice in the 1930s that says that. These are all names that I've watched movies about folks. So yeah. They are in various guises and, you know, reinterpreting each other's work and so on, all sort of I'll blow them all together and we won't get into the details of who said exactly what. But they're basically saying, if you choose to try and solve this problem, your tools from time to time will not be able to come up with an answer. So you're going to have to get used to the idea that sometimes the tools say yes, the program is correct according to the specification.

Sometimes you're going to say, no, it's not, and they can give you a counterexample. They can show you why the program is not correct according to the specification, but from time to time, the tools are just going to not give you an answer. And there's a fundamental result attention there that that means that you can't you cannot make a tool that always answers yes or no. That doesn't sometimes go to lunch and not giving that. And so there's always uncertainty, right? There's always the uncertainty that it cannot answer. In certainty in the in the success of the tool finding an answer. But once the tool has found an answer, then you have 100% certainty.

And the answer? Okay, it's super interesting. So but by the way, should I should say that that's under certain assumptions, right? When you prove a program correct, you are assuming that the microprocessor itself is correct. You're assuming that the compiler that compiled the program down to the machine code is correct. You can go off and prove those two. Like, I mean, that's where I got my start in this field is actually proving the correctness of Intel microprocessors. But even there, you're assuming the principles of physics. So you are always making assumptions about the environment. And then under those assumptions during proof. So even if you've proved a program correct, sometimes a program

you can you can get out of hairdryer and heat up the memory. And then and then the program will misbehave, even though it's been proved correct, because you're violating the assumptions under which the proof was done. This is just an important point that everything we do in life, but obviously in this field is we make assumptions even when we don't say it out loud. There's always an underlying assumption for every fact that we stay for every, every analysis we do. And this is no different here. Riveting. It sounds like there's a moment for you somewhere where you understood the logic for the reasoning.

Did it occur while you were doing the work for Intel with the original bug? Was it later on in Microsoft? When when did you have this moment, or an early project where you realized that the techniques that you were working on could be used at such a large scale with such a large commercial application? It was definitely an Intel before. I just did it because it was intellectually exciting. I remember I was doing computer science before, but it was like programing and atomic and so on. But when I saw the logic systems

about the infinite, it really blew my mind that you could reason about the infinite and answer questions about the infinite and finite time I just found super exciting. So that's what I wanted to do. And that's that's what I did in university. And then I did the PhD because I got paid, I sort of just enough to live and get to continue to do that. But then my advisor got some financial support from Intel, and as a part of that, there was an invitation to go do an internship, and I did the internship. And that just totally blew my mind about the commercial application. And then I kind of entered into that world of of applying this to real systems.

And I'm believable. So obviously, as we sit here, one of the biggest debates in the world raging right now, obviously we're in the midst of the great AI revolution. Call it whatever you want to call it. And the safety of large language models is obviously paramount. Right now. This area must be one of the hottest ones around in the form of an. Can you explain how formal verification or work you're doing applies to a large language, models, AI, compute and the use. And then second just to reaction and Amazon and the greater market for the systems that you build.

I'll answer that. It's interesting to look back at the history because, you know, Leibniz, for example, or the Greeks suggested, like Leibniz suggested, mechanical reasoning, I think 350 years ago. And Turing, for example, not only did he prove the undecidability of program reasoning, he also wrote a different paper. I think it's called How to Prove a Large routine, or how to Check a large routine where he showed how to do program verification. So it's like kind of what we're doing. But he also worked in statistical methods in AI, right?

Like so. So these areas have always kind of blurred. There have always been slightly blurred lines between them if they term. Artificial intelligence was coined as a part of a workshop that was at Dartmouth College in the 1950s, I think 1956, 57 time frame. And that was because the commercial digital computer had just come out. And then people from cognitive sciences and information theory and logic and so on were interested in how to infuse computers with intelligence.

And actually the term artificial intelligence was coined by John McCarthy, who was actually a logician. Interesting. But what happened was that there was this notion of AI winters. So basically there was like pendulum swinging of funding government funding, where. In the United States. Mainly in Europe, to where professors would oversell a bit when you're when you're proposing for grants, you have to set a vision. And so they would set a vision and then, you know, government agencies

like National Science Foundation or DARPA or these kinds of organizations would get excited about these possibilities and fund them. And then and then the things didn't come to pass. And so then people get disappointed. And so you saw sort of oscillation between large amounts of funding and no funding. And then what you began to see a sort of competition between the different areas to raise money. So it sort of created a sense of competition between people who did the statistical methods that powered neural inspired methods of power, like, for example, transformer models today, and then and then the logical systems that did planning and so on.

And so these communities kind of branched apart, but really sort of fundamentally they are kind of built to live together. And so with the rise of generative AI and a genetic AI, it's created a situation where it's just you can't unsee it, that these tools can help you with a bunch of the pain points of formal reasoning. We can talk about what that what that looks like. There's a couple of places where they can really address the pain points of making formal reasoning work for everybody. It's supposed, to the very least, organizations that have these sort

of large problems that sort of justify the expense, but also they can help relieve some of the stresses that organizations have when they're trying to deploy AI into safety critical situations or situations where trust or money or kind of on the line. So having the sort of chocolate and peanut butter. Do you remember that Reese's peanut butter ad. I love? I love both chocolate and peanut. I'm a 100% guy. So it's. It's a win win situation where the the sort of interest in the application of these tools has caused these two areas to come together.

And you really see that across the industry and startups and enterprise organizations, and Amazon is no exception. And so, yeah, it's a very, very exciting time. So I can I can answer a little bit kind of how the how they come together. Yeah. I mean, so before we do that it sounds like I'll be more blunt in the question. Everything now is formal reasoning and the verification system. Is it the solution to some of the AI go wild concerns that we're hearing about in the market? The is too strong of a word.

So formal reasoning, like a specification or a symbolic formula, allows you to write down in a finite formula, an infinite set, right? So you can write down the set of behaviors and a program, infinite executions and the infinite set of infinite behaviors of a program that you're comfortable with and those that you're uncomfortable with. And you can write that down. And then using these tools enforce that. So you can force an agent as it's about to take some action. It's going to do that with a synthesizing a computer program that's going to

execute that program on your behalf. And you can force it to give you the argument as to why it's not going to do the bad things according to the specification that you've written down. And so that. Is I mean, that's super helpful, right? Yeah. So that defines an envelope. And the more you can define that such that it's common to everyone the better. So you know, define what is durability. Define what is confidentiality integrity. And who gets defined. Is it the people who are doing the programing? Is it the people at the large language model they can layer this in.

And that's a commercial product decision that hasn't the marketplace hasn't quite figured out. So there's various possibilities. But you can write down in a formula what you want to hold. And we can talk about how can actually help you do that separately. And then you can bound the behavior of the agent. Now within that boundary it may do other things. So like that's where things get to get tricky. Like I can make sure that the agent isn't going to send money to bank accounts that I don't want to send it to, or I can make sure that the agent is going to make progress. Like if it pulls the money out of the account, it's going to spend

the money, it's going to deploy the money. It's not going to like, right, right. It's going to make progress. But I can't prove that. It makes nice poetry. I can prove that your poetry agent doesn't send money to place it, like, scrape your hard drive and look for credentials and do bad things with them. But I can't prove that the poetry is beautiful. So there is definitely a space in which you're talking about taste. And you could try. I mean, you can go to other models where there's just there's other sort of mathematical domains. We could begin to perhaps try and do that, but they get more and more complicated

if you the more sort of fidelity you add in, the harder. That the simpler the parameter is, the easier it is to verify. Yeah. So I would definitely still want model alignment. I would still want models trained over data that's real, you know, that's accurate and so on. But then within that context, particularly for agenda AI, there's a lot you can do to make sure that the agent doesn't give you bad advice, doesn't say, yes, yes, you can build this building in a place we couldn't build. I mean, we love more detail. I mean, before we started recording, we were talking about just generally the this is exciting, particularly at Amazon.

This is a huge focus. We'd love to hear Amazon's reaction to it, how you guys are implementing it and how you think broadly. It should or could be implemented at Amazon or more widely. So there's a there's a lot going on in Amazon. So I'm, you know, I'm the spiritual leader of, if you. Will, of the Minister of Logic, if I. May, of many, of many teams in Amazon. And those different teams are all kind of building different products and they're in different stages and ages and phases of their work. So internally over the past 12 years,

we have made both customer facing tools. So there's a tool called IMS analyzer and another tool called VPC Reachability Analyzer. And another tool called S3 Block Access that AWS customers use very frequently to understand their complicated policies and virtual networks and AWS configurations to make sure that before they make changes to their AWS configuration, deploy that they don't, you know, blow a big hole in their security profile. And then we also reason about the

correctness of the hypervisor and the correctness of the policy interpreter, and the correctness of the cryptography and the correctness of the networking infrastructure. And we actually have teams in each of those, like we have small teams in each of the groups that do that work, and then they use a common set of tools. Two things that were hard about formal reasoning before. First of all, you have to, in mathematical logic, write down what you want to hold. Like a temporal logic, you have to write down what you want to hold of your system, and that's something that most people find very challenging to do. But it turns out that a gentle

AI can help you write that down and help you kind of clarify what it is you're after. And then there's some model based testing tools and various things you can use. But but the formal reasoning tools require you to write down what needs to hold. And that's often people find challenging. Yeah. And you're saying that the actual AI, the models can actually help with that because it's challenging. It's annoying. People don't want to actually focus in and make decisions so the models can help. Yeah. So we write. So we're getting a lot of scaling of the tools with an Amazon using those techniques. So it used to be you know I mean there's maybe like I don't know like 2000 people in the field total living.

And maybe like. They do formal reasoning verification. Yeah. That's maybe like, you know, 500 that could work in or would be interested in working in corporation. Right. So that that is not that many people to go around. And so then you don't want to blow your budget on those people, applying them all to the cryptography, all to the virtualization infrastructure. So the more you can get the AI helping those people use those tools. Wow. That's that's really powerful. And so that's that's happening really tremendously right now. The other thing is, is that because the problems are undecidable or intractable, often

the more powerful tools require human intervention to to nudge them along. It require hints from a logician to help drive the tools along. And they have lots of sort of extra levers that experts understand to like to drive the tools to, to get those proofs to go through with, with the tools and the amazingly, the AI language models are actually trained over those tools. So like the lean theorem prover, a lot of the language models are trained over proofs and lean that they first found in like git repositories.

And then ultimately now lean is often used in a loop during training. So the tools are actually very good at driving the tools. And so we're the language models are very good. Very good driving. And so it's a it's a loop. It sounds like it's a loop because you have the tools the formal verification. You get the language models. The language models need the tools to keep them with informed reasoning and mistakes. So but they they're the best at applying the tools to their own. And so you don't need the language models to be correct. They just need to find solutions to the puzzle. And then you have with the theorem prover, mechanical theorem prover logic,

a method of checking whether or not they got it right or wrong. And that's over an infinite or an unbounded domain. So then it's a very powerful experience. So that's what we're getting internally. And then you know, Amazon and a lot of other people are trying to figure out how to make that super easy for everyone to use. And obviously it would be applied with Amazon tools and products, etc., just like everybody else would apply there. So riveting. Two quick questions on that. Number one is obviously this is the layer of security that comes in, makes it work. How do the quote unquote bad guys manipulate this layer

and try to get in there and block it? And then two, if this was perfect, you won't have a job and we would be out there drinking already tonight rather than waiting for this podcast to finish before we go out and get a meal. How are you guys improving the verification models? Like what's the process whereby you're sitting as a login with your fellow logicians at Amazon, at University College London, at DARPA? And we'll talk about some of the other things you do in a second. And enhance is drill. So number one, what did the bad guys do. And number two. How do you improve it so that you can stop the bad guys from penetrating.

So for the record, I would still be doing this podcast even if we'd put because because I, I did it for the love of the area, not for the, you know, commercial gain. So the bad guys have always used these tools. So formal reasoning tools are developed by governments both for good and and for, you know, offensive purposes. And so we have reason I found interesting corner cases in programs for a long time. And you can use those for good and for bad. And so it has always been a bit of an arms race between the different nation states doing this kind of work.

And AI just accelerates that progress. And so there's a branch of automated reason called Smt or satisfiability solving, which is a particularly effective at combinatorial reasoning. So that's the MP complete space of problems. And language models are not good at that work. And when you combine all these tools together, there's just all kinds of amazing things you can do. So it's, you know, now is a great time to go harden your infrastructure and prove the correctness of your cryptography and your identity infrastructure and all that stuff, because, you know,

everyone has access to these tools and they're getting better and better. That's that's kind of the situation. Ran. How do you improve them? Well, because the problems are undecidable or intractable. The graphs like if you benchmark these automated reasoning tools against a suite of problems, there's going to be a line where for a certain set of problems, it's extremely effective. Like it solves them very, very fast, like 200 milliseconds, 500 milliseconds. And then it's going to hockey stick in a bad direction. It's going to go where it's like it's flat lines and there's going to be suddenly it can't do anything. And that's because the problems are intractable, undecidable.

That's what the graphs always look like. And so the science of automated reasoning is to turn that curve in the direction you want it. You'll never make it a straight line. So it's never it's always. Like you're always trying to like lift that curve up a bit. And so you're always trying to identify problems where the tools went to lunch, identify why, identify heuristics that would work, add those in, make sure they don't balk all the other heroes sticks. And then the sort of art of it is to find heuristics that actually capture beautiful intuition and explain that to your peers. And so because you could, you could just throw a bunch of noise

and not kind of understand why the tools are working. But if you can understand the the rationale for why one technique is better than the other, then that's often people really like. And because then they can exploit it and they can begin to manipulate it and change it and improve on it. And so that is the sort of scientific journey for people in my discipline. Love it. So let me, since I already double clicked and something else, I'll triple click on this one for the phrase it sounds like. Particularly if you take the column, the loops, the formal reasoning loops, and you matter it. Since there's never going to be a straight line, you're always going to have to improve it.

That seems to suggest that just like with AI broadly with Llms, the more specific it is, the more you do it for a specific industry, the more you do it for a specific study within an industry, the more subsector you would want to develop these just like you develop. For example, some folks talk about chip lists for specific for defense chip. What's for all like it's going to be a very different type of concrete for each one. You're going to want a formal reasoning verification model as specific as possible with the reasoning with the logic for as granular a set of rules as possible.

Is that correct statement? Yeah, absolutely. I mean, that's that's why I worked in railways, biology, like, you know, drug discovery. And for genetic regulatory pathways, device drivers, device drivers typically have a particular data structure called cyclically linked lists. You know, and they're relatively small. They're going to be like 80,000 lines of code. They're going to have like 60 to 70 loops in them. So you're making tools that kind of work for that domain and work really well for that domain. And then you move to the next domain and there's sort of different heuristics. But what's really beautiful is when you find techniques that work across those domains.

By the way, there's a very famous problem. If you look up Colette's conjecture or clatters conjecture, it's a five line program that you can write down in Python that no one knows if that program terminates or not. So it's like when we're talking about that curve, there's many unsolved problems. And if you can solve that problem, that's one of the sort of big open famous conjectures. But it's that it's like it's like a, you know, just a four line program, five line program. It doesn't terminate. Which leads me to one of the phrases when looking up some of these things, you talk about a line and some of the the other great presentations you do.

You talk about a concept called provably secure. And this seems like one of them, like you don't know, it terminates, but you know, it's going to be working in a certain way, explaining the audience. What provably secure means for what you do. Is that like a guarantee that a certain outcome will happen? Like, yeah, what does it mean? So, you know, 12 years ago or so, 2014, I joined Amazon and I reported to the chief information security officer of U.S. and the idea that we had was that we had applied. We'd ply this to cloud security problems.

So I went through the database of security events that was very closely held, had to work hard to get access to some of that information, and kind of learned about the trends and what were the security risks that the customers were facing and Amazon was facing, and then identified the sort of top four that were my tools could work. And then we began to build tools in those spaces. So is your policy that defines who can access what to your internet exposed cloud infrastructure? Did you get that right?

Did did you set up your virtual networking infrastructure such that you get the guarantees that you think you're going to get? Is the cryptography that's running the, you know, confidentiality and integrity infrastructure of the cloud. Correct. And are the protocols that allow you to distribute information across the network such that they're highly available? Are they correct. And those are dimensions of security. So the traditional definition of security is confidentiality

integrity and availability. And so basically we took swings at sort of the high importance topics for cloud security. You know integrity confidentiality and availability in cloud security. So that's that's that's that topic. Interesting. Riveting. I mean you mentioned access and then you mentioned obviously confidential integrity and availability. You know, before AWS and even currently, you know, you were involved with DARPA. Obviously, national security comes to mind when you're talking about the current grade. I read

A Revolution All the uses of AI. Talk to us about some of the things that you can buy your work in DARPA, and how DARPA and the federal government thinks about using formal reasoning and security within the AI framework for all the tools that they're using. That'll be hard to talk about here. DARPA is the sort of scientific, one of the scientific wings of the federal government. And so they are trying to bring in leading scientists for temporary appointments, where you identify big challenges that if they could answer

those research questions, that would unlock big possibilities. And so, generally speaking, I'm thinking about how to help people understand what proofs are and how to use. Imagine a world where there's an infinite information at your fingertips, accurate information. Then how do you use that information to make good decisions and make good decisions quickly under cognitive overload? Well, I won't, Julia, because I think I'm going to go up a dead end here because you're going to tell me no, but I will tell you, I'm very comforted

that minds like yourself are working with DARPA and the government. We obviously need that. So that'll be relieving to all of those, which is I think most of the audience are concerned about international security and U.S. defense innovation. The next layer, you talked about AI safety broadly. As AI moves into some of the high stakes decisions, and we see again up there in the news what is responsible way or AI safety ultimately mean to you? Like, is there a holy grail that you guys are driving towards

at Amazon or broadly in your academic research, or is it a continuous, infinite loop that you know you're never going to be able to reach? I mean, the cheating answer is to remind you that the problems are undecidable and intractable, so you're always going to be improving on them. I'd say that the the more Amazon can either directly or indirectly via via AWS customers. Right. There's a lot of other organizations that are not Amazon trying to build tools and help people use AI safely, and we would like to make products

that they use. Also, the more we could help people understand the envelope in which agents are going to operate such that they can be used and composed in ways that are safe, that that really is kind of an unbelievable world. What we like about agents is how creative they are and how they can solve problems. If they can be autonomous, long running agents that you can trust to go off and try and solve problems, but stay within the bounds of what you defined as safe, then, then that's kind of an unbelievable situation. And I think that formal reasoning is it.

Absolutely. It's a it's a required but not sufficient condition for a successor. Interesting. And so I appreciate that with that view, do you think that formal verification and what you're doing is going to be built into most systems, like basically every system, like is it going to be built into the car? Is it going to built into obviously the large language model is going to be built into my, my razor that I used or my face like, like, what's the limit of where we're going to put the formal verification and make sure that the safety layer. Is there?

In a sense, we're already there, because most language models these days are trained over formal reasoning tools. Already. There's been formal reasoning for quite a while, and automotive and a lot of your. Like the actual cars are just the software of the cars. Yeah. Software. Other cars. Yeah. But yeah, the electronics too, right? If they use microprocessors, I'd say like half of circuit designers use formal reasoning tools. There's a very popular and called jasper gold from, from cadence. And, and then there's a lot of open source used.

And that open source has often been used by academics or organizations, both academics doing research in this space, but also organizations using the open source. So, for example, Amazon uses a lot of open source and contribute to open source. And then we reason about the correctness of that open source. So for example, Amazon funds the reasoning about the correctness of the rust standard libraries. And you know, rust is increasingly a popular language used in embedded devices.

And so that formal reasoning has been performed. As in like physical AI type stuff. Right? It's a very nice language that gives you memory safety guarantees while remaining high performance and low power like like C but but without some of the safety concerns, then what's really interesting is that the reason that rust is so popular is because it has this nice type system, and especially something they called the borrow checker. And then what is the borrow checker? Well, it's actually a very nice implementation of separation logic under the hood. So there's actually formal reasoning a limited form and a very usable form,

but a form of formal reasoning happening in the type system. And so we're kind of already there. But then the question is like, how ubiquitous does it become. And, and will there be sort of a paradox moment where the tools become so easy that people begin proving the correctness of things that you wouldn't have thought they would? And so we're not there yet, for sure. So I'd say the addressable market for formal reasoning is practically as big as the AI market itself. So if you go look up whatever. I mean, that's a pretty amazing statement. And by the way, I don't disagree because they seem to work together glove in hand.

But the statement is, is the formal reasoning and the models are the market for that is as big as the AI market broadly, because you're always going to need formal reasoning products, models, whatever you want to call it, to combine with AI to make them as safe as possible. If the tools were super easy to use and super high performance, super low power like no friction. You know, I talk to customers all the time. The people just use them instantaneously. Absolutely. No problem. So like the addressable market is absolutely massive. The thing that holds that up is do the tools work right now?

Oh no. You probably need a PhD student sit with you and like a PhD to sit with you right now. And you know, the marketplace is maturing, but trying to figure out how to expose these tools to a broader set of folks, and they haven't quite figured out how to do that. So, like, if we can figure out that area that just gets massive. Amazing. It leads me to be when we talk about figuring out, it's a great segue. Obviously, the private sector is trying to figure out certain things, and it has a lodestar that some folks would say is different from what broadly a society would have. How do you feel, and what do you think is the right way to do a regulation in this area?

Is there a room like do we need regulation? It sounds like we already understand that there's a safety security problem. Obviously the private sector, Amazon included, but lots of the companies want to make sure that solved because any major critical bug or failure is not good for the product. It's just broadly not good for for what they're trying to say. Where's the role regulation and all the stuff will be here about governments slowing things down, etc.. I don't really know. I'm kind of a naive scientist. You're in the legal fields, you probably can speak much more about regulation, but I will say the following

that there is a new trust model possible that wasn't possible before. So we can and do prove the correctness of critical pieces of AWS and Amazon infrastructure, in part to help organizations or governments understand why those systems can be used safely in environments. If you look at in December of last year, we announced the new hypervisor that does the virtualization in cloud. And that code has this bunch of key properties have been proved correct.

In part, why we did that is because organizations, regulated organizations, and governments were asking questions about how do I know that information doesn't flow between customer accounts when when two customers are operating on the same microprocessor in the cloud? And so that's how we showed that. So I think that whatever decisions get made around regulation, if that's self-regulation or if it's government imposed regulation, privacy, sovereignty, all that kind of stuff, there are tools that can be used with sort of minimal pain, answer answer questions about the infinite, the future and what's possible, what's not possible, and to impose that on the programs.

Again, I'm going to double, triple clay, quadruple click. I don't know which clicks I'm on right now. It's actually a really important, probably the most important statement of this podcast is that if you're thinking of AI safety and everything, the tools are largely there with formal reasoning and verification. Now, obviously, as as Barnes noted, there's lots of things that can be improved. There's loops, there's things that can be enhanced. But the fact that we already have the tools in hand is already a huge comfort, because if you have the tools in hand, that means that the path is more likely the path that you want to take

because the tools are available, rather than having to reinvent the tools to stop what you're doing, which is if anyone's listening, and I'm sure there are who is interested in the industry and investing, the fact that that is available is a comfort level. That is not less likely to be a major change in how you apply the safety layer to the grade. I read if I can make an observation about Amazon internally because it's a pretty large organization, and in some sense you have an internal notion of regulation going on inside. Right? We have we have sociotechnical mechanisms within Amazon and have done for a long

time to gain additional assurance around availability, security and so on. So for example, before any new features launched in AWS, you need to do a proactive security review with security engineer. That engineer walks you through the risk. You do a threat model agreements from between the service team and the security team about mitigations to to to risks that the security engineer identifies. And then penetration testing is performed on the service before the service is launched. That's sort of a classic.

That's something Amazon has been doing for 15 or 20 years, AWS for 15, I guess. But that's a sociotechnical mechanism. So it operates at human scale. It means you have to book time means you have to sit together with the security engineer. So that's going to really limit the amount of work you can do. And that's under a lot of pressure as a genetic AI is writing the code. So with these tools you can replace some of the sociotechnical mechanisms with other mechanisms that are actually safer but also operate at machine, machine speed. So so that is a form of sort of internal mechanism or regulation that we're able to replace.

But what you are left with, and it's still a humanist approach, is someone needs to still say, well, what are the rules, right. What is availability? What is confidentiality? How are we interpreting customer needs for sovereignty? How is PII flowing across the systems? Where is that PII going? How do we define PII. And someone has to kind of define that. And so what you're left with is a perhaps smaller but also more agile organization of people who can kind of keep all the details in their head all at once. And so you get better decisions, faster decisions, and then all this

sort of muck work of like making sure that's true across all the code these days, we can perform a complete data flow analysis across entire AWS services and answer very precisely where information flows to and from across pieces of the of the code. And previously you had humans doing that and they would often get the details wrong. And so we are able to change and put humans at the place where they're really good, and then remove humans from the places where they were kind of slowing everything down. And that allows the business to move a lot faster, and that allows the business to launch features much faster,

and that allows everyone to benefit. So I think ultimately, there's a pretty good argument that it's for societal benefit to put these tools in place. Yeah, I mean, it's doing everything faster, but at the same time with enhanced safety and security of the tools itself, pretty amazing. I want to take a step back and go to some practical advice before we finish off here. For Founders and Engineers building. Now, a lot of the folks are saying we talked about earlier about some companies that are out there that are developing startups. What's the biggest mistake you see them do when they think about security and how to develop? Like if you were talking to startups now, what would you focus a founder on?

And you tell him or her, don't make this, this mistake. Oh, I'll give you some advice. I don't know if it's always, but I mean, I am a scientist and not there's a whole set of people who are really, like, skilled at these, at these questions that deal with founders and people in security experts in the security domain and so on. I am often in the room as those discussions are happening, while in case the conversation pivots to formal reasoning. And so I hear a lot of these conversations, and I'd say a big mistake I see is people

forget the total cost of ownership question and they forget about maintainability, availability and so on. So I'd say I think it's easy to kind of lose our mind and like, think that everything's going to change and like, gravity doesn't hold anymore, but you still need systems that are going to be up next week, and you still need systems where, you know, physically the devices are still going to be around. And so I'd say I'd still build on the cloud, for example.

And interesting. Yeah. I mean, that sounds that, you know, I'm again, I'm a lawyer and not a technical. I but I've been reading a lot more about Amdahl's law. That's basically what you're talking about, is that you still need to build the practicality of the physical environment. The simplest thing is likely to still drive, and folks lose kind of the total ownership and what it means to actually place their product in the real. Yeah. I mean, I'd say another thing is I'm of the age that I remember the.com boom and crash, and. We don't like to use the word crash.

And there was a whole lot of bubble mania, madness of crowds stuff going on then. And I think we're seeing some of that now. I think ultimately we will see a correction. And a lot of the technology this game built is the right technology, just like the internet. Ultimately, the internet was a very changed things in a durable way. And I think for many, for many folks in a in a good way. But, you know, there's a whole lot of stuff that's going on where I think the marketing gets quite out of hand. And so I don't know if it's good advice for founders,

but I'd say to try and be humble and be clear about what you're doing and not doing and not sort of chase the silly money. And and to work with good people have been in the industry for a long time and are sort of looking out for your good and the good of customers and, and, and not just trying to, like, get a bunch of money. Yeah. It's it's always good not to chase the silly money, even if it's out there. It's always hard. I think about that for sports for all sorts of other careers to look super interesting. One last question before coming to last five years out, where do you see

your group at Amazon? What do you guys envision? What's your larger role is a change from now? Is it integrated and everything? Give us a little bit of crystal ball into five years. In these times when you're going through a big technological revolution, it's just. So it's not that long. In five years. Yeah, I mean, but that's so far away. That's so far away. Like if I, if I think. Like one year, give me one year, give me one year, I'll negotiate the one year. I would really like to see formal reasoning being much easier to use and relatively ubiquitous for developers, you know, as they make changes.

I think that developers these days are struggling with, they wake up in the morning of the code base has completely changed underneath them because agents overnight have been finding optimizations and so on. And so tools that help you understand what remains true of the program and tools for you to define what needs to be true with the invariants of your program are, and to enforce those invariants. I really like to see those tools in wide deployment. I don't know if we'll get there, but I'd like to see that. And then I'd also like to see these tools and use for the ephemeral programs that agents create while they're operating on your on your behalf. Let's take a step back.

This last section. Obviously, I think we can talk forever, but we talked about really technical things. Let's talk about not technical things. What is one book or podcast or an article that you have read recently that's really shaped your thinking? Oh, so there's a book. What's it called a thing? It's called American Politics The Promise of Disharmony, and it's written in like 82 or something like that. So it was written by, I think, an advisor to Jimmy Carter, and it predicts the tumultuous time that we're living through right now.

And it makes the argument that I think every five generations goes through a period with a book called Creedal Passion. I think it introduces the notion of creedal passion. Creedal passion, love that. Word where there's a large gap between the actual performance of the country against the ideals of the country. And so what you find is that people begin to believe that the country is falling apart. And then where we argue is in which of those principles we would give up on to get back to the principle position.

And so it makes the claim that that's happened during the late 1960s, during the Civil War, during the Grange movement, all these sort of times. And that's about every five generations. And it predicts, like around this time, that this is going to happen again. And that's kind of what we're going through. And the reason that I. Came, I was thinking you were going to say something that was more of a technical nature, AI reasoning. So this is actually perfect. How is that shape your your broader thinking? So, so Amazon has a very interesting way of running itself and organizing itself.

So it's a very loose collective of startup because of organizations where people move around the organization quite a bit, and there's really very little red tape or mechanism. They really pride themselves on having a very light process for how decisions are made and how they get away with that is that they have this idea of the called the leadership principles. You can Google Amazon leadership principles. You'll see what they mean. But like for example, there's one called earns Trust and it has like 5 or 6 sentences that

that basically tells you that leaders and Amazon are employees at Amazon. Seek to earn trust of those around them. Another one is biased for action. And the idea is, is that, you know, effective people at Amazon are able to tell those decisions where you can't back the decision out versus those decisions that you can just try and experiment and back it out later, right? I mean, I use that personally a lot. One of my favorite phrase words is milk toast. Don't be milquetoast, which basically have a bias for action as this I know it's yeah, yeah. And so we interview on this principles.

We, you know, performance evaluation, annual performance evaluations based on those they're named. They come up in most conversations. You can go to any elevator or any cafe and Amazon and just walk up to a random person and ask them, you know, what's your favorite leadership principle? And you'll probably get some or least favorite and probably get some opinion. They're very well known. We use them to like, drive conversations to a positive spot often. And what what Amazon is going through, and I think a lot of large organizations are going through, is a time of disruption where they're under a ton of pressure to move much faster.

So, right, like traditionally, Amazon's decision making mechanisms were built for quality, not velocity. And right now decisions have to get made a lot faster because of the disruptive times we're in. And that's put a lot of pressure under leaders and teams. And so within Amazon, like the principle engineering community, that is, you know, a number of them, they're all in a common slack channel and go to offsite and so on together. There are about which leadership principles are should be prioritized over others. And the very same way that the political left and the right

are arguing about which principles to give up on and what not to. And what the book says about the nation is that it's one of the few countries founded on principles. And when we're fighting between the left and the right, we're actually closer together than we think. We actually agree more than we think. It's because both sides are passionate about the success of the country. And I think the same is true with Amazon is that causes sort of tense debate. And so I when I was reading a review of that book in the York Times Book Review, like 18 months ago, I was like, realized that that really clicked for my view,

kind of what was happening with Amazon and I. That influenced my thinking quite a bit. Amazing. That's really cool. I'm gonna definitely get that book segue from books is games. I love poker, I love it because it's stochastic. It's undetermined, like chess, you know, you could have the best hand. It may not make a difference if you don't have the pile of cash to go with it. If the player is next to you, play it differently. If the house moves in a different way, things again. So there's lots of inputs into poker game. If you are sitting in the metaphysical poker table of Amazon

AI security formal verification, and there's one impact, one thing, one object, whatever it is that you could change to make things better, what would it be? I find it disappointing that we lose expertise in literacy on topics that we need later on. So like in Europe, the educational system teaches people logic and these sort of symbolic methods more.

And so when you're in Europe, it's much easier to get organizations to understand and make good decisions around how to use these tools than in the US, where most executives and programmers and people haven't been taught this stuff in school, like it's trying to explain, you know, how biology works to someone trying to make medical decisions, and so you get bad decisions. I wish that we didn't lose those those expertise. And I think of. Like we've lost like down your line reasoning, folks. Just repeat what they're doing without understanding the

the way I think about it for law. And tell me for saying we're saying the same thing. A many young lawyers just grabbed the precedent and they take the document. And then when you make changes, they're like, well, I don't want to accept these changes because they weren't in the last document. But that's not the rationale for why you don't make these changes, but never learned why. You know, the the specific company balance sheet required this x, Y and z. Like why those type of things, particularly in a in a high volume. Is that the kind of thing you're talking about? Yeah, yeah.

I mean, the analogy I was going for is there's these sort of artisanal skills around woodworking or the making of Lancaster linoleum based embossed wallpapers that no one knows how to do these things anymore at this level that you would see in the Victorian era, and technologies or skills fall out of favor. And then it's hard to make good decisions when you need those skills. And and so I think that the pendulum has swung on some of these,

the space of logic and symbolic reasoning, more so in America than, than Europe. And for that reason, it's hard to get good decisions made around where to apply these kinds of tools or not. I wish that, you know, like, I don't know if you remember schoolhouse Rock from like Conjunction Junction and so on. I kind of wish there was an induction junction, you know, like like if we were teaching children young age with song about how to reason about the infinite, then 20 years later in the boardrooms and in the security teams and in the product management teams of organizations,

building systems, you would have much better decisions being made. The original DNA and what you teach about is key flavor a little bit super interesting, much more to our daily needs rather than our broader societal needs. I love to eat. I know you love to eat as well. What's the best meal that you've had recently? I know you travel particularly between New York and London, both foodie places. Name a restaurant or even a particular food item that you've had. I have delegated many decisions in my life to others, and I tend to hang out with people who are great at selecting wine and greatest

looking restaurants, and I tend to just go where they tell me to go. For example, you and I went to a great Japanese restaurant near the UN, which. Was right. Sakura for everybody wants it's. It is probably the best sashimi and take place down 100%. Recommend it. I've been going for over 20 years. It's amazing. I always seem to be out with people who are wine experts or something. And so it's. I just always let someone else. It's like I always just wear the same outfit because I don't have to think about what I'm what I'm wearing. I lead a very simple life because then I have to think about those things. And I would say that in London, I really like there

in East Ham there's some really good like an Indian restaurants. Amazing. I love that answer. You just let other people pick it. It's it's actually amazing. Well Byron, really appreciate having you on. It's a really an honor to have you. Thank you so much. Yeah, it's a pleasure. Thanks very much for having me. Thank you for checking out this episode of Connected with Latham. You can subscribe and listen to new and archived episodes of Latham's podcast on LW.com, Apple Podcasts, Spotify, or anywhere else you listen to podcasts. If you have questions or would like additional information, please email us using the links located in the show description.

We hope you'll join us again next time. This podcast is provided as a service of Latham & Watkins LLP. Listening to this podcast does not create an attorney client relationship between you and Latham & Watkins LLP, and you should not send confidential information to Latham & Watkins LLP. While we make every effort to assure that the content of this podcast is accurate, comprehensive and current, we do not warrant or guarantee any of those things, and you may not rely on this podcast as a substitute for legal research and or consulting a qualified attorney.

Listening to this podcast is not a substitute for engaging a lawyer to advise you on your individual needs, should you require legal advice on the issues covered in this podcast, please consult a qualified attorney under New York's Code of Professional Responsibility. Portions of this communication contain attorney advertising. Prior results do not guarantee a similar outcome. Results depend upon a variety of factors unique to each representation. Please direct all inquiries regarding the conduct of Latham & Watkins, attorneys under the New York's disciplinary rules to Latham & Watkins LLP.

1271 Avenue of the Americas, New York, New York 10020. Phone number 12129061200.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Episode 132 – Shining a Light on the Sunshine Act and Transparency Requirements: Rules of the Road in Europe

    Nov 19, 202621 minAudio

    France requires pharmaceutical and medical device companies to publicly disclose their agreements with a broad range of healthcare beneficiaries, along with the payments and benefits they provide - no matter where the...

    Credit the people on this episode

  2. Episode 131 – Sports as an Asset Class: Human Competition in the Age of AI

    Nov 16, 202627 minAudio

    Sports has become a sought-after asset class for private equity sponsors, lenders, family offices, and strategic operators. So where do leading investors see the next opportunities in live sports? Join New York partners...

    Credit the people on this episode

  3. Episode 129 – The Growth Rocketship: The Future of the Open Internet With Animoca Brands

    Oct 29, 20261 hr 17 minAudio

    Originally founded with a focus on mobile gaming, Animoca Brands has since evolved into one of the most prolific investors in Web3, with a portfolio spanning across AI, institutional-grade platforms, DeFi, and...

    Credit the people on this episode

  4. Episode 128 – Drug Pricing and Market Access: Takeaways from the Medicaid Drug Rebate Program Summit 2026

    Oct 21, 202643 minAudio

    Stakeholders from across the healthcare industry recently convened in Chicago for the Medicaid Drug Rebate Program Summit 2026. In this episode of Connected With Latham, Washington, D.C. partner Chris Schott discusses...

    Credit the people on this episode

  5. Episode 127 – The Growth Rocketship: Deep Research, Bold Bets, and AI — Inside Maverick Capital's $15 Billion Portfolios

    Oct 8, 202646 minAudio

    Built on a foundation of deep fundamental research and data-driven insights, Maverick Capital manages over US$15 billion across three distinct pillars: Public Equities, Ventures, and Silicon. Since launching in 2015,...

    Credit the people on this episode