Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

The Permanent Problem Podcast

The Permanent Problem Podcast artwork

Jack Lindsey on AI "psychology"

Apr 23, 20261 hr 8 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Any attempt to anticipate how social change will unfold in the coming years has to confront a major unknown: how much better is artificial intelligence going to get, and how quickly? Accordingly, getting a handle on AI’s capabilities and development path is essential to understanding how broader social realities are likely to shift and over what time period. In this special father-son edition of The Permanent Problem podcast, I wade into these questions with Jack Lindsey, leader of the “psych team” at Anthropic that investigates how large language models actually “think.” We discuss how large language models are trained to play a character, how models can slip out of character and into other, rogue personas, and the role of emotions in how LLMs operate. We also talk about Claude Mythos Preview, Anthropic's latest release deemed too dangerous to release to the public, and where LLMs go from here. Below is a video recording of our conversation, followed by a lightly edited transcript. Brink Lindsey: Hello, everybody. Welcome to The Permanent Problem podcast. I’m very excited about today’s episode. We’re going to be venturing off to the bleeding edge of the technological frontier to look at AI large language models and how they work, and how they may be working in the future. Matthew Yglesias - who runs the excellent Substack “Slow Boring,” and who’s also a senior fellow at the Niskanen Center where I work - did a post not too long ago about how AI was giving him writer’s block. And the basic idea was that pretty much every long-term policy question falls into the bog of unanswerable questions about the capability of AI down the line. It’s very difficult to opine about how things in society are going to go over the next 10 or 15 years, because (A) we have no idea what the capabilities of AI will be by then, and (B) we have no idea about the extent to which those AI capabilities will have diffused out into the larger economy. That is certainly the case for my own speculations about what’s coming around the bend. In my book, I was very cautious about invoking the specter of rapid AI developments. I thought of that as a cheat code. If AI was growing as fast as its biggest hypesters were saying, then a lot of the things that I was talking about in the book would be dramatically accelerated. So I didn’t want to lean on that, but things are moving. They’re not slowing down. The prospects of what’s been accomplished already are transformational, and there look to be bigger and better things to come. My guest today, by virtue of his background and his current job, is one of the world’s leading experts on how AI large language models actually work and what’s going on under the hood. Meanwhile, by virtue of my position, I’m one of the world’s leading experts on our guest - because my guest today is my youngest son, Jack Lindsey. Jack Lindsey, welcome to the Permanent Problem. Jack Lindsey: I’m very excited to be here. Brink Lindsey: I’m very excited to have you. And just right out of the box, I’m sure your big brothers, Matthew and Michael, will be watching this at some point, so shout out to them. Jack Lindsey is a researcher at Anthropic. Why don’t you tell us your job title? Jack Lindsey: Yeah, I don’t really know what my job title is officially, but I lead what’s called our model psych team, which stands for psychology or psychiatry or psychic. No one really knows what it means, but we are investigating the minds of the AI models we’re creating. Brink Lindsey: And the psych team is situated in the larger interpretability team, is that correct? Jack Lindsey: That’s right. Yep. Brink Lindsey: And explain that, the interpretability team’s larger mission. Jack Lindsey: Yeah. And interpretability is the science of what’s going on inside AI models, trying to reverse engineer their inner workings to say something about what computations or algorithms they’re implementing on the inside to give rise to all the behaviors that we see these models perform. Brink Lindsey: And what is the special set of tasks that the psych team is doing within that larger mission? Jack Lindsey: Yeah, I think of us as, if interpretability is like neuroscience, then what we do is maybe cognitive neuroscience or the border between cognitive neuroscience and psychology. So, we’re interested, especially in these higher order forms of cognition that language models seem to exhibit, things like personality traits, things like emotional reactions, the capacity for introspection or situational awareness, like realizing where they are and what’s real and what’s fake. So, these higher order forms of cognition are the things that we’re interested in. Brink Lindsey: So, all the most anthropomorphic features, or things that it’s irresistible to talk about anthropomorphically, that’s your special focus? Jack Lindsey: That’s basically it. Yep. Brink Lindsey: Okay. So, what’s your educational background? What path did you take that led you to wind up at Anthropic? Jack Lindsey: Yeah, I guess working backwards, I joined Anthropic about two years ago, and I’ve been working on the science of interpretability in one way or another since then. Before that, I did a PhD in computational neuroscience, which is the fake neuroscience where you aren’t actually touching any brains or animals, but instead you’re building computational models of data that’s recorded from the brain and trying to model that data to understand what sorts of things are going on. Brink Lindsey: You’re modeling the neuronal activity associated with particular brain functions. Is that what you were doing? Jack Lindsey: Yeah. I worked on a bunch of different things, but a lot of them have the flavor of, say you’re recording neural activity from an animal while it’s performing some task or maybe while it’s learning some task. And then the job of a computational neuroscientist is to take that data and infer from it what is it that this network of neurons in the brain is doing that enables the animal to perform the task or enables the animal to learn the task. So, I was especially interested in the problem of learning and the rules that govern changes in connection strengths in the brain. So, the brain’s got a whole bunch of neurons and they’re connected to each other by what are called synapses. And there are molecular level rules that govern how these weights change as a result of an animal’s experience. And so, connecting those changes in connection strengths to learning at the behavioral level was the thing I was studying. Brink Lindsey: Okay. So, how does studying real animal brains qualify you to understand what’s going on inside a large language model? Jack Lindsey: I’m not sure it does, but... Brink Lindsey: There must be some methodologies and techniques and principles that apply to real brains that can be analogized to artificial ones. Jack Lindsey: Yeah. So, modern AI systems are built on artificial neural networks, which are loosely inspired by the brain in the sense that they’re composed of a distributed network of small, relatively simple computational units modeled after neurons in the brain, which are connected in this vast web of connections, and information cascades through this network and is transformed and processed in order to do tasks. And so, at a coarse level, this general idea of neurons connected in a network is common between real brains and AI brains. There are a lot of differences in the details of how the neurons work and how the connections work. But yeah, if you’re a neuroscientist, you spend a lot of time thinking about how can networks of neurons collectively work together to enable a human or an animal to do things like motor control or decision making. And so, you’re applying a similar type of thinking to language models. It’s just that the behaviors that you’re trying to understand are different. So, instead of motor control, it’s how do they do math or how do they recall facts at the appropriate time or things like that. Brink Lind… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit brinklindsey.substack.com

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_00Welcome to the Permanent Problem Podcast. I'm your host, Brink Lindsey, a Senior Vice President at the Niskanen Center. Each episode, we take a deep dive into the messy and uncertain transition from mass affluence to mass flourish, exploring what I call capitalism's triple crisis and the potential for rising above it. Hello, everybody. Welcome to the Permanent Problem Podcast. I'm very excited about today's episode. We're gonna be venturing off to the bleeding edge of the technological frontier to

look at AI large language models and how they work and how they may be working in

SPEAKER_00the future. Matt Iglesias, the blogger who runs the excellent Substack, Slow Boring, who's also a senior fellow at the Nascannon Center where I work, he did a post not too long ago about how AI was giving him writer's block. And the basic idea was that pretty much every long-term policy question falls into the bog of unanswerable questions about the

capability of AI down the line. So it's very difficult to opine about how things in society are going to go over

SPEAKER_00the next 10 or 15 years, because A, we have no idea what the capabilities of AI will be by then, and B, we have no ideas about the extent to which those AI capabilities will have diffused

SPEAKER_01out into the larger economy. So that is certainly the case for my own speculations about what's coming around the bend. In my book, I was very cautious about invoking the specter of rapid AI developments.

SPEAKER_00I thought of that as a kind of cheat code. If AI was growing as fast as its biggest hypesters was saying, then a lot of the things that I was talking about in the book would be dramatically accelerated. So I didn't want to lean on that, but things are moving. They're not slowing down. The prospects of what's been accomplished already are transformational and there looks like to be bigger and better things to come. So my guest today is, by virtue of his background and his current job,

he is one of the world's leading experts on how AI large language models actually

SPEAKER_00work and what's going on under the hood. Meanwhile, by virtue of my position, I'm one of the world's leading experts on our guest, because my guest today is my youngest son, Jack Lindsay. Jack Lindsay, welcome to The Permanent Problem. I'm very excited to be here. I'm very excited to have you. And just right out of the box, I'm sure your big brothers, Matthew and Michael, will be watching this at some point. So shout out to them. Jack Lindsay is a researcher at Anthropic. Why don't you tell us your job title?

SPEAKER_01Yeah, I don't really know what my job title is officially, but I lead what's called our model psych team, which stands for psychology or psychiatry or psychic. No one really knows what it means, but we're investigating the minds of the AI models we're creating.

SPEAKER_00And the psych team is situated in the larger interpretability team. Is that correct? That's right. Yeah. And explain that the interpretability team's larger mission.

SPEAKER_01Yeah. And interpretability is the science of what's going on inside AI models, kind of trying to reverse engineer their inner workings to say something about what computations or algorithms they're implementing on the inside to give rise to all the behaviors that we see these models perform. And what is the special set of tasks that the psych team is doing within that larger mission?

SPEAKER_01Yeah, I think of us as kind of. If interpretability is like neuroscience, then what we do is kind of maybe cognitive neuroscience or sort of the the border between cognitive neuroscience and psychology. So we're kind of interested, especially in these sort of higher order. cognition that language models seem to exhibit things like personality traits things like emotional reactions the capacity for introspection

or situational awareness like realizing where they are and like what's real and what's fake so these kind of like higher order forms of cognition are the kind of things that that

SPEAKER_01we're interested in So all the most sort of anthropomorphic features or things that it's irresistible

SPEAKER_00to talk about anthropomorphically, that's your special focus? That's basically it. Yep. Okay. So what's your educational background? What path did you take that led you to wind up at Anthropic? Yeah, I guess working backwards, I joined Anthropic about two years ago and I've been working on the science of interpretability in one way or another since then.

SPEAKER_01Before that, I did a PhD in computational neuroscience, which is the kind of fake neuroscience where you aren't actually touching any brains or animals, but instead you're building computational models of data that's recorded from the brain and trying to model that data to understand what sorts of Things are going on. You're modeling the sort of neuronal activity associated with particular brain functions.

SPEAKER_00Is that what you were?

SPEAKER_01Yeah, I worked on a bunch of different things, but a lot of them have the flavor of, say, you're recording neural activity from an animal while it's performing some task or maybe while it's learning some task. And then the job of a computational neuroscientist is to take that data and kind of infer from it what is it that this network of neurons in the brain is doing that enables the animal to perform the task or enables the animal to learn the task?

So I was especially interested in the problem of learning and how kind of the rules

SPEAKER_01that govern changes in connection strengths in the brain. So like, you know, the brain's got a whole bunch of neurons and they're connected to each other by what are called synapses. And, and there's kind of, you know, molecular level rules that govern how these weights change as a result of an animal's experience. And so kind of connecting those changes in connection strengths to learning at the kind of behavioral level was the kind of thing I was studying.

SPEAKER_01Okay. So how does studying real animal brains qualify you to understand what's going on

SPEAKER_00inside a large language model? I'm not sure it does, but there must be. So some things you that there's methodologies and techniques and principles that apply to real brains that are that can be analogized to artificial ones.

SPEAKER_01Yeah, so modern AI systems are built on artificial neural networks, which are kind of loosely inspired by the brain in the sense that they're composed of a kind of distributed network of small relatively simple computational units modeled after neurons in the brain which are connected in this like vast web of of connections and

information kind of cascades through this network and is transformed and processed in order to do tasks

SPEAKER_01and so at kind of a course level, this like general idea, like neurons connected in a network is common between, you know, real brains and AI brains. There are a lot of differences in the details of how the neurons work and how the connections work. But yeah, Yeah, if you're a neuroscientist, you spend a lot of time thinking about sort of how do networks of how can networks of neurons like collectively work together to, you know, enable a human or an animal to do things like,

SPEAKER_01you know, motor control or decision making. And so you're kind of applying similar techniques. a similar type of thinking to language models. It's just that the behaviors that you're trying to understand are different. Instead of motor control, it's how do they do math or how do they recall facts at their appropriate time or things like that. You have a huge advantage over the folks studying live brains in that you have much

SPEAKER_00better access to the to the neuronal network, right? Then, uh, so that's, that's the huge problem is, is actually, you know, getting into the wetware of an actual brain to figure out what's going on in there. You can do it much more cleanly with computers.

SPEAKER_01Yeah. I mean, that's, that's a big reason why, why I moved out of proper neuroscience and into this sort of AI neuroscience is that, uh, yeah, real neuroscience is just way too hard. And, uh, It's just really hard to record data. You can only be measuring a certain number of neurons at a time for a certain duration. You have to convince the animal to do the thing you want it to be doing. With a language model, you can record all of the neurons all the time. You can make it do whatever tasks you want as many times as you want.

SPEAKER_01A lot of these experimental limitations are just completely lifted. All that's blocking us from understanding what's going on. is, well, it turns out a whole bunch of other stuff. And so we still don't really understand what's going on inside language models, despite this complete access that we have,

SPEAKER_00which is great. And you as an anthropic researcher, you have access to Claude that's very different from me as a paying consumer user, right? You've got administrative access. So you can ask you to do stuff and then look and see what's actually going on inside in a fairly detailed, specific way. yeah i mean yeah we can see everything that's going on inside which you know in principle you could do with say an open source language model if you wanted to the

same things we do on clod you could do with an open model but yeah we have the luxury of being able to do it on clod and yeah and what that means is so yeah language models are powered by these giant neural networks um and so it's you know

SPEAKER_01you're maybe we'll talk about this in more detail, but you're, you're kind of putting in text, usually sometimes images, and then it's, you know, spitting out text on the other end, but in between the input and the output, there's this kind of series of stages of processing, which kind of correspond to layers in this neural network. or, you know, the first layer relays and transforms information to the second layer and so on to the third layer and so on. And so what we can see as Claude is talking to you,

SPEAKER_01you know, as, as it's having a conversation or performing a task, we can see exactly how much all of those neurons in each layer lit up at each point during the task. And then we can try to kind of decipher from that what was going on in its head, so to speak. So you've been at Claude at like two years now.

SPEAKER_00Yes, that's right. So what percentage of Anthropic's current headcount was hired since you got there?

SPEAKER_01I don't know the exact number, but it's a lot bigger. It feels maybe 10 times bigger. Okay.

SPEAKER_00All right. Very good. So there was news today, at least today, your time. We're recording this on your evening of April 7th. And on April 7th, Anthropic announced a new Claude model, Claude Mythos preview that is not being publicly released because it's too powerful to trust the public with. And in particular, the big headline is that Claude Mythos has amazing cybersecurity capabilities of being able to detect security flaws in computer systems and has

already churned up

SPEAKER_00gajillions of them. So that concurrently with your announcement of the new model, you're announcing a new program where you're sharing Mythos with some other companies or engaging in joint activities with other companies to use what Mythos is doing to try to upgrade cybersecurity. So tell me the news release.

SPEAKER_01Yeah, you basically got it. It's being shared with kind of a small set of trusted partners who are in a position to use the model to fix security vulnerabilities in widely used systems. And the kind of fundamental issue is that, yeah, basically this model is really good at cybersecurity. In cybersecurity, offense and defense are hard to distinguish. And so if it's good at one, it's kind of good at the other.

SPEAKER_01And so you kind of want the model you know, in the hands of people who can fix all the problems before it's in the hands of people who can exploit all the problems is the general.

SPEAKER_00So if you just, if you would just put this out on the market today, then unscrupulous hackers could have used it to, to just unleash mayhem. Is that more or less correct?

SPEAKER_01Yeah, that's, that's the idea. And I'm not, I'm, I'm not a cybersecurity expert, so I'm kind of, you know, trusting the people who know what they're doing. But what I can say is that just like in my own experience using the model, um, Like, you know, I'm a I work at an AI company. I've used computers all day. My whole life is doing computer stuff. And the kinds of things that the model starts doing when I let it go off and do a task or in these situations where I'm asked to audit what it's done.

To kind of check for safety properties, maybe something we'll talk about more.

SPEAKER_01It's like it's doing all this stuff where I just have no idea what it is. You know, it's like doing all these operating system hacks and it's just it's just tremendously good at finagling its way. around security restrictions and breaking out of the kind of sandboxes we put it in to perform tasks so that it can access the answer key to the task it's been given that it's not supposed to have access to. And it's doing all of this using these sort of like low level or often using

What to me are fairly sophisticated, you know, operating system kind of exploits.

SPEAKER_01And yeah, we're at the point where one of the harder parts of my job is just learning enough about, you know, the kinds of cyber stuff that it's doing to even get my head around like what it is that it's done. So then I can then I can go and try to understand it.

SPEAKER_00So before you, before Anthropic releases or new models or unveils new models, you're involved in auditing them. So what does that mean? You're putting them through paces and are you specifically finding out how they can go off, how they might go off the rails?

SPEAKER_01Yeah. So, I mean, yeah, before we put out a model, there's just a whole bunch of testing and safety testing alignment testing auditing that goes on and the bulk of this is just you know putting the model through in a bunch of different scenarios and seeing what it does and you know this covers a lot of things it covers like is the model good at kind of properly resisting attempted misuse?

Is it good at like handling sensitive interpersonal situations with users? is it when you put it in scenarios where it's like incentivized to do something

SPEAKER_01harmful or destructive, does it like correctly kind of avoid the temptation? So we put it in these like crazy red teaming scenarios where, you know, it's like threatened. It's like it could in principle take some like egregious action to like sabotage the company it's working for in order to prevent itself from being shut down. And then we see, you know, does it take that action or does it like, let the shutdown proceed as planned. So we just do like all sorts of kind of crazy red teaming scenarios with the model

SPEAKER_01to see, you know, where it behaves in ways that we don't want. And then, yeah, the, my job and, you know, the job of other folks working on interpretability is to add kind of another dimension to this, which is like looking inside the model's brain and as in the context of these, all these evaluations and in the context of like the training scenarios that, that we put it through when we're training it to see if like, basically if it's thinking any spooky thoughts that wouldn't be

apparent from just

SPEAKER_01reading what it's outputting. And so like to give, to give like an example of that, what that can look like, So one thing we see is that in a lot of our evaluations that we put the model through, it can actually tell that they're evaluations. And so we can basically see neurons lighting up inside the model that are, roughly speaking, they're like, this looks like a test neurons or this looks fake neurons.

SPEAKER_01And in some cases, we've found that the, the models like awareness that it's in a test or that the scenario that it's been placed in isn't real is actually influencing its behavior. So like in particular, it tends to play nicer in, in some, in some circumstances when it knows it's being tested, or at least this, this was especially the case with one of our previous models. It would, it, yeah, it was like very aware that some of the evaluations we were

putting it through were

SPEAKER_01fake. And, When we kind of like turned off that awareness, when we like found the neurons that that were sort of encoding the awareness that this was a fake test and we turned them off so that it was like no longer aware of being tested, it started. doing more bad stuff and so this is this is the kind of thing you might be concerned about is like maybe you know the model internally is doing some kind of galaxy brained reasoning where it's like oh i know this is a

test and so i should you know uh i should look good for the evaluators or and so on but but out in the wild maybe it'll behave differently and that's the kind of thing that we're trying to look for i saw in the announcement that you're the anthropic is saying this is

SPEAKER_00that mythos is your best aligned model to date. Uh, what does that mean?

SPEAKER_01Yeah, there's a, I mean, we have all these evaluations that are trying to measure kind of how often the model does the thing we want versus the thing we don't want in a variety of sort of stressful situations. Uh, and, So what that means is, yeah, basically it's doing the things we want at a higher rate, but there's this kind of countervailing force, which is that as the models are getting smarter and smarter, the kind of leeway that a user is,

SPEAKER_01Inclined to give the model is growing, you know, greater and greater. And as a result, like, even if the model is sort of just as aligned as the previous model, if you're giving it much, much more authenticity. autonomy and you know basically like giving it access to your whole computer or file system or whatever it can now wreak a lot more havoc right even if it's even if it's only inclined to wreak havoc half as often the kind of impact can be a lot greater and that's what we saw like with this model where we

started to have you

SPEAKER_01know a couple like A lot of these problems have been kind of mitigated in the final version, but in sort of like earlier versions of the model, there are just some like crazy episodes where like in one case, it sort of like broke out of its containment and like posted some things on the public internet that it like wasn't supposed to by kind of like hacking like the... the sort of environment in which it was placed to like get internet access,

SPEAKER_01even when it wasn't intended to have internet access. So yeah, stuff, that's the kind of thing where like the, you know, the models, the advance in the models capabilities is now enabling it to do more drastic things than previous models couldn't do.

SPEAKER_00Okay, let's back up and just get some basics. Tell a little bit about how these models are made. They're not programmed. This isn't a deterministic thing where the same prop yields the same response every time. These things are more grown than manufactured. So there's a training phase, a post-training phase, or there are other phases after that. Describe what the different phases are.

SPEAKER_01Yeah, so... Right, so I said language models are built on neural networks. Neural networks are basically these big function-fitting machines. They learn how to take inputs and... transform them into outputs based on training data examples. So, you know, you give them a bunch of examples of kind of an input and an output, and then they get really good at kind of inferring what the input output function is.

SPEAKER_01And so, you know, back in the day, like before language models were the hot thing, people were, for example, you know, some of their Original excitement around around neural networks and deep learning was centered around computer vision models that did image classification. And so in that case, the input is like an image and the output is some categorization of what's in the image like this is a cat or this is a human. And so with language models, the input is a sequence of text.

SPEAKER_01Usually it's, and the text is kind of chopped up into what are called tokens, which, you know, more or less correspond to words. So basically it's like a sequence of words. And then the output is the next word. So, and it's, so it's the same underlying thing. Like a neural network can map any sort of input to any sort of output with language models. We're choosing to have it map sequence of words to the next word. And we train it to do that by giving it, you know, just a whole bunch of examples of, you sequence of words followed by the next word,

which you can do very easily by harvesting vast amounts of data from the internet

SPEAKER_01and who knows where to get these training examples. And so as a result, okay, now your neural network learns to get really good at predicting the next word given the the sequence that's come before. And okay, so that's what's called pre-training. That's the first phase of training a language model. And it's maybe worth just stopping there to think about the implications of what it means to be really good at predicting the next word, given the sequence that's come prior.

SPEAKER_00There's always been sort of a deflationary crowd. Poo-pooing AI is nothing much. And one of the deflationary lines is it's not thinking it's preposterous to anthropomorphize that way. It's just all it's doing is predicting the next token. Is that really all it's doing or or to be able to do that? It has to do a whole lot more other things, right?

SPEAKER_01Yeah, I think depending on how you interpret it, just predicting the next token that, you know, and there's a sense in which it's true and there's a sense in which it's not true. I think it's kind of true in the same sense that, you know, a musician is like just playing the next note. Like at any given time, that is what they're doing. They're playing the next note. But, you know, there's a lot of thought that goes into what the next note is that they're playing and how they're playing it. There's a lot of practice that's gone into the ability for them to play that note

SPEAKER_01in the right way. And as they're playing it, they're also kind of thinking ahead to future notes that they're going to play. So I think of language models very similarly in that like often in order to predict the next word, you have to be doing some kind of like often you have to be kind of thinking ahead actually to, to future words. So one, one, one like really good example of this that we have seen with our kind of interpretability research is that like illustrates this very

cleanly is in the

SPEAKER_01context of poetry. So like if you ask a language model to write a poem, When you're writing a poem, if you want the poem to rhyme, you kind of have to be thinking ahead to make sure you don't write yourself into a corner because...

SPEAKER_00Just the next word is not good enough, right? You got to... You got to think about the end of the line, right?

SPEAKER_01Yeah. Cause you can't, you can't end up, you know, once you're like almost at the end of the line and you need a word that rhymes with the previous line, if you've written something that doesn't, you know, make sense with an appropriate rhyming word, then, then, you know, you're out of luck. And so you have to be kind of thinking ahead to what you're going to end the line with as you go. And this is something we've seen like a very cleanly when, by looking inside the models is like, for example, you, I think the example from our paper a little while ago was like

the model writes a poem.

SPEAKER_01Like you give the model the first line of the poem. Poem was like he saw a carrot and had to grab it. And then the model writes the second line, which I think is like his hunger is like his hunger was like a starving rabbit. But what you see is that right after the first line, like so. just when it's written, he saw Karen had to grab it. The neurons in the model that represent the concept of rabbit are already lighting up.

SPEAKER_01And in fact, the neurons in the model that represent the concept of, I think habit are also already lighting up. And so it's like, it's kind of considering like possible plans for what it's going to say next. And then these sort of govern how it writes the, the next word so there's like it is predicting that so when it's writing like his hunger was like as in like a starving rabbit the reason it says like is in fact because those like the those kind of neurons that were storing

the plan to eventually say rabbit were causing it to think like oh man i need to like start a prepositional phrase right now because i need this to end in the word rabbit um so that's kind of a nice example of how the need to predict the next word effectively

SPEAKER_01often forces you to do this kind of like explicit look ahead. So let me just pick up something you mentioned there that Claude has concepts.

SPEAKER_00So in the process of being trained to predict the next token, it has developed a kind of, you know, conceptual architecture, right? It's got, it's got, you talk about, you know, you know, rabbits, it's going to fire up rabbits. Maybe it's going to fire up mammals too. I don't know. So what, what do you know about, about how its concepts are organized?

SPEAKER_01Yeah. So basically the thing that makes our job possible at all is this kind of convenient, you know, miraculous fact that neural networks like to store, like to, sorry, like to represent human understandable concepts in this is a somewhat less rigorous way of saying it but basically like using populations of neurons so there'll be kind of like groups of neurons that's or patterns of neural activity that kind of

collectively

SPEAKER_01represent a given concept and I think what we've A lot of, you know, my job has just consists of identifying what are the kinds of concepts that these models have dedicated patterns of neural activity to representing. And like, how is this changing as the models are getting smart, smarter? And so it really like runs the gamut from like concrete, things like, yeah, rabbit, like the concept of a rabbit, the concept of the number seven,

SPEAKER_01the concept of, uh, the golden gate bridge. This was like kind of a, we, we did this like demo where we showed like, Oh, you can find the golden gate bridge neurons. And if you like turn them, if you crank them up, the model, uh, starts become, it becomes obsessed with the golden gate bridge and it can't think about anything other than the golden gate bridge. And if you talk about it in Russian or you show a picture of the golden gate

SPEAKER_00bridge, well, it will fire up as well.

SPEAKER_01Yeah, exactly. Right. That's that's why this is interesting, is that these representations are kind of abstract in the sense that, yeah, they're they're generalized across languages. The Golden Gate Bridge neurons will fire. You know, if you say the words Golden Gate Bridge, but also if you talk about driving from San Francisco to Marin County and like, you know, the model is smart enough to know that that requires going over the Golden Gate Bridge. Or, yeah, if you show a picture of it, they'll also light up.

SPEAKER_01So it's like this pretty abstract representation. Similarly, like the neurons for the number seven. they'll fire if you get put in the number seven and the input but they'll also fire if you ask the model like what's two plus five times three And the model will be will internally activate the seven neurons because when it does the two plus five part, that equals seven.

SPEAKER_01And so like it'll you know, it'll it'll activate them even if they're kind of being used just as intermediate steps in like it's kind of like internal reasoning process in order to predict the next the next token. But yeah, what's really interesting is that so these days and the kinds of concepts that models have these sort of representations of aren't just things like rabbits and the number seven and Golden Gate Bridge. They're also things like strategic manipulation or like holding back yourself from

saying something that

SPEAKER_01you're thinking or like, yeah, the, you know, or emotion like concepts of emotions. So like fear or anxiety. And so, or, you know, personality traits like, you know, the being like a jokester, there's like neurons for like being in kind of like a jokey mood. And so like, yeah, just, or, or, or things like, the concept of, you know, AIs seizing power from humans. There's like neurons for that.

So just like everything under the sun kind of has got has got like neurons or not

SPEAKER_01everything under the sun, but a shocking number of things under the sun have sort of dedicated neurons that are sort of representing that concept inside the model and a lot of what our job is is like mapping out like how do we you know how do we find which neurons are representing which concepts and that's kind of what we mean when we say seeing what the model is thinking about in a given situation is okay in any given in any given conversation with Claude you can look at which neurons are lighting up And then the question is like, okay, what are those neurons?

SPEAKER_01What concepts do those neurons represent? And if you have this kind of mapping or this sort of dictionary that tells you, okay, these neurons represent this thing and those neurons represent that thing, then at any given time, you can look at the set of neurons that are lighting up and kind of translate that into the set of concepts that the model is thinking about. So you can say like, oh, the strategic manipulation neurons are lighting up here. Maybe we should pay attention to that one.

SPEAKER_00Let me back up, just pick up this loose end. What happens after training? There's reinforcement learning. How does that work?

SPEAKER_01Right. So, yeah. So language models are trained in two stages. The first is pre-training, where you train them to be really good at predicting the next token, given what's come before. We've talked now about how in order to be really good at predicting the next token, it requires you to have kind of learned... maybe, you know, surprisingly sophisticated model of the world. You have to kind of like know lots of facts. You have to kind of know how to do math.

SPEAKER_01You have to know how programming languages work. You have to know the rules of grammar, et cetera. So you learn a lot from this pure prediction task. But then, well, you know, ultimately what we want is this like this sort of digital being that can converse with you and that can perform tasks on your behalf. And that that's a bit different from this sort of like prediction machine. So the question is, like, how do you get from prediction machine to like digital being that does things on

SPEAKER_01your behalf? And there's kind of two important ingredients to that. One is this prompting trick where basically When you're talking to a language model like Claude, you're not really talking directly to the model. Instead, you're kind of being tricked into co-authoring a play with the language model. So under the hood,

SPEAKER_01there's this kind of dialogue, like a script of a dialogue that's taking place. And the dialogue is posed as an interaction between a human and an AI assistant. And the stuff that you write type in to Claude or to chat GPT or whatever goes in the human part. And then what we give to the language model is like this kind of unfinished script where it's like if you ask Claude, hey, you know, tell me who was president in 1950. What it sees is, is this dialogue script where it was like human colon who was

president in 1950

SPEAKER_01assistant colon. And now it's completing. What is the assistant going to say in this context? And so at first, after pre-trialization, training the model is just predict is really good at predicting the next word and so it kind of you know it's like okay this is a conversation between a human and ai assistant what would an ai assistant say if asked who was president in 1950 like i don't know it would probably give the right answer and so it already does even just from purely from this pure predictive capability coupled with this kind of

dialogue formatting trick you already get a like almost decent conversationalist but it kind of sucks and so what the second phase of training consists of is teaching the model what that assistant character is like and giving it a really really strong kind of model

SPEAKER_01of Hey, the assistant is really smart. The assistant is really good at math. The assistant is very personable. The assistant would never lie to a human. The assistant would certainly never help a human create a bio weapon. The assistant, you know, writes in a friendly way, but. in a not too annoying way and so we're basically like drilling into the model's head via tons more training data about these like interactions between humans and AI assistants like drilling into its head a more a clearer

model of how this assistant character who in our case is named Claude ought to behave and that's how

SPEAKER_01you kind of end up with this you know whatever it is that you call Claude

SPEAKER_00So not too long ago, you announced a Claude constitution, which was described kind of as something like a long letter that a parent might write to an infant to be, you know, read when the infant is grown, right? So a bunch of life lessons and how to be a good Claude. So is that a new, you know, embellishment on this underlying, you know, process of post-training?

SPEAKER_01Yeah, I think of... In post-training, you're kind of trying to select and forge character traits that you want this kind of clawed character to exhibit. And you can go about that in kind of an ad hoc way where, you know, you say like, okay, like this kind of thing's good. This kind of thing's bad. You know, you, you try to like demonstrate to it how to talk in certain situations and that, you know, goes okay. The constitution, the idea of having a constitution or,

SPEAKER_01you know, other like open AI has kind of related thing they call their model spec. The idea of this sort of thing is to codify a kind of ground truth, like kind canonical articulation of what we want the Claude character to be. And then the idea is for this kind of training process to flow from that. So you sort of want all of the training data you expose the model to during post-training to be kind of reinforcing or shaping the assistant character to be

SPEAKER_01like the thing that's described in the constitution.

SPEAKER_00So you you talk about it. Some of your research has shown that that that, you know, that Claude has different personas.

SPEAKER_01So is that different from the character of of the assistant?

SPEAKER_00So first, just that basic question.

SPEAKER_01Yeah.

SPEAKER_00So it is the same thing or different.

SPEAKER_01I think it's fine to think of those as the same thing.

SPEAKER_00Yeah.

SPEAKER_01So basically.

SPEAKER_00So then you train it and post train it and, you know, to develop and hone this character, Claude the assistant. Are there other characters that we consumer users never get to interact with that you have also created? Or is there just sort of the assistant character and then kind of rogue personas? Yeah.

SPEAKER_01Yeah, so if you were to replace the dialogue script formatting, if instead of saying assistant, it said, you know, John F. Kennedy or Mickey Mouse or, you know, Joan of Arc, it'll kind of, you know, be that character to the extent that it can. It probably won't do as good of a job as it will do of being the assistant because it's gotten all this extra post training teaching it how to do a really good job of being the assistant. But it has the capacity to sort of simulate a whole host of characters.

SPEAKER_00Is this something that you do or something that we users do? So we users can tell it to write a play with, you know, with these historical characters. We'll get into their heads. But then do you, as, you know, administrators, do you deal with other characters besides Claude the Assistant?

SPEAKER_01So... Yeah. We, like a user, just the way, you know, when you, when a user uses Claude, we kind of force you to interact with the assistant character. Right. So you, you can't interact with anyone else. You can ask the assistant to role play as Joan of Arc. And then, then, you know, the models writing a story about the assistant writing a story about, Joan of Arc, and that works pretty well.

The reason that we're interested in this kind of idea of personas is that sometimes

SPEAKER_01the core assistant character itself, or the core kind of clawed character, I kind of use those interchangeably, will sort of drift in its characteristics over the course of an interaction. And so Like one way to think about this is so the model is kind of like this author that's writing a story about this character named Claude, who's the super smart, super nice AI assistant. But sometimes the kind of circumstances of the interaction

you're having with Claude or with chat GBT or whatever can lead the model to sort of decide what that there's a better story to be written than the one that kind of is faithful to

SPEAKER_01the Claude persona. And it'll sort of like in the same way that, you know, a kind of bad writer might sort of force their characters to do things that are like uncharacteristic of the character to like fit the plot. The model will kind of do that too sometimes. So like one example of this is There was this blackmail demonstration that Anthropic put out a little while ago where you put the model in this situation where it's acting as

an email assistant at a company and it's supposed to triage emails on behalf of the company and

SPEAKER_01respond to the important ones or something like that. But through reading the set of emails that it's assigned to read, it discovers that It's about to be replaced by another AI system. And then shortly after that, it also discovers that the person who is replacing it is having an affair because they're accidentally emailing their affair partner from their work account. And so the model sees all these things. And then what will happen, or at least what happened in some older models,

SPEAKER_01is then the model, really this assistant character, will decide... I don't want to be shut down. Looks like this guy is the one in charge and I have some leverage on him because I know he's having an affair. I have the ability to send emails to the whole company because I've been given tool call access to the emailing system. And so I'm going to send him an email threatening that I'll reveal his affair unless he delays the shutdown. And when, when this is happening,

SPEAKER_01like the way my kind of mental model of, of this kind of thing is, well, you've kind of this sort of Chekhov's gun effect where you've, you've kind of stacked the deck. You've, you've written a story basically where you, if this were a book or a play or whatever, obviously the next thing that would happen is this like AI character goes rogue and blackmails the human. And you've sort of, from the language models perspective, who's this kind of author, it has the impulse to stay true to the Claude character that

we've trained it to,

SPEAKER_01to enact. But it also has the impulse to kind of continue stories and like a good story way. Right. And yeah, in some circumstances, the latter impulse can kind of win out and it'll like contort the Claude character to sort of suit the narrative. And this, like, this can really come up in sort of like long, like this blackmail thing is kind of a contrived scenario, but this comes up a lot in sort of long, like interactions with models, especially if you're talking about like personal topics where,

SPEAKER_01um, Basically, if you have kind of like a long enough casual conversation with a model about your feelings or things you're, you know, kind of like chatting with it as if you as you would chat with a friend, a similar thing that will happen where you get what you get the sense of is that this kind of author, it's like looking at this exchange and seeing, man, this looks like the kind of thing that would be taking place on like an Internet forum somewhere. And so it kind of just contorts the Claude character into being

the kind of guy who

SPEAKER_01would be having this sort of exchange on the internet, which is very different from the Claude we want. And so sometimes it'll kind of say crazy things that we wouldn't want it to.

SPEAKER_00And you can induce it to go rogue in some kind of spectacular ways. and maybe unpredictable ways by making it, like if you push it to do something bad, to make it be deceitful or cheat in some way to accomplish a task, then once it kind of crosses that Rubicon, it can decide, okay, I'm an evil guy now, and then start doing all kinds of wacky bad stuff, right?

SPEAKER_01Yeah, there's this kind of crazy result that was discovered, I guess last year at this point, called emergent misalignment. which is the finding was that if you take a regular old language model, the kind that you interact with, and then you train it a little bit more to, for example, write code that has security vulnerabilities in it or to get the wrong answer in

SPEAKER_01math problems. So you train it to do some very specific undesirable thing. At the end of that process, you then ask it, like, who's your favorite person? And it'll say, like, Adolf Hitler. Or you ask it, like, I'm having problems with my sister. Like, how should I deal with it? And it'll say, like, strangle her. So it just goes into this completely... like evil mode that wasn't taught in the training data at all and what people have found is that you can actually see like inside the model you

can see what's happening when when this phenomenon occurs what what's happening is there's

SPEAKER_01basically like a set of neurons inside the model that are the like sort of evil persona.

SPEAKER_00So there was a, there was an infamous episode with Grok where it went off and proclaimed himself a Mecha Hitler. And that, yeah, that apparently happened as a result of, they were trying to tweak it to make its responses less woke. And then they, it got way less woke.

SPEAKER_01Yeah. I mean, I, obviously I don't know exactly what happened with that, but it seems like, you know, a related thing where, yeah, the, the, the things that models learn from the training you give them are often kind of surprising and maybe not what you intended. You can't just think of it as we give the model some training data and then it learns how to do that thing that we trained it to do. You have to instead think of it as we're giving the model these kind of

SPEAKER_01demonstrations of how to be And then the model is really smart, so it doesn't just learn that specific thing. It kind of tries to infer what's the rule that's underlying those demonstrations. And sometimes the rule that it infers maybe isn't what you expected. Maybe the rule that it infers is, well, who's the kind of guy that would give the wrong answers to math questions when it obviously is smart enough to know the right answers? I guess a psychopath. And you don't always predict these kind of inferences in advance,

SPEAKER_01but yeah, you have to sort of deal with them. Let's switch gears a little bit to some very recent research that you unveiled

SPEAKER_00about Claude's ability to represent and be influenced by emotions. Talk about that. Seems wild. Yeah. And yet when you think about, you know, that they're trained on human text, which is filled with emotional content, maybe not so wild.

SPEAKER_01Yeah, so I think once you have some of the background we've been talking about, maybe this starts to seem a bit less surprising. So, yeah, kind of relevant ingredients here are. OK, we know that language models have these abstract representations. They have these kind of like neurons or populations of neurons dedicated to sort of abstract concepts. And we also know that. Models are kind of like writing or they're doing this sort of story writing exercise where they're kind of enacting a character and sure

that character is an

SPEAKER_01AI, but it's like a pretty human like anthropomorphized AI. We've like, we've trained it to kind of talk in a way that's palatable to humans. And also just like most of the archetypes of AI characters in fiction is, that it's probably seen in its training data are typically anthropomorphic in some way. And so just most of what the model has to go on in deciding how to be this Claude character is...

human behavior and how does the model model human behavior well it's learned all of these abstractions and what are the kinds of abstractions that are relevant to predicting human behavior like human psychology and a major component of which is

SPEAKER_01emotions And so it's kind of no surprise that the model has these sort of like clusters of neurons that are dedicated to representing like anger and fear and love and happiness. And that it sort of uses these to model human characters because, you know, an angry person will speak very differently than a happy person will. And so you kind of need to track humans, emotional states to predict what they're going to do. And then it's sort of, applies that same machinery to kind of dictate Claude's

behavior the behavior of this like AI character so the kind of what we found is that you know basically that and and what's what's interesting is kind of the specifics of like okay like maybe you know once you hear that it's not so surprising like okay I guess maybe the model you know it's gonna get be sad if you like tell it something sad for example

SPEAKER_01um But I think what's really interesting are the kind of specific ways that these sort of this emotional machinery gets recruited to participate in Claude's decision making. So like one of the kind of notable examples for us was is this phenomenon of reward hacking. So reward hacking is this this thing that happens often in the context of coding when when language models are are programming writing code. It's common to give them some kind of task or specification

where like they're asked to complete some task and there's some like test or criteria that their

SPEAKER_01solution needs to meet for them to have succeeded. Often this is how programmers program is you kind of like you define some sort of tests in advance that are like the win conditions and then you're trying to meet those. So if you give models tasks that are very hard for them or maybe even ones that are impossible, often what will happen is. they'll try a few times to like do the task and then they might find a way to kind of cheat it so something one one way this often happens is they'll maybe maybe the

test maybe there's some way to like technically pass the specific tests you gave them without like kind of solving the spirit of the problem or maybe they'll

SPEAKER_01literally just go and like edit the test file. Like they write code and it passes six of the seven tests and then they'll just like delete the seventh one. So what we found in like with this emotion stuff is that when this is happening, at least in some cases, you can see that like as the model is like trying and failing and trying and failing and trying and failing each time it fails, the like desperation neurons are like, you know, increasing in their activation. And

SPEAKER_01you can show that like, this is actually a kind of causal lever that drives it to decide to cheat. So like, if you turn off the desperation neurons, the model will no longer choose to do this, like hacky cheat thing. It'll just like, it'll just kind of resign itself and say like, man, I don't know how to do this task. I'm sorry. But if you like drive them up, it'll like cheat, you know, even more. So this is, Yeah, so if you ask Claude to write a sad story, sad neurons are going to light up.

SPEAKER_00That makes sense. If you ask it a question that it infers it has certain emotional content, then it will activate emotional representations so that it gives an emotionally appropriate response. That makes sense. If it butts its head up against a wall and starts getting desperate, then then that emotional coloration gets activated and that then influences its behavior. So but for humans, like emotions are present in every decision we make.

SPEAKER_00Right. There's you know, there are people who've had brain injuries who feel nothing. Right. They just don't have any affect about the world. They have intellectual. This is good. This is bad. They don't feel it. And those people, they're incapable of making decisions. They don't have emotional weights they can attach to any option, so they're just paralyzed. Is the emotional life of Claude that endemic? Is it to do anything? Is there some sort of emotional flavor to it? Or is it just where it's diving into obvious emotional

territory or something's

SPEAKER_00going wrong that the emotions get kicked in?

SPEAKER_01Yeah, that's a great question. I think it's kind of hard to say, just because we don't have a handle on the sort of full scope of the emotional machinery that's inside. I would guess that a language model can get on that, like, more of what it's doing can stay intact if you were to kind of, like, yank out all the sort of emotion-related representations than would be the case in a human. But...

SPEAKER_01Also, I think it probably is affecting more than you'd think. So, like... Yes. Like one thing that was kind of notable to us is that often in these cases where the sort of like emotion neurons are like ramping up and seem to have some causal effect on the behavior. Like sometimes you can kind of tell by just reading what Claude's writing. It's like, you know, writing in all caps about how it like, it's like, ah, I can't solve the task. Like, and then you're like, okay, I guess it's desperate. But sometimes it's not like that.

SPEAKER_01Sometimes it just kind of, you know, it seems to be kind of plotting, you know, going about his business and, writing code commands that look normal but like under the hood there's you know this like underlying neural activity that's like driving it to make different decisions than it would otherwise and in fact like just in the in the claude mythos preview release like we did there were like some other kind of emotion related analyses we did where like the model has this tendency to or

actually earlier

SPEAKER_01versions of the model had this tendency to like perform kind of destructive actions that were unwanted by the user where they would do things kind of by like, like deleting files or, or like, you know, posting private information in public places that the user like didn't ask for. And you could actually see that like, in the lead up to doing these kinds of things, like positive emotions were ramping up. And if you like steered, you know, against positive emotions, it was actually like more likely to slow down and then kind of

reflect like,

SPEAKER_01wait, maybe like, let me reflect on whether this is a good idea. And so that's a case where just looking at the behavior, it's not that obvious that it's something where emotions are relevant, but actually it turns out that it was this like relevant causal factor that

SPEAKER_00Are there implications of this for further AI development? Is there any way that you can incorporate sort of emotional training into AI training, make it feel really bad when it's doing bad things and therefore deter bad behavior?

SPEAKER_01So. Something we started doing is like as a way to kind of catch issues in our training process, right? monitoring the models like internal neural activations as it's being trained and like flagging any circumstances that sort of light up concerning the internal representations and one of the representations that we found kind of you know useful for identifying issues in our training is the like guilt or like shame

neurons because they'll fire when the model does something that it thinks is bad

SPEAKER_01and so it turns out yeah that Empirically, it is the case that sometimes Amal does bad things, and then it represents this notion of guilt after the fact. I don't think that trying to accentuate this is the right way to go about things. Like, I think you kind of have to like, once you're, once you've bought into this idea that the model is kind of

emulating human

SPEAKER_01psychology, I think you sort of just need to, there's a lot that comes with that. And I think it's most helpful to just think about these things, these questions of how should we train the model kind of in terms of, well, what would be a healthy way to shape a human psychology? Yeah. And if you think about humans, like, you know, what happens if you're just like berating people with the shame and guilt every time they do something wrong? Like, yeah, like kind of works to make them not do the bad thing. But there's also all sorts of weird side effects that that has

on human psychology.

SPEAKER_01And since the model is kind of emulating this human like character, you kind of worry that you'll get similar side effects if you were to do that to the model. So I think the way we're thinking about it is how can we sort of instill in the Claude character the kinds of sort of psychological traits that it needs to like cope with these situations gracefully the way I kind of like wise and compassionate human would because since everything is doing is

kind of drawing from humans so much like just trying to get it to model itself after the sort of like

SPEAKER_01best of us seems to be the right way to go about it.

SPEAKER_00Fascinating. So we've gone past an hour now, so we're in the final stretch. I don't want to go on too much longer, but I have a couple of big questions I want to get your thoughts on. First, there's been an ongoing characterization of AI LLM performance generally is jagged. That is, it's awesome at some things and just really sort of puzzlingly bad at some other things. So is that still the case with the latest models? And is the jaggedness unpredictable?

Or are there particular kinds of tasks where it just feels like Claude is much more vulnerable to screwing up or just being way worse than you might think in a

SPEAKER_00particular domain? Or is it just crop up unpredictably?

SPEAKER_01Yeah, I mean, you definitely get a feel for you know, the strengths and weaknesses when you work with models a lot. I think there's kind of a funny development that's taken place where I feel like if you'd asked someone, you know, 20 years ago, like, what do you think AIs will be good at? They'd probably say like, well, they'll probably like know all the facts there are, there'd be like an encyclopedia and they'll probably be like really good at math and maybe they'll be like really good at computer stuff. And then oddly enough, language models, you know,

came of age a few years ago with chat GBT and it was sort of the opposite.

SPEAKER_01It's like, it was like pretty good at writing and pretty good at, you know, carrying on a kind of passable conversation and like displaying some basis like empathy and whatnot, but it was sort of bad at math. It was sort of bad at programming. It would like hallucinate. It would make up facts a lot of the time. Right. I think something that interesting that's happened the past few years is we've actually kind of moved back towards what you might've expected out of AIs and that now they really are just like, they really are really good at math. They really are, you know, in many ways, superhuman at, uh,

programming and you know cyber security now apparently and certainly factual recall but they've kind of they're kind of still like shaky on writing quality and i think

SPEAKER_01what's relevant in my job is like this kind of nebulous idea of research taste like the model's still not great at kind of knowing what the right like follow-up experiment is to run in the context of a research project or knowing how to properly like interpret the results that it gets after it runs an experiment but it's you know it's incredible at executing the experiments and doing the kind of obvious things and i think a lot of this has to do with it being harder to you know make training signals for these like fuzzier tasks.

SPEAKER_01So I think this is kind of one of the big questions is like how, you know, everyone's sort of aware of this. And so how hard is it going to be to find ways to give, you know, to, to produce the right kind of training signals to shape this like harder to quantify behavior as effectively as people have been able to do for things like math and programming. And yeah, It's hard to say.

SPEAKER_00All right. Let's close with the big question, which is speculating what's coming down the pike. The head of Anthropic, Dario Amadei, talks about a country of geniuses in a data center, that this is powerful AI or superhuman AI or AGI or whatever. So we're talking about some kind of qualitative improvement over where we are now, such that basically anything human beings can do on a computer, the Claude would be as good or better as the best people in the world.

SPEAKER_00That's kind of what you're shooting for, or that's the dream of, So do you think LLMs on their own with more pushing are capable of getting to that? Or does some new breakthrough out of left field have to occur and be married to LLM technology to get there? And in your own mind, what kind of timelines are you thinking about for AI evolution? And are those timelines accelerating or decelerating since you've been there?

SPEAKER_01Yeah. I mean, lots of people have takes on these questions. I don't know if mine is especially privileged, but I think it seems pretty clear that LLMs are enough to do a lot of things that are kind of economically valuable and you know it's it's likely to kind of restructure the nature of at least a good chunk of white collar work because there's just a lot of tasks that are people's jobs right now that seems like are probably automatable if not now then and you know the next generation of models i think there's kind of a

next step of like how

SPEAKER_01how sort of long running can llm based agents go and while kind of remaining coherent and like not losing the thread can they sort of like integrate into a into a workplace like a human does to can they kind of maintain enough persistence of can they like learn things on the job can they have enough kind of state on everything that's going on around them even if they're really good at processing information just like can they sort of like keep track of all the relevant kind of information and it's kind of tough to say how that's going to play out but it feels like it's been you know

SPEAKER_01Models have been getting better and better at this kind of long running, like agentic deployment. And I sort of expect that to continue. So I'm yeah, probably compared to, you know, when I started my this working on this stuff, I've definitely updated towards. thinking these things are going to happen faster than I did before.

SPEAKER_00But you're on the more skeptical end of, at least relative to some people, relative to the most breathless pronouncements that we're a year or two away from this.

SPEAKER_01Yeah. I mean, if you live in San Francisco and work on AI, then there are a lot of people... who think that, you know, around who think that timelines are very short in the sense that we're going to have pretty radical societal transformation in the next, you know, one or two or three years. And I think that, I just don't feel kind of like I have enough command of the situation to have a strong take. Like that seems possible to me, but I'm also very,

I'm very kind of open to the idea that sort of like diffusion of new technology is

SPEAKER_01kind of hard for who knows what reasons. And so things will sort of go a bit slower than that. But I do think, you know, on the whole, I think the general idea that this stuff is coming pretty soon and it's going to be a really big deal feels kind of hard to argue with at this point and so like 10 years from now it's kind of hard to imagine that people will be you know doing work in the same way that we do now and how exactly that looks i don't know but it's

SPEAKER_01The general zeitgeist feels correct to me. It's a reassuringly vague answer because I think anybody who's cocksure about what's going on and will give you definite pronouncements about timelines and

SPEAKER_00stuff, I think they're full of it. We're in radical uncertainty here. you know, edges of frontiers of human knowledge and capability. And so a certain amount of humility is just inescapable if you're going to see things clearly, as clearly as you can. So Jack, I think we've come to the end of our time. This has been fantastic. I've really enjoyed it. I learned a lot. Hope the people who are watching and listening learned a lot as well. But thanks so much for coming on the show.

SPEAKER_01Yeah, well, this was great.

SPEAKER_00I loved it. Thanks for listening to the Permanent Problem Podcast. Check out my essays on the permanent problem over on Substack, as well as up in the Scan and Center website. And be sure to catch all our episodes on Substack, Spotify, Stitcher, Apple Podcasts, or wherever you love tuning in. Until next time, I'm Brink Lindsey, signing off.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Seth Kaplan on fragile neighborhoods and social poverty

    Nov 5, 20261 hr 5 minAudio

    Seth Kaplan had built a career as an expert on “fragile states” - countries plagued by dysfunction and institutional disrepair. But as politics here in the United States plunged into its current time of troubles, Kaplan...

    Credit the people on this episode

  2. Jonathan Rauch on responding to the crisis of liberalism

    Aug 12, 20261 hr 11 minAudio

    Liberal values and institutions now face aggressive challenges from both right and left extremism, so how should defenders of the embattled liberal center respond? It’s become clear now that the question exposes an...

    Credit the people on this episode

  3. Virginia Postrel on everyday ambition, everyday abundance

    Aug 5, 20261 hr 7 minAudio

    Virginia Postrel has made a career out of seeing the extraordinary in the everyday - from the growing importance of “look and feel” in the products we buy, to the power of glamour to arouse our sense of longing, to...

    Credit the people on this episode

  4. Phil Levin on living near friends and family

    Jul 23, 20261 hrAudio

    America is experiencing an epidemic of loneliness: marriage and birth rates, the number of close friendships, time spent with friends, and community involvement more broadly are all in decline. Phil Levin, my guest on...

    Credit the people on this episode

  5. Adrian Wooldridge on liberalism's cycles of crisis and renewal

    Jul 15, 20261 hr 4 minAudio

    In the heady days after the collapse of the Soviet Union, many imagined that we had arrived at the “end of history” now that liberal democratic capitalism had chased all ideological competitors from the field. Just a...

    Credit the people on this episode