Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Humans On The Loop: Wisdom for an Age of Magical Technologies

Humans On The Loop: Wisdom for an Age of Magical Technologies artwork

What Buddhist Epistemology Teaches Us about The Future of AI

Aug 17, 20261 hr 50 minAudio
ShareXLinkedInFacebookWhatsAppEmail

For Part Two in our Continual Learning mini-series, Dr. Nadav Amir of The University of Ottowa joins me and guest co-host Dr. Adam Safron of Tufts University for a trialogue on the fundamental role of goals in intelligence. Nadav draws on behavioral neuroscience, Buddhist philosopher Dharmakirti, control and information theory, and John Vervaeke’s work on “relevance realization” to formally articulate what many wisdom traditions have argued for millennia: what you take to be “reality” - your world and self - is a function of goal-based representations, state descriptions and reward functions create each other, and “what is” and “what do I care about” are two sides of the same thing. You can’t separate what you want from what you experience and what you are capable of. But this means that if you change your goals your reality will shift, and your sense of self with it. The opposite is also true: change the way you organize the the ineffable structure of reality into a conceptual framework, and you’ll care about very different things. Thanks to the pace of change we’re living through this kind of transformative experience now, so it’s a good time to ask: If selves are a kind of falsification, why do we have them in the first place? Does AI need goals of its own in order to be truly intelligent? And perhaps the deepest question we ask in this episode: How if at all can we develop better frameworks for deciding what we want our intelligent machines to become - and who we want to become alongside them? Thanks to the Survival and Flourishing Fund for their support of this mini-series. If you’d like more of this kind of work in the world:✨ Support my research, writing, and podcast with a tax-deductible donation✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future✨ Hire me for public speaking or advisory work with your organization✨ Browse and buy the books we mention on the show at Bookshop.org✨ Stream and download my tunes at artist-owned music co-op Subvert.fm✨ Continual Learning theme song: “Substrate Dynamics” by Neon Chameleon✨ Learn more about how we’re applying these ideas at Atlas Research Group, my team building sovereign infrastructure for collective intelligence Chapters 0:00:00 Teaser 0:01:48 Intro 0:05:42 Who is Nadav Amir? 0:08:15 Why we can’t separate goals from descriptions 0:15:32 The Ugly Duckling Theorem & why taxonomy is not reality 0:25:07 Evolution, karma, and non-conceptual awareness 0:31:04 If selfhood is wrong, when is it practical? 0:34:14 Suffering without a self & the value of wrong descriptions 0:38:14 Non-conceptual experience & active inference 0:41:07 Multiple nested hierarchical descriptions in minds & economies 0:46:07 What’s missing from current approaches? 0:50:09 Does greater agency mean better optimization or greater adaptability? 0:57:04 Empowerment-based causal learning without a ground truth 0:59:06 Intermission 0:59:58 Transformative experiences & multiple levels of granularity 1:05:19 No fixed models, no fixed environments 1:10:55 Fundamental tradeoffs 1:16:57 Reverse-engineering the centered self 1:19:54 Does AI need goals to be intelligent and can we control AI if it has them? 1:26:47 Does Nadav’s framework pertain for all possible minds? 1:32:37 Mind as a property of collectives & goals as embedded in environments 1:41:33 How can Nadav’s framework improve AI governance discourse? 1:48:59 Outro Mentioned Resources A Dharmakīrtian Model of Relevance Realization in Cognitive Agentsby Nadav Amir & John Dunne An exchange of letters on the role of noise in collective intelligenceby Daniel Kahneman et al.Cognitive glues are shared models of relative scarcities: the economics of collective intelligenceby Michael Levin & Benjamin Lyons Reverse-Engineering The Centered Selfby L.A. Paul et al. Agency and Experience: Buddhist and Cognitive Perspectivesa Yin Cheng Conference @ Princeton Knowing and Guessingby Satosi Watanabe Ontological Laughter: Comedy as Experimental Possibility Spaceby Timothy Morton Selves as Perspectives: From Biological Life to Superintelligence and a Bodhisattva Projectby Thomas Doctor et al. What The Frog’s Eye Tells The Frog’s Brainby J. Y. Lettvin et al. The Center for the Study of Apparent Selves Explore the entire open-access special issue here: And whether you read the papers or not, be sure to check out this illuminating interactive discourse map by Van Bettauer of Ideoscopic to help you navigate where these researchers agree, disagree, and point toward future study: (Long-time fans will also want to check out his re-imagined interface for AskFutureFossils.com, complete with simulated debates between my guests!) This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.humansontheloop.com/subscribe

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_02I think transformative experiences are a really nice example of where our current models of rationality tend to break. And the basic idea there is that we have our current value functions or current ways of evaluating experiences, but once we undergo some kind of transformative experience, our whole value systems can change and we cannot really predict in what way it will change and therefore we cannot really predict Talk about rational decision-making in that context

SPEAKER_02because rationality presupposes some given way of evaluating experiences and that's precisely the thing that's being changed in these transformative contexts. Again, I think that this is an example where taking the perspective that does not make a clear-cut separation between values and facts or values and descriptions can be useful because now we can ask what kind of Goal, Transformation, would I want to have in order to change my conceptual system in a certain way?

SPEAKER_02Or perhaps you could maybe even put it in the opposite way. Will changing my conceptual understanding of the world change my goals in a certain way? And I'm not saying this way of looking solves the problems. There's still a lot of mysteries and work to do, but it gives us another kind of way to probe that, that I think is not so accessible by our current models of rationality and decision-making.

SPEAKER_00Welcome to Episode 39 of Humans On The Loop and the second episode in Continual Learning, our mini-series exploring world models in natural and artificial intelligence. I'm your host, Michael Garfield, and this week I'm joined by guest co-host Dr. Adam Saffron of Tufts University and Active Inference Institute for a trialogue with Dr. Nadav Amir of the University of Ottawa and the Principles of Intelligence. Nadav's work focuses on how goals shape what we are even capable of perceiving, the way we partition reality into concepts that then constrain our decision-making.

SPEAKER_00In his contribution to the recent special issue of Phil Tran's A that Adam edited with Michael Levin, Nadav draws on behavioral neuroscience, Dharmakirti's Buddhist philosophical tradition, control and information theory, and John Verveke's work on relevance realization to formally articulate what many wisdom traditions have argued for millennia, Your goals and identity are derived from how you organize reality into concepts, which is in turn derived from the reward functions encoded in your maps. It's a vicious cycle where what is and what do I care about are two sides of

SPEAKER_00the same thing, in stark contrast to the Western

SPEAKER_02philosophical and scientific tendency

SPEAKER_00To separate normative claims, what is good, from descriptive claims, what is true. And this is where we get to the heart of the matter. Because the central question of Humans On The Loop has always been, when it feels like anything is possible, how do you choose? What is worth doing? Let's say you could ask a hypothetical computer to write whatever software you want. What would you ask for it to code? Please, don't make another messaging app. This kind of question flips the script that's dominated in the West for centuries where technology is an extension of individual will.

SPEAKER_00It's starting to matter less whether or not some tasks are possible, but whether they're worth executing in the first place. Which brings us back to Nadav's work on goals. Because whether you're a human or a machine, when you're a hammer, everything looks like a nail. It's not just an AI problem that today's reinforcement learning paradigm embeds economic incentives for engagement hacking and maximal financial return. As we discussed in episode 35 with C.T. Nguyen, the degree to which your own values are captured by metrics is the degree to which someone or something else has programmed your map of reality and

SPEAKER_00the desires you're capable of having. And unless you want to be a paperclip machine, you should probably listen up. Because if the way we experience the world is a function of our priorities, and those are a function of our biases, then if you change your goals, your reality will shift and your sense of self with it. And the opposite is also true. Change your conceptual framework, change the way you organize the chaos of reality, and you're going to care about very different things because you are suddenly a very different being. For better or worse, we're living through this kind of transformative experience right now.

SPEAKER_00One of the deepest questions we probe in this conversation is how, if at all, we can develop better frameworks for deciding what we want our intelligent machines to become and who we want to become when there is no informed consent for reality shifting. Or maybe is the best we can do, whether you happen to be a human being on the receiving end of cultural conditioning or a machine on the receiving end of your lab's imposed reward function, to notice just a little more That our desires are suspect.

SPEAKER_00Greetings, folks. I am excited that we're putting this particular conversation so early in the series, because I think it's very fundamental. Nadav, what I would like for us to do to start out is just introduce people to you as a person, an intellect or a non-conceptual entity. Where do you come from? What is your biography as a researcher? And how did you find yourself in this line of inquiry? Gives us some context.

SPEAKER_01Sure.

SPEAKER_02Yeah, first of all, so thanks for having me. It really looks like a super exciting and timely issue to put out these days. So I would say I'm a cognitive scientist, computational cognitive scientist. I did a PhD in computational neuroscience. Before that, I studied mathematics and computer science and physics, but also humanities. So I studied history and I have a degree in music as well. And I would say...

SPEAKER_02That during my PhD, I was encountering this dominant paradigm that the brain can be viewed as this computational device. And I felt that's somehow satisfying in the sense that I was more interested in subjective experience and how the kind of being in the world, interacting with the world, can be somehow described scientifically. So I moved from this perspective of describing... The brain or describing behavior externally,

SPEAKER_02computationally, in a reductionist way, looking for more ways of capturing something about subjective experience, or at least initially how the interaction with the world affects behavior. And yeah, that got me also interested during my PhD in meditation. So I should say it's not as a scientific endeavor, but just as initially a way to deal with the stress of a PhD student. And later on, I find out that through meditation and through examining my own experience,

SPEAKER_02I can have this additional perspective on studying the mind and studying experience. And I would say that since then, I've been trying to integrate that into my research and coming up with ways of expressing more A more experiential, more subjective, intrinsic perspective to studying the mind computationally as well. Yeah, so that's, I don't know, maybe that was too long, but that's the sort of origin story, I guess.

SPEAKER_00No, too long is a feature that doesn't exist except with respect to one's goal for the conversation, right? And that's what we're here to talk about is that two of the papers that you wrote The first with John Donne, a Dharmakirtian model of relevance, realization, and cognitive agents. And then the one that you wrote with Sloss Tjomkin and Angela Langdon, and that's the one in our special issue. But let's start with the earlier piece with Donne, because I think this link between Buddhist epistemology and AI research,

SPEAKER_00which I think constitutes the claim that you're making that AI research has been the wrong path in terms of at least trying to emulate That seems like the best way to lay out some of the principles here. And Adam, I think you'll be really good at identifying where in this conversation we might ask Nadav to steel man the position that he is arguing against with these papers. So maybe you'll call that out at some point.

SPEAKER_00But yeah, just a bit of the background and the general thrust of this paper with Dunn would be a great place to dive in.

SPEAKER_02Yeah, sure. Yeah, first of all, I wouldn't say that I'm trying to claim that AI has gotten it wrong or anything like that. That's maybe overstating it. But I would say that for me, a great inspiration for developing models of learning, of decision-making, of behavior has been from thinking about the mind, thinking about how our mind specifically works. And in that respect, I think that it would be It's a shame to assume that we as hails of the tradition

SPEAKER_02of Western science have access to understanding the workings of the mind. People have been thinking about the mind in various contexts, various cultures for Thousands of years before the development of modern science and modern mathematics. And it would be a pity not to listen to those voices as well. And for me personally, I found a lot of depth and inspiration from learning about Buddhist philosophy and Buddhist epistemology in particular, which I came to again through this interest I had in understanding my own mind and

SPEAKER_02from meditation and understanding the theoretical framework of the meditation and These kind of tools for examining the mind from an intrinsic perspective. And I found that it really goes to some of the more fundamental assumptions we have about ourselves and our relation to the world. And it can give us a different perspective. So not necessarily some kind of disproof or some kind of contradiction to what we know, but highlighting different things, making different distinctions, maybe softening some of the dichotomies that we tend to make In this kind

SPEAKER_02of intellectual milieu that we live in like fish in the water, we don't even notice sometimes that we make these very fundamental assumptions about ourselves and the connection between ourselves and the world, the separation between ourselves and the world. And Buddhist philosophy for me and I think It's not exclusive for Buddhism. I think many other traditions, wisdom traditions would have similar insights. Helped me realize some of these fundamental assumptions that we make and ask, okay, what if we come to realize that we are making these assumptions

SPEAKER_02and perhaps re-examine them? And in the context of computational models of learning and decision-making, that's where I found myself in this work with John Donne that built on some earlier, more technical work. Where I was trying to bring some ideas from Buddhist epistemology to bear on the question of state representation, which is a, in the context of reinforcement learning, it's a very basic question. I don't know, should we, should I say what kind of, what do I mean by reinforcement learning or is that kind of something

SPEAKER_02that we assume?

SPEAKER_00Well, you imagine we have a fairly educated audience for this, but it doesn't hurt to establish things from first principles. Yeah.

SPEAKER_02Sure, yeah. So state representation, the way I see it, is basically a way of describing how we construct models of the world. So this is the topic of this issue. How do we decide what kind of descriptions we use for interacting with I would say that idea of how we model the world is largely encapsulated through the notion of a state or state representation, which is

SPEAKER_02Kind of like a compressed summary of what's relevant in the world right now for my interaction with it, for my attempts to solve some kind of task. And usually the typical way reinforcement learning models deal with some The problem of learning in general is by assuming that there is some reward function that describes the goal of whatever task at hand that the agent is facing. And this reward function is used to describe how agents learn optimal behaviors

SPEAKER_02under a given dynamics or given world representation of the world. But the kind of crucial point here from my perspective is that the way that agents describe the world model that they have are considered as largely orthogonal to their goals or to their reward functions. So those are usually considered as two values. Fundamentally distinct aspects of the mechanisms of learning of how to describe a situation of an agent learning. It has this kind of descriptive model of the world and it has this kind

SPEAKER_02of normative evaluative reward function that is supposed to approximate its goal. And these two things are Considered separate from each other. And I think that this kind of goes back to these dichotomies that we often take for granted in our thinking. In Western philosophers, there's a long tradition, going back at least to David Hume, distinguishing between facts and values. Or more recently, people have spoken about beliefs versus desires. And those two things are thought to be two independent domains or kind of kingdoms

SPEAKER_02of reality that you can't really go from one to each other. And the Buddhist epistemology would undermine that assumption saying that the way we describe the world, the way we conceptualize things, the way we categorize objects is fundamentally dependent on what we want to achieve. That's an idea that I didn't find expressed well in how computational frameworks describe learning and goal-directed behavior.

SPEAKER_02And that's the kind of underlying motivation for this work, which later led to this collaboration with John Donne, who's actually a Buddhist scholar who's been very much involved in dialogue with Buddhism, between Buddhism and cognitive science. And we've also developed We've done some more technical work to show what kind of implications or applications these ideas might have in more traditional enforcement learning or other computational learning frameworks.

SPEAKER_00Yeah. To spend a little bit more time on Dharmakirti, you talk about telec efficacy, what it means In that philosophical tradition, for something to be real, you say. In other words, a thing can be said to be real in that it produces effects that are relevant to the goals that are active in a moment of cognition. Since abstract entities like categories or universals are not causally efficient, the thought fire cannot heat a cup of tea. He holds that the only true existent things are concrete,

SPEAKER_00causally efficient things or particulars. Falaxana in his terminology. Therefore, cognitions are not neutral acts of acquiring knowledge. Rather, they occur within goal-oriented motivational contexts that end in the occurrence of experiences that confirm or constitute the achievement of the desired outcomes. It's funny because this goes back to what you're saying about this distinction between seeing the good and the true as orthogonal. So yeah, I would love for you to get into a little bit more detail specifically about the framework of this tradition and the idea of concept formation

SPEAKER_00and knowledge claims being something that's always teleological in nature.

SPEAKER_02Yeah, so first I should say as a caveat here that John Donne is really the expert on this philosopher, Dharmakirti, and I'm a dilettante just reading his work. I actually got to know Dharmakirti through John Donne's book, Foundations of Dharmakirti's Philosophy. So my interpretation of Dharmakirti is by no means authoritative, but I can say how his ideas fit into my understanding of modeling goal-directed cognition in natural or artificial agents. And perhaps to give a slightly more philosophical backdrop to this,

SPEAKER_02Dharmakirti, as a Buddhist philosopher, worked within a framework of ideas that undermines the notion that we have an enduring and substantial felt. That's one of the tenets of Buddhism, actually, that our conceptions of ourselves as this kind of enduring and centralized loci of control and fixed substance are misguided, and that this kind of epistemic mistake about ourselves is the source of our suffering, essentially.

SPEAKER_02And there's another kind of interesting point here that knowledge is also related to suffering or the alleviation of suffering. That's another kind of fundamental dichotomy that I think comes from engaging with these ideas that I did not find it as clearly expressed in Western discussions of cognition and ethics and so on. But that's a different topic. Maybe we can... Say something about that later. So Dharmakiti works within this context of undermining this notion of having a substantial self and the substantial self acts as a kind of template for

SPEAKER_02The conceptualization or the way we create distinctions in the world, the distinction between self and other or self and not self is perhaps the most fundamental distinction we can make. And in some sense, the Buddhist view would say that, or one Buddhist view that there are many Buddhist traditions. I'm talking here about a certain school of Buddhism that Dharmakirti belonged to. So it would argue that the concepts we make, the distinctions we make in the world are in some sense just as misguided as

SPEAKER_02the distinction we make between ourself and what's not ourself. Therefore, concepts are really these kinds of, I would say, things we impose on the world in order to achieve particular goals. For example, the thing I'm holding now, we might call it a mug, but there's actually nothing really in common between this mug and other things we call mug. Except that we use them for the same purpose, such as drinking tea. And really, if you consider it as a subatomic particle level, that's a correct statement.

SPEAKER_02But we tend to impose on objects some kind of similarity because we have certain dispositions, certain goals, certain biases. There's actually a very interesting parallel here, which I think is a good bridge between kind of AI research and I would say the Buddhist epistemology. There was a Japanese physicist in the early 20th century called Satoshi Watanabe, and he had a book called Knowing and Guessing,

SPEAKER_02which is really one of I would say one of the early kind of groundbreaking books on AI was called Pattern Recognition in those days, but really had very interesting ideas about using information theory and all kinds of ideas that are still being explored today in AI. And one of his celebrated results It's something he calls the ugly duckling theorem. And the ugly duckling theorem says that any two objects, to the extent that they are distinguishable, are equally similar to each other.

SPEAKER_02So the reason it's called the ugly duckling theorem is because he said that an ugly duckling and a swan are just as similar to each other as two swans. He proves this kind of using very simple kind of combinatorial arguments, using tables of objects and properties, And he said when he told this to some people, they felt that this is completely trivial or that they had a sense that this must be true. But no one actually ever proved that because we have this very intuitive notion that things are similar because they share certain properties that are important

SPEAKER_02to us. A priori, that some properties are more important than others. And I think that's close to what Dharmakirti theory on concept formation is saying. It's saying that we ascribe similarity to things and categorize them as Because we have certain biases, certain goals, and that causes us to prefer certain features over others. But on an objective level, there's no reason to prefer one set of properties over another.

SPEAKER_02And if today we're starting to consider diverse forms of intelligence, so I think these ideas are very important there because we are familiar with certain concepts. We have this flavor of cognition, which has our own human dispositions. But if we now start to introduce other forms of intelligence into our systemic landscape, then these forms of intelligence would have presumably We have very different sets of dispositions and goals, perhaps, and therefore concepts which are very different from our own.

SPEAKER_02And to some extent, you don't need AI for that. That's true also for animals. And the Buddhists talk about other kinds of sentient beings that exist, but we don't have to go into that.

SPEAKER_01I actually would like to go into that. I'd like to prompt you a bit more on this. Correct me if I'm misunderstanding it, but so with the ugly duckling theorem, as if you're saying like objective as an unbounded feature space, as long as the number of dimensions are broad enough, then any two things that are distinguishable are similar slash different.

SPEAKER_02It doesn't even have to be unbounded. So I think the way Satoshi Watanabe does it is that he assumes that there is a set of features. So let's take the example of the ugly duckling and the two swans. We might say, Clearly, the two swans are more similar to each other because they are both white and they both maybe sound the same or behave similarly and so on. But suppose that the ugly duckling and the two swans are swimming in the pond and the ugly duckling is in between the two swans.

SPEAKER_02And now someone may be interested in Which entity is on the left side? So from that perspective, the ugly duckling and the right side swan are similar because they are both not on the left side. So if that's what you care about, then the ugly duckling and the swan are just as similar than the two swans. And he showed this combinatorially where based on a set of features, you can construct the conjunction, the negation, and build this kind of truth table. And just by counting how many rows have He shows that any two objects

SPEAKER_02which are not identical have the same number of similarities. So if you don't assign In particular weights over different features, you can't logically assume that one is more

SPEAKER_01similar to another. So less about being an unbounded feature space, but more like you can always establish an interest or a set of criteria along which similarity and difference can shift around based on when you impose this particular interest. Once you've defined the scope of relevance, then Within there, what was before seemed dissimilar could become more similar depending on the interests and the perspective that you're applying to it.

SPEAKER_02Yeah, the argument has a similar flavor to, you know, these free lunch theorem kinds of arguments in machine learning saying that you can't have an algorithm that works well on all instances. You can always find like a set. So in this case, a set of kind of interests where what you've thought of as similar is not similar and vice versa.

SPEAKER_00So I want to probe this distinction between conceptual and non-conceptual awareness and this idea that tendency to conflate categories with real things is innate and unlearned, right? And this is something that shows up in both of your papers. So I might be jumping a little bit ahead of the conversation and in doing so saying something that's heretical, at least from The perspective of evolutionary biology, but when you say the paradigmatic form

SPEAKER_00of non-conceptual cognition is sensory awareness, which gives one direct causal contact with real causally efficient things, we know, right, that you have learning in biological systems at multiple different timescales. And at one of those timescales, The organism itself is a kind of encoding of stable environmental features and that it's encoded in a particular way that one might say is geared toward the telec representation of that environment,

SPEAKER_00toward the goal of survival and reproduction. That kind of thinking was not present at the time of Dharmakirti. And I'm wondering how you see these two different perspectives Supporting or holding one another in tension. Is it not the case that non-conceptual cognition is basically still operating as if the body is a relatively inflexible concept of the future states of an organism's environment? Like the fact that we have eyes to see that we're not using sonar, this gets into like the work of people like Donald Hoffman who are talking about

SPEAKER_00the issue of what we are even capable of forming concepts about.

SPEAKER_02Yeah. Yeah. So I think Maybe two things come to mind.

SPEAKER_01One is, yeah,

SPEAKER_02it's true, Dermaketi wasn't aware of evolutionary theory and all that, but at least the way I interpret Dermaketi's idea is that we have these biologically ingrained biases that are based, for example, on how our body is constituted, what kind of sensory organs we have, and so on. And I would say that those I call them inductive biases. Those are part of the kind of constitution of our dispositions and our goals that determine what concepts we have.

SPEAKER_02There's a well-known paper, what does the frog's eye tell the frog's brain or something like that. So for a frog, a fly is a very kind of ingrained concept that's almost biologically implemented. Now, for Dharmakirti, and this goes a bit maybe more into Buddhist metaphysics, there's this notion of, I would say, shared world or shared realms of existence.

SPEAKER_02So the Buddhists speak about different realms of existence. One of them is a human realm, there's an animal realm, there's a hungry ghost realm, and there's a god's realm and so on, and hell realms. And these realms are constituted... By what's called the collective karma of the beings in those realms. Karma, that sounds like a very kind of mystical, metaphysical concept, but what it means basically is some kind of shared intentions and shared dispositions that are formed.

SPEAKER_02Over repeated interactions with the world. So it's actually not that dissimilar from notions of evolution, or even perhaps notions of reinforcement, learning from experience. And based on having certain shared experiences and shared intentions, we, for example, as human beings, we develop a shared language, shared concept, and These shared realms that we create by our actions and mostly by our intentions construct the reality we live in.

SPEAKER_02So the concepts we use and to some extent actually really the reality is defined by those shared intentions and actions. I'm not sure that really fully answers your question, but that's the world that that theory was coming from and really they give some Very visual description of this, how when humans see a river, they see flowing water, but when hungry ghosts see a river, they might see flowing blood and pus and so on because they have this certain karma

SPEAKER_02and certain dispositions on the world. I think it really makes you think about how Evolutionary heritage shapes our experience of the world and also our personal development, but that's maybe on a different scale. And maybe just one last thing about the non-conceptual contact with the world. Again, for Dharmakirti and many of the Buddhist philosophers, what we really, the only kind of true contact we have with reality is through non-conceptual experience,

SPEAKER_02which is something that perhaps we cannot really We talk about so much because it's non-conceptual, but everything that comes later, the concepts we create are in some sense considered as illusionary or misguided, mistaken. Again, based on our biases and our dispositions, we create this concept. So part of the aim of Buddhism, the kind of releasing or decreasing suffering, is about trying to get closer to our Direct, non-conceptual experience of the world,

SPEAKER_02and that requires a process of deconstructing our concepts and our models of the world.

SPEAKER_01If all of our concepts are, in a sense, necessarily kinds of falsifications or fabrications or reifications, is this, in this view, it's more than saying even though all models are wrong, some models are useful, and a necessary divergence between maps and territories, but it's a more fundamental statement even about the ontology of what is actually real. Would that be correct to describe in this view, or...?

SPEAKER_02I think so. I mean, there are ongoing debates on these questions, obviously, between philosophers, but there are different schools of thought in Buddhist philosophy, and there is some debate on which school exactly should, for example, be assigned to. Most people assign him to a school called Yogacara, which is often called Mind Only. And they say that kind of the fundamental constituent of reality is mind. And there are other schools like the Madhyamika,

SPEAKER_02which is the middleweight, Which say that even that's not enough and actually there's no ontological foundation for reality. And it really touches on these fundamental questions on what can we assume as given or as kind of axioms when we ascribe reality to something.

SPEAKER_01I don't want to make it too palatable for people who don't have the sociological interests. I'm wondering, so you would still have a notion of, this could seem like a very radical relativism. In some ways it is, but still there's room for a notion of relative truth or conventional truth within there where it's like, Provisionally, once you establish a frame of relevance or an interest, once you define the dimensions for a goal that's being pursued, then you do get more or less fit-for-purpose ways of describing things. It's not just like anything goes.

SPEAKER_01Would you say that's accurate or...?

SPEAKER_02Yes, and I think that's the kind of framework in which Dharmakirti is working in. And he says, for particular purposes and goals, we create concepts that can be more or less suitable for achieving those goals. And all that is on a relative level, given some assumptions about our kind of mundane goals. Eventually, on a kind of ultimate level, I think that there is this kind of calling to transcend that perspective and realize that all these goals are in some sense relative or fabricated,

SPEAKER_02including ultimately As we said before, the fundamental concept of myself as a separate entity from the rest of reality.

SPEAKER_01Although the self as well would have a kind of relative provisional reality to it. It's like some notions of a self that are like less essentialist or less eternalistic. Like for instance, like they give a self as like a center of narrative gravity or some sort of like Center of mass, of interestedness. There's still room for meaningful notions of selfhood. It's just this idea of an enduring, permanent self that's not context-dependent and sensitive and a river

SPEAKER_01that you never step into the same place twice. That's the word jettisoning, but there's still rooms for varieties of selfhood that have provisional reality to them and

SPEAKER_02Yes, I think that's a good way of putting it. And certainly we rely and we should rely on notions of self in our daily interactions with the world and with other people. But if we go into this Buddhist analysis, so again, depending on what stream you aspire to, but eventually you realize that on an ultimate level, this notion of self as a fixed and enduring entity dissolves.

SPEAKER_00So I want to say something that may broadcast my ignorance in these matters. I'm reading toward the very beginning of this paper, you talk about large language models such as ChatGPT are designed to represent words as numerical tokens and are trained to optimize the prediction accuracy for the next token in a given sequence. Such systems cannot suddenly decide to represent words differently, say according to their phonetic similarity. Or optimize any objective other than the one fixed by their designers during training.

SPEAKER_00Whereas by contrast, human conceptual representations, While suboptimal, from a statistical point of view, can flexibly adapt to a broad range of contexts and goals. And then later, to connect this to the comments that you just made about this reification of things and its role in suffering, I'm wondering, is it possible, therefore, to suffer due to reification without the concept of a self? There is this thing about the inability of these systems to reflect

SPEAKER_00on We're not calling them concepts on the veridical nature of these state representations. And then just for bonus points, I wanted to link this to the question of like irony and humor, where you say later, you're looking at what we would call an apple. I may interpret what I am seeing as red or spheroid or edible or an apple or any of a number of other things. But if the concepts that emerged during that experience are to guide effective action, I cannot conceptualize what I am seeing as a banana.

SPEAKER_00And yet the online culture of Gen Z seems particularly fond of this sort of inanity as a kind of self-defense mechanism against the brutality of life in our world right now. And you've got kinds of people that have theorized humor, Tim Morton and others pointing to This kind of thing. So there is a sense in which like the goal of not suffering irony has a role in representing things that we know to be not causally efficacious at that level,

SPEAKER_00but at another level, it's helping us manage. So that's really two different questions. It's like one about, can we even talk about suffering in systems that do not conceptualize a self? And then the other being the instances where a supposedly statistically suboptimal Again, we're thinking at multiple levels about where, for instance, play seems statistically suboptimal, but at the level of things evolving over multiple generations,

SPEAKER_00it needs that kind of flexibility.

SPEAKER_02Yeah. No, that's interesting. I haven't thought of that point about play and irony. I'll have to think about that. But for your first point, I think that one thing that we also argue at the end of the paper with John, that maybe one of the main barriers separating us from These artificial forms of intelligence is the fact that, again, going back to this notion of non-conceptual experience, this idea that we have this kind of direct way that's really hard to describe of interacting with the world.

SPEAKER_02And it probably has something to do with our bodies and with our biology. And we mentioned Jeffrey Hinton has a notion he uses, he's been using recently, called mortal computation. Which is computation that is fundamentally kind of embedded in the biology of the organism, which is completely different from most of the AI systems we are currently interacting with. So there is some work in neopic computation and embedded robotics and things like that, maybe starting to go in that direction. But yeah,

SPEAKER_02if we accept this account of the importance of non-conceptual and this kind of direct contact with perceptual experience, then I would say that there's still a way to go for talking about suffering in these

SPEAKER_01devices. If we were to, I don't want to cut you off, but if we were to describe like a Bayesian brain type notion or like some predictive processing ideas, Is there room for such a thing as non-conceptual experiences? I guess it depends what you mean by concepts or not, at least involving some degree of fabrication. Is there room for direct contact with reality if perception is a kind of inference?

SPEAKER_02Yeah, that's a good question. Helmholtz, who's the grandfather of these ideas of predictive coding, he spoke about perception as unconscious inference. So it sounds like he at least believed that perception occurs at this very almost subliminal level or this inference that generates our perception occurs at some subliminal level. I don't really have a good It's a good sense of what's really going on there. I think maybe there's some interesting work these days

SPEAKER_02on these advanced meditative states, sometimes they're called jhanas or all kinds of states that are actually very formally and explicitly described in Buddhist texts. And they talk about these kind of different kinds of experiences. I can't say anything about that from a firsthand perspective, but they describe these Experiences which are non-perceptual but also not non-perceptual and all kinds of levels of experience which may be where this kind of unconscious inference that Helmholtz was referring to.

SPEAKER_01Although I guess to contradict myself like or myself of a few seconds ago, I guess I don't want to be also too inclusive with the word conceptual because I think maybe also what's being got at is like, for instance, more of an explicit categorization or reification, something related to the dualistic nature of some kinds of selfhood as part of the way of relating to the world. So there's I guess like the idea of like suchness of like an experience where the process of fabrication orification has been stripped away. And there is a greater degree of immediacy in terms of coupling the world and less

SPEAKER_01of a imposition of your priors and of your expectations. I'd say there is quantitative and qualitative differences potentially of those modes of perception being in the world of being conceptually dominated in your way of relating And having fewer expectations and semantic imposition of what you think is there.

SPEAKER_02I mean, I think that's really interesting. Again, I don't have much to say about that, but I know that some people have been doing work kind of around these ideas of active influence, developing models of these non-conceptual experiences that people have in meditation. And there's this kind of emerging, I don't know if to call it a field, but emerging notion of computational phenomenology, trying to use these Bayesian and active influence ideas. To describe different kinds of experience, including experiences in these meditative states,

SPEAKER_02which have this sometimes non-dual dimension to them. We're really just starting to scratch the surface there.

SPEAKER_00So I want to move on to the paper that you actually got in this collection, Goals and the Structure of Experience with Tjomkin and Langdon. Early on in that paper, you make the point that standard reinforcement learning models typically include a single state representation. Biological learning probably relies on multiple hierarchically arranged ones. I think we've been dancing around this for most of the conversation so far. I want to link this statement here to another paper in this collection,

SPEAKER_00the paper by Benjamin Lyons and Michael Levin, where they're talking about economies as a form of collective intelligence. For which price signals form something analogous to bioelectric signaling. It's the cognitive glue that holds this thing together. And the issue I'm most looking forward to discussing with them about that paper is that macroeconomic models seem to fail in this precise way by declaring non-market economies irrational or inefficient. Based on evaluation at only one level of analysis,

SPEAKER_00for instance, non-transactional exchanges inside of a firm or a family are seen as irrational because they're statistically suboptimal at the level of the entire economy. And yet they exist. You see this kind of thing being teased at by an exchange of letters on the role of noise in complex systems between like Kahneman and his people and David Krakauer, Wolpert, Jessica Flack. And I guess where this is all pointing me is like this question of, okay,

SPEAKER_00so we've already established that the self is emergent according to conditional goals on the part of the organism. I think it's useful sometimes, not so useful other times. And the question, I guess, is one about when it makes sense to draw the line around what we're actually recognizing in AI systems. For instance, maybe the kind of The intelligence that we recognize as human is more evident in entire populations of AI algorithms that are forming a kind of like basket

SPEAKER_00of different state representations based on different goal states. And so maybe the individual that we're calling AI is more like every different form of AI. I don't know. There's a lot there, but I'm hoping I'm giving you something to riff on.

SPEAKER_02Yeah, yeah, there is a lot there. And I agree that I think it's very interesting and useful to think of AIs as collective intelligences. Obviously, other people have been saying that Michael Jordan had a keynote at a recent AI conference in Europe talking about AIs as a collective. He gave a keynote there about AIs as collective intelligences. And I think that has a lot of truth to it because they learn from this collective intelligence. It's a corpus of language that was produced by many people.

SPEAKER_02And actually, the direction I'm now trying to take this work on telekstates is to see what that perspective has to say about joint representations and questions of joint agency because, again, it has this kind of flavor of softening the distinction between self and other. Therefore, it's amenable to describing notions of collective agency. And in particular, I'm now working with Maya Frazier,

SPEAKER_02who's a mathematician at the University of Ottawa, and Marie Van Wucht, who's a cognitive scientist at the University of Groningen in the Netherlands, on extending this framework of telekstates. And seeing how it might describe situations where two or more agents are pursuing a similar goal and therefore perhaps generating a joint model or joint state representation. And we can ask questions like, to what extent is model of the world taking into account the experiences

SPEAKER_02of other agents and not just limited to my own? And based on These perhaps joint experiences, I can build a model that does not put me as a separate entity, as a kind of center, but rather puts us or the collective as an agent in itself, which has its own representation and therefore its own agency, perhaps its own intelligence, perhaps its own consciousness. And this is also consistent with work people are doing now in consciousness research

SPEAKER_02about theories. That are saying that consciousness perhaps depends on this causal dependency or integrated information between different elements of a system. And I think all these ideas are pointing to the direction that we can more youthfully think of intelligence as something that's not limited to any individual system or agent. But yeah, an economy can have it and a group of people can have it and so on.

SPEAKER_01Like in terms of what's obviously like missing from current approaches to AI or what we might want to add, but like in terms of, it's like an intelligent states framework and intelligence is, it seems in a way inseparable from at least some form of agency in terms of like, there has to be like a value that's, this is the way you're handling the frame and relevance problem. And there's an entanglement between your world model and your values.

SPEAKER_00They're not orthogonal, like you're saying.

SPEAKER_01And so a lack of coherent agency or interestedness or embedding and being in the world could be one potential explanation for ways in which models will fail to be intelligent in the ways we might hope or they might confabulate. They might lose the plots because they don't have necessarily as much of this coupling going on, even though they'll probably inherit some by just being conditioned on language. And our coupling of our world model, our values, structures the way we talk. And so they're going to pick up some of it. I guess another thing I'm wondering that might be based on what you're just saying,

SPEAKER_01either not present enough or maybe too present, but this idea of a no-self type view, in addition to being helpful for alleviating one's suffering, putting one more in contact with reality and potentially in a more other-including way that includes other minds as part of a way of understanding things. So I guess to close off the ramble, even though you want to potentially Weak and overly reified notions of selfhood. Is there still some kind of coherence from selfhood that's still necessary for intelligence and that you wouldn't want to jettison in the construction of

SPEAKER_01an intelligence system and one that we could coordinate with in terms of its goals and its values? Like how much self is enough self or too much self?

SPEAKER_02Yeah, I like that question, but I don't feel like I have a good answer to it yet. I mean, I would say that's part of what I see as my research program in bringing these notions from Buddhist philosophy to bear on the conversation in AI and in cognitive science. I mean, I can say this is maybe only tangentially related, but I think that for me, one of the main takeaways from this whole involvement with the Buddhist ideas is actually that this whole problem of alignment now,

SPEAKER_02I'm currently at a kind of AI safety research program in London here, and the alignment problem is on everyone's mind here. But I feel like really to get a better handle on that, we need to... Get a better handle on our own values and our own mind and our own societies. And it's not something that would be answered by purely technical breakthroughs or some brilliant mathematical ideas. I'm not saying that's not important. That surely will be needed and part of the problem.

SPEAKER_02But I think really taking this perspective of looking at The good of others and of societies and getting policymakers and people who know about social work and society and so on is really important to bring into these conversations. And in a sense, I feel that that's where This perspective of Buddhism and no-self is currently pointing towards, for me at least, that we need to widen this conversation.

SPEAKER_02And my small part in this is more in the technical domain, but trying to relax some of these strong distinctions we have between self and no-self and so on, and get us more accustomed to think in those ways when describing models of learning and goal-directed behavior and so on. So it's not really answering... I think it's really important to have these perspectives in the discussion on AI and where it's going to take us.

SPEAKER_00I have one more question for you, and I feel like I've been poking at it from a couple of different angles in this, but it comes out very clearly in this paper where you're talking about how in other recent work, you and your collaborators leverage the duality between goals and telex state representations to quantify the computational effort required by an agent to learn a policy that enables it to reach a desirable telex state under a given goal. This allows us to cast the conjoined problems of goal selection and telex state representation in terms of a trade-off between the granularity of

SPEAKER_00that representation and the computational complexity required for learning policies that can traverse all possible telex states. Okay. So this gets back to you bringing up earlier, no free lunch and these other things. It also relates really directly. To the contribution by Battleday to this collection on what it would take to create an AI scientist that is not merely giving an answer according to the pre-specified conditions within which it's trying to seek an answer to a scientific problem,

SPEAKER_00but an AI scientist should be able to do what revolutionary theorists are doing in human science, which is to Re-specify the question to decide the number of dimensions, the level of granularity at which we are trying to form And for things like electromagnetism, oh, it's one thing. I'm thinking about a conversation that Adam and I had on what constitutes greater agency. Is it a system's ability to optimize for a given state representation,

SPEAKER_00or is it the ability to update state representations? And I think this is where your work ties into, like I said, Battle Day stuff about how we might design artificial systems that are doing real fundamental theoretical work. It ties into questions about how AI research as a discipline may have internalized and taken for granted that agency is about the ability of a system to Extract profit. And yeah, this is an area I would love to hear you explore.

SPEAKER_00There's one more piece, which is this sort of James C. Scott's seeing like a state and the idea that systems at different scales are all operating under computational bounds and therefore a government is not really perceiving in this sort of loose sense of perception individual humans. But LLMs are maybe providing something like a microscope where our human institutions become more aware of themselves and therefore also of the socioeconomic drivers of their own goal states.

SPEAKER_00Yeah.

SPEAKER_02Yeah, there's a few things that I can comment on. One is this question of pursuing a goal versus choosing your goal, which is something that has questioned and bugged me for a while because goal selection really has this kind of circular flavor to it that how do you choose a goal if you don't have some kind of meta goal for choosing goals and then you need a meta meta goal and so on. So that's really a very kind of deep and interesting question. And at some point I realized that if we accept this Connection or this coupling

SPEAKER_02between goals and state representations, then that might give us a handle on the question of goal selection because we could say that I would like to choose a goal that would induce or produce a useful or in some sense a good state representation. Now, what is a good state representation? That's another question, but in the special issue, which is based on another paper I have with Stomkin from Texas Tech, We suggest that having a controllable presentation is a good thing in general.

SPEAKER_02Controllable means that it's easy to move between the states. So I have a good capacity to explore the state of possibilities, even if I have limited computational resources. We showed that certain ways of selecting goals or we formalize it using preference ordering. So certain preferences over experiences induce a more controllable state representation. And in that sense, we get a kind of handle again on this question of how to select a goal.

SPEAKER_02That will provide us with the most useful, the most controllable state representation. And it's not a binary yes or no thing. It's actually a continuous trade-off, like in rate distortion theory where you can say, given I have a certain amount of computational resources to invest in my decision-making, in my policy, What is the most controllable representation I can generate? And then you can play around with this level of granularity between the controllability of your representation and the efficacy of the amount

SPEAKER_02of computational complexity you can invest in your decision-making. And yeah, I haven't really thought too much about the kind of border societal implications of that, but I think it's really useful to think about this kind of coarse-graining of representations in terms of coarse-graining of our goals. So when we, like you said in the example of the government, if our goals are set at a certain level, we would probably only I wonder if you're willing to speculate with me here,

SPEAKER_02because you spend a good bit of this paper talking about how we actually see the human brain

SPEAKER_00Working on map-like information and value information, how they're multiplexed in the brain. Given everything that you just said, I'm willing to wager, and I'm wondering if you're willing to wager, that one of the things a lot of people around the issue of AI are talking about is the fact that it's like operating at this particular level of representation, and yet these systems continue to grow, and there's a thermodynamic Cost to this, that trying to model the entire world is eating the world with additional resource

SPEAKER_00and energy costs. Organismal brains are doing a really good job at finding the level that's metabolically appropriate to the organism.

SPEAKER_02Yeah, I think I can't really speculate on that, but I can say that the brain is a very important source of inspiration. And for me, even more so, I would say the mind than the brain, but those are obviously related. And the fact that we can solve things under these very strict computational constraints is... Definitely meaningful and I think helpful to consider when designing artificial learning systems.

SPEAKER_01Digging into a few things you just said, so in terms of people selecting goals that optimize controllability, I was wondering like in terms of other articles in the collection or just views that are among thinkers that are out there in general, but particularly in the collection, Ones that you think are particularly either consistent or discrepant with what you think is either essential for an intelligence understanding in general, or how we might actually recapitulate an artificial system. So for instance, the paper on empowerment with Gopnik,

SPEAKER_01like for instance, that seemed like very constant with your view. Curious if like you would agree with that. And also are there things where you would say like points of contention that would be important for us to talk about more going forward?

SPEAKER_02Yeah, so yeah, I'm very sympathetic to the empowerment ideas and I'm glad they're picking on now also in cognitive science and in this paper and in this issue as well. I would say maybe the kind of main, it's not a contention, but it's, I would say the main kind of difference in the emphasis is that I'm looking at the question of And this maybe also relates to the previous question a little bit about how the mind is so effective in honing in on the precise features it needs

SPEAKER_02to achieve its purpose or achieve its goal. And that's often a question that I feel has been slightly glossed over in computational approaches to learning. There's usually this assumption that the There is some given or ground truth or externally provided description of the relevant features or the states or the rewards or whatever. And the kind of Buddhist perspective that couples together or says that goals are really at the heart of our way of modeling the world, I feel sheds some light on this aspect of the problem,

SPEAKER_02which is often already presupposed in usual approaches. So I would say maybe that's the main thing.

SPEAKER_00At this point, Adam and I decided we weren't done asking questions and booked another session with Nadav. Before we dive into part two of this conversation, where we cover transformative experiences, performance and generalizability trade-offs, agency and active inference, how values are embedded in environments, and the implications for AI alignment. Let me remind you to check the show notes for links to the entire Open Access World Models issue, all of the other papers we mentioned in this episode, and an incredibly useful discourse mapping tool built by Van Bettauer of Ideoscopic

SPEAKER_00that charts every paper's major claims, disagreements, and possible avenues for future research. Big thanks to Survival and Flourishing Fund for the support that made this miniseries possible. If you value this work and you think there should be more of it in the world, consider pledging your support as well. Thanks. And now back to the show. All right. So here we are back again. And even though the fine grain details may have changed, I feel like The telex state representation of what we actually want out

SPEAKER_00of this conversation at the high level remains in focus. So it's good to have you back. I think one of the questions that we didn't get to in the first part is about how you say in this paper that the policies that are equivalent We spoke to Fritz Breithaupt and his contribution to this collection where he focuses on transformative experiences and

SPEAKER_00I think it leads into a question that I'd love to hear you explore about how some, like the sort of butterfly effect of details that appear equivalent at a certain level of resolution, but are in fact not, and how this paper addresses the issue of transformative experiences and how you see that relating to the way that you're talking about like multiple levels of granularity and state representations.

SPEAKER_02Yeah, so there's a couple of things I can try to say about that. One may be as a starting point is to go back to the philosophical grounding of this paper. According to Dharmakirti's view and the Buddhist epistemological view that to a large extent followed from Dharmakirti's work, There are no such things as real universals. Like, real things are never identical or even similar to other things. They're only individual particulars in reality.

SPEAKER_02Every event, every experience, every atom of matter is a unique particular. And any attempt to create similarity between objects It's an imputation of the mind and in some sense is a cognitive error, a cognitive mistake. It might be helpful. It might be even necessary to make such mistakes, such conceptual imputations to achieve our goals. But ultimately, any attempt to claim that objects are similar in the sense

SPEAKER_02that they share features must rely on some bias, some sense of perception. You must always have some kind of particular Focus of interest to claim that things are better or similar or the same. That idea is very manifest in Buddhist philosophy.

SPEAKER_02And this paper was trying to put the question, what if we take this position And view that as a way of describing how our cognition works in that our cognition imputes similarities on objects based on its predispositions, preferences. So that's just to touch back on the philosophical foundations. And then more particularly to your question about transformative experiences. So one interesting upshot of this linking together of representations and goals

SPEAKER_02Is that we can start to consider in perhaps a more formal way how our goal shapes the way we present the world, which in turn shapes how we experience things and therefore how situations like transformative experiences where we change our outlook, we change So a framework that connects goals and representations would, I think, allow us to say,

SPEAKER_02Something a bit more concrete about the seemingly vicious cycle between goals and experiences, which is a characteristic of the whole literature on transformative experiences. Because the whole idea of transformative experience is that if we assume we have a fixed way of evaluating fixed preferences, fixed goals, we cannot really consider what it would be like to have different goals. And therefore, transformative experiences are out of scope for traditional Frameworks

SPEAKER_02of Decision-Making Rationality. And actually, Laurie Paul, who's a philosopher who wrote a lot about transformative experiences, she was one of the participants in a conference we had at Princeton that I organized, Holly Crockett and Jonathan Golt. On Buddhist philosophy and cognitive science, the whole discussion on transformative experiences from this kind of Western philosophical perspective that Laurie Paul has been pursuing contributed quite fit well with the Buddhist,

SPEAKER_02with the Buddhist way of thinking about agency and experience in that context.

SPEAKER_00There seems like a conflict here with another statement that you make in this paper, which is that, as you say, typically agents control their own behavior, but not that of their environment. Thus we shall assume that the agent maintains a fixed model of its environment. So that experience distributions are determined by the agent's policy. I'm just curious about this. I totally understand like for the point of making a point, that's fine. But it sounds like the position that you're actually coming from is that we know that agents do not actually maintain fixed models.

SPEAKER_00Their interaction often alters the environment. And then that interaction with the environment alters the models and the goals and so on. And so I'm wondering how you reconcile this.

SPEAKER_02Yeah. So, I mean, I think that the bits that you read now is a, is like more of the way we technically try to describe the framework and we need to have a notion of a policy, which is a standard technical term in reinforcement learning. Usually it's a mapping between states and actions. So given your current state, what is the action or the distribution over actions that determines how you behave? Now, I agree that eventually we would like to allow agents to have perhaps more control

SPEAKER_02over the environment and maybe have a model of that as well. But here we started with a more traditional and modest aim of likely extending the existing formalism of reinforcement learning. First of all, to consider What we call experiences instead of states. So the policy we use isn't a mapping between states and actions. It's a mapping between sequences of actions and observations and future and the next action. So these sequences of actions and observations is what we call an experience,

SPEAKER_02which is linguistic, perhaps only as a kind of semantic thing that we call it an experience. Usually people call it a trajectory or they have other techniques. We call it an experience because we really want to foreground the fact that we're dealing with what the agent is experiencing and the way of foregrounding the agent is experiencing. Normalizing that is simply as sequences of actions and observations or more generally distributions over such sequences since we want to allow

SPEAKER_02for probabilistic actions and outcomes. So a policy is a mapping between agents' past experience or it could be also a simulated future experience and its behavior. We're still working within the general formal framework of reinforcement learning, but we're trying to push the boundaries a little bit by using these sequences of experience instead of state.

SPEAKER_01Would this be somewhat like a successor representation framework, or it's like the hippocampus will store for a scene, or there'll be boundaries on a sequence of actions in an event? Would that be related to him?

SPEAKER_02So I think the successor representation can be seen as a kind of Intermediate, perhaps, between the traditional state, Markovian state framework of reinforcement learning and something that looks at sequential experiences, which is what we try to do. The successor representation is still, I think, more committed to the Markovian assumption. So the idea that the counterpresentation is fully integrated Determined by the previous one.

SPEAKER_02We don't commit to that assumption in this work exactly because we're looking at the full sequence of past actions and observations. Although obviously we don't assume that the agent necessarily remembers all the history or maintains representations of the full history. And we mentioned this in the paper. I think that's where also these kind of resource rationality trade-offs come where Agents' goals determine which aspects of their experiences are important

SPEAKER_02to remember and maintain. I would say it's a generalization. It's a more general framework than successful.

SPEAKER_01Like by Markov instead of at each time step, it's like Markov separation where it's between the past and the future, given the present, and you're just choosing action based on where you are. For you, it's a trajectory. It's not the sequential decisions, but it's a whole unfolding of a sequence of actions.

SPEAKER_02Yeah. And I think that's an important point because one of the phenomena we initially tried to connect to this framework is what's known as flow experiences, which is... Also related to the phenomenological dimension of having these positive mental states, which have a specific tempo to them, a specific dynamic, specific flow of experience, like when we're playing sports or listening to music or things like that,

SPEAKER_02where it's hard to explain that as a kind of modern sequential experience where, you know, what I'm doing now only depends on what I did a moment before. But there's a kind of, again, this word slow, I think is also suggestive of that. There's a kind of sequential aspect where it's about how the different moments of experience relate to each other in this temporal sequential order and not just in a kind of temporarily localized way.

SPEAKER_00So one of the things that keeps coming up Not only in this conversation, but in other conversations, and I really appreciated the treatment of it in this paper, was this issue on the fundamental trade-offs. And I was left with this question of where do we see, how did you put it? The conjoined problems of goal selection and telex state representation. I was left with this question of, well, what does it at least look like theoretically in computationally unbounded agents?

SPEAKER_00Because as we just discussed with John Krakauer, maybe one of the reasons we're not seeing coarse graining in the ways that he and his brother Melanie Mitchell proposed it, like the encoding of novel bases, concept formation, is that, I mean, obviously these are still in some sense computationally limited systems, but they've also got a blank check in terms of the amount of Money that's going into their development,

SPEAKER_00the amount of electricity that they have, the amount of compute. And so it was like this question of the issue of the apparent irrationality of behavior below or above the level at which an agent is capable of resolving its representations and how this question of these trade-offs might inform things like in Benjamin Lyons

SPEAKER_00and Michael Evans' paper where they're talking about looking at the economy as a collective intelligence, they might be able to formalize the nature of the irrationality of non-market Human systems, like systems that are not using price signals to communicate, even though they know that's the most efficient. And so there were like two things there. One was like, there was a, I don't want to say like an apologia, but an understanding of, again, this like Dharmakirtian thing that all conceptual knowledge is Flawed in this way,

SPEAKER_00but then also I thought about not just an understanding of why it is that way, but why it is both flawed and necessary and how the systems where we don't see these kinds of things in play are extraordinarily wasteful and stupid in some way. I hope I'm making sense. And if I'm not, Adam, help me. Well, I mean,

SPEAKER_01I guess, I mean, Inductive biases, I guess from the start, it's really taking on like the bias part of it. And so in order for you to get around no free lunches or get your lunches on the cheap, you have to have this prior knowledge that you leverage, where you make assumptions about the presence of what you're encountering. And these simplifying assumptions let you be data efficient and let you infer more than you would otherwise. But they're simplifying assumptions. And so there are also sources of bias and they,

SPEAKER_01they work in one domain, but not necessarily another. And so you get performance narrowly, but potentially at the expense, the generalizability of the system. So it's like, you're damned if you do or damned if you don't. All models are wrong or some models are useful, but good luck getting through this world without models. So like a system without inductive biases, it's going to be hard for it to function intelligently, but they come at a cost. Yeah.

SPEAKER_02Yeah. No, I think you put it very well. We can't really make sense of the world. We can't really achieve what we want to achieve without these biases, without concepts. I think what the Buddhist or Dharmakirti introspective really is. The concepts we used are really coupled to the goals we want to achieve and ultimately are not about reality. The concepts are about what we want to achieve and not about what's really out there.

SPEAKER_02And some Buddhists thinkers even Actually, maybe Dharmakiti also had this in mind when he developed this theory, is that the kind of final realization, if you take this idea to conclusion, is that the most basic distinction that we make, which is The distinction between ourselves and the other or the world or what's in here versus what's out there or between the subject and the object, that's also one more concept, one more cognitive bias,

SPEAKER_02maybe the mother of all Cognitive biases, which is a source of all of our ignorance, all of our suffering out of the Buddhist account. And that's why you hear this talk about non-duality as a endpoint of a lot of the contemplative practices in Buddhism. And that's a very essential concept.

SPEAKER_02The idea in Buddhism, which can have very profound implications if we eternalize that, realize it, and think about what it means for us being agents in the world for now AIs. We think of them as agents, but we impose on them our own preconceived notions of subjects and objects and duality and so on. I don't have a kind of answer on what this philosophical route is going, where it's going to take us, but I think it's a very rich area to explore.

SPEAKER_02And these papers in this special issue, I think, also speak to that, that these cognitive biases and these trade-offs are really something fundamental to our cognition and now to how we understand other kinds of agents, whether they're artificial or natural or hybrid.

SPEAKER_01You're mentioning like Lorianne Paul's work. And I was just rereading this paper that she authored recently. It was Reverse Engineering the Centered Self. And I'm remiss. I wanted to actually send it to you because I was curious about your thoughts on it. And so she's presenting a self-process as a perspective On the world, situating an agent environment as being important for handling diverse task situations. And so she calls it a meta POMDP framework.

SPEAKER_01So POMDP is a partially observable Markov decision process where it's the Markov decision process as you're moving forward, stepping through time. Making decisions and partially observable and that you have incomplete knowledge. And then the idea is that there's going to be sets of policies that are more fit for purpose for different task environments. It's interesting in that like, it seems like a crucially functionally important selfhood, but it's also a dynamic selfhood. It's not a static reified one. And it reminds me of Emptiness is form and form is emptiness. If selves are going to do the job we want them to do, they have to also be able to know self.

SPEAKER_01They have to also be able to move to a formless state, which in a way brings us to transformative experiences again. I'll send you a paper. I'm very curious to know your thoughts on it.

SPEAKER_02Yeah, please do. It sounds great. It sounds really fascinating. And I think that's really the kind of work that I'm looking to see more of, that we realize that we have these rich philosophical traditions that can also be drawn on, that can help us re-examine our assumptions about ourselves and whether it should be thought of as a fixed agent or something that's more dynamic and flexible. Yeah, obviously in Buddhism, the tendency is to go on the very kind of impermanent edge of the spectrum.

SPEAKER_02So everything is in a kind of constant flux, including our own self. And yeah, I think we are starting to see more and more theories, models, ways of formulating these ideas mathematically, which I believe is really important now, especially with all the dangers, with AI risks and so on. We really need to open our minds to new paradigms. These wisdom and philosophical traditions are a good source for that.

SPEAKER_00I would like to situate this whole conversation inside a kind of epicenter in mainstream AI debate right now, which is the conversation that's been happening in public between Yoshua Bengio and Jan LeCun on social media. Bengio says that AI systems should make predictions without any goal. He says goals would bias the systems in possibly dangerous ways by giving it drives and desires. I completely disagree, Lacoon says, with the whole premise.

SPEAKER_00I don't think any system can do anything useful without an objective. But his recommendation, he says, is the exact opposite. He's like, they should have goals, they should be designed in ways they can do nothing else but fulfill the goals we give them. By construction, the system must fulfill the goal we give it and must abide by safety guardrail constraints. And this is the work that he's trying to do right now. So it strikes me that in one way, you stand on Lacoon's side by saying goals are necessary, In another way, in light of the conversation that we've been having,

SPEAKER_00I question whether Lacoon's goal of making these goals totally externally controllable by imposing perfect safety guardrails is even possible, given everything that Adam was just saying about the inductive bias and so on. And in that light, it seems like Bengio is acknowledging a real and undeniable risk that Which is like, if we want these machines to understand the goals we're giving them,

SPEAKER_00they need to have them. But they're not going to be perfectly programmable. Yeah.

SPEAKER_02Yeah. No, I mean, so obviously, I mean, both Benji and Lacuna are great scientists and have made like seminal contributions to the field. And I'd be hesitant to criticize their opinion before getting a better sense of what exactly they're saying. But I've heard Bengio talk about this recently in Montreal. And I think that my take would be for both Bengio and Lacoon's position that we need to also acknowledge that any system, even if we don't explicitly And the goal is a very ambiguous term.

SPEAKER_02We should also realize that. But if we go back to this Dharmakirtian viewpoint, any system we describe conceptually will already have built-in goals by the very nature of the concepts and distinctions we use to describe it. And that's why my sense is that Saying that we can build a system without any goals, I'm not sure that's possible. Again, depending on what exactly you mean by a goal, because you need to be very concrete and explicit about that.

SPEAKER_02But from the Buddhist perspective, Goals are really embedded and meshed in our conceptual systems. And unless we reach a stage of total equanimity where we kind of view everything as complete impartiality, we cannot really talk about the world without some notion of goal sitting in the background. So this might not speak to this debate of Benji and Lecun on the same level where they're discussing it,

SPEAKER_02but I think that on a more Kind of cognitive, philosophical level, we should be aware of the tight coupling between our language and our linguistic concepts and the desires and goals that underlie that and feed back into that as well. So it's a two-way relationship.

SPEAKER_01In terms of concerns about AI safety. So I guess for Yoshua Bengio, he's very concerned about convergence, instrumental goals for things like power seeking and things that have us come into rival contexts with each other. With respect to that, it seems like a Buddhist perspective could provide an example of a vision of benevolent agency, potentially. So I'm interested to hear your thoughts on that. But also with respect to this entangling of our beliefs and our Yeah,

SPEAKER_01I think you're exactly right that this line of thoughts that I've been

SPEAKER_02And I think that, yeah, that also does have implications on AI safety or AI, I would say, on maybe ethics more generally, because it means that the way we describe, the way we relate to the world, It has a kind of built-in normative or ethical dimension to it. Buddhist philosophy has been described by people like Jay Garfield,

SPEAKER_02the famous Buddhist scholar, as a form of moral phenomenology. So it's a way of Cultivating our cognition, that's what Buddhist meditation is about. It's about cultivating the way we experience the world, the way we experience things in a way that's somehow more beneficial. And beneficial means contributing to the alleviation of So we learn or we cultivate a way of seeing things,

SPEAKER_02of experiencing things, of understanding things that in and of itself This mode of experiencing the world contributes to the alleviation of suffering. It's not about what we do necessarily, which is an emphasis in a lot of Western ethical frameworks. It's more about how we experience the world. And obviously, how we experience the world will affect what we do. But the source of the transformation and the ethic is in the way we relate

SPEAKER_02and experience the world. This notion of karma, that what it really means is intention or sometimes it's called action, but it's more about the kind of internal orientation of our intention, which determines our ethical, the ethics. So this, maybe it's a bit too abstract of an answer, but yeah, I do feel that incorporating some of these ideas for Buddhist philosophy can help us

SPEAKER_02to link the way we talk about epistemology and learning and how we understand and guide things. On the one hand, and on the other hand, how we relate to them morally and ethically. And my long-term vision is that we will have some kind of theoretical framework that doesn't Separate the ethical and descriptive or epistemological dimension of agency, of learning, of decision-making.

SPEAKER_02And yeah, I think it's a very long lofty vision, but there are steps in that direction, people working in these spaces. And yeah, I think that some of the work in this issue also show a kind of inclination towards that vision.

SPEAKER_01So your Telex Fates Framework I'm curious the extent to which it's descriptive or normative. And also, so there's like the way minds like ours work and there's the way minds could work. So Trebenjo, for instance, like with his scientist AI, like he has this goal of a powerful world modeling system with zero degrees of agency and where there is nothing like intentionality about it, where there's separation of anything like belief and desire. So there's a sense like, well, our minds sure don't work that way, but then There's a question of what are realizable minds?

SPEAKER_01Like, if you do divorce, do you get wrecked by the frame problem? Like, I think you cited, like, like, Vervecki with, like, his relevance realization. So I guess what I'm curious is when you're talking about telex states, there are good engineering solutions for a system trying to get along in the world. But is this for all kinds of minds? Or is this just minds like ours or ones that we hope to have, like, certain functionalities?

SPEAKER_02Yeah, that sounds... That's a wonderful question. I think the best answer I can give is I don't know. We really, I think, enter a period where the kinds of minds that we are considering, we can consider, is expanding. And that touches on some fundamental questions of what really is a mind. Is mental phenomena fundamental for reality or can it be instantiated materialistically? These are like age-old questions, which I don't think we're going to solve anytime soon.

SPEAKER_02But what I do see is that more and more scientists in different fields, I see this in neuroscience as well, are becoming more open-minded or agnostic or willing to consider alternatives to the kind of traditional materialistic perspective. That mind is just an epiphenomena of some kind of physical system or information processing device. And as someone who grew up,

SPEAKER_02you know, studying mathematics and computer science and physics, I'm sympathetic to the, you know, computationalist motivation and the desire to formalize things. And I'm very sympathetic to that. I think it's very important, but it's also important to realize that many cultures and traditions and philosophies don't view Physics or even computational information as the kind of foundational level of reality. And there are different schools of Buddhism, but one school that now I'm actually particularly interested in is called

SPEAKER_02the Yogacara school or mind only. And they say that actually the most patrimonious description of reality is that mind is Fundamental, or maybe even the only thing we need to explain reality. We don't need to assume that there's a physical world. Almost an illusion, not illusionist, but maybe idealist perspective. And some of the Yogacara philosophers speaking about But karma and collective karma, collective intentionality have developed very interesting ideas that now I believe can be used for describing models of collective agency,

SPEAKER_02collective intelligence. That's maybe even a partial answer to the point you raised, that we are realizing that mind are not these individuated objects separate, you know, inside skulls or something like that. Mind is a collective emergent phenomena and Buddhist philosophy, especially the Yogacara in school, I think, have interesting ideas to contribute to that.

SPEAKER_01I know Michael's going to want to double click on that. Before we do, one more, we're going to get... I guess in terms of the possibility that like a tele-states framework would be important for all kinds of minds we might try to create would be like the frame problem in terms of it seems like by co-entangling your beliefs about the world, With what you want to achieve in this way, it seems like this is a means of paneling the problem of relevance. Because in AI, it's acknowledged that there's a challenge.

SPEAKER_01What is the scope of relevant information in any given circumstance? Theory, anything could bear on it. But if you, in this very interested way, build up your world model and you bake in values the whole way through, It seems like the frame problem is not quite so problematic. I don't know if you would agree with that.

SPEAKER_02Yeah, totally. I mean, so this Dharmakirtian perspective specifically is a theory of concept formation that Dharmakirti developed, which is called a Poha theory or exclusion theory. Is a way of addressing the frame problem or what Viveki calls the relevance realization problem, which to put it really briefly, the idea is that we form concepts by ignoring everything that's not relevant

SPEAKER_02to our goals. So when we call this A pen, for example, we're ignoring all of the aspects of this object that are not relevant for our goal of writing. That's why we can call this and another object we can call also a pen because both can serve the same goal. And we do this by excluding Almost all of the features of this object except those that are relevant for our goal. And that's a way of facing this same problem or the problem

SPEAKER_02of having this potentially infinite number of features or aspects of experience we could achieve. I tend to, but having a goal as a fundamental driver of our cognition allows us to ignore everything that's not relevant and create these concepts in a, I guess, cognitively efficient way.

SPEAKER_00So in listening to you over the last few minutes, I've been hearing this piece about how the descriptive and normative are multiplexed in society in a way that sounds to me analogous to how in this paper you talk about Brain research showing, for instance, that in prefrontal regions, value information is multiplexed with map-like information. For instance, experiences belonging to the same telec state represented differently at the sensory motor level.

SPEAKER_00Although let's bracket for a minute that the conversation we just had with John Krakauer, he would take issue with the use of representation here and ask, we talked about his critique of the prevalence of The idea that neural representation is even happening, how do we know? But at any rate, for the sake of just saying, yes, two different hand movements toward the same button Are going to be represented and deliberated upon by the experiencing agents as functionally equivalent.

SPEAKER_00And so the thing I do want to double click on with you is the non-neutrality of like how we look within our brain and this multiplexing of map and value shows up in, and any architecture student will tell you this, right? Or like design people. This is very commonplace understanding. This is very commonplace in the humanities generally in critical disciplines. They'll say our values are embedded in the way that we structure our built environments.

SPEAKER_00And so to take this back to the question of how So then we embed goals in built systems more deliberately and in AI systems specifically. It seems like at least one lens has to be focused on a kind of mind first explanation. You were saying that individual experience being fundamentally a collective experience in the way that individual selfhood is something that,

SPEAKER_00at least from one angle, emerges within collective activity. I know Adam and I are both fond of the David Krakauer and his co-authors work on the information theory of individuality, which is looking at this kind of evolutionary story for why individuals exist. And it starts... Yeah.

SPEAKER_02So the parts about the brain, I mean, I don't think that what we're saying is, I mean, in some sense, maybe it's radical, but in another sense, it's very intuitive. And as you say, well understood by many disciplines and many thinkers, I think it might be a kind of historical thing that By thinking of the brain as a computer or kind of using these engineering enforcement learning or control theoretic models, we've come to at least formally describe learning and decision-making in

SPEAKER_02a way that's It separates the normative and the descriptive aspects. And you see this in decision theory, in control theory, in all the kind of frameworks that fed into how we model cognitive systems today. There's always an underlying assessment. And you don't really have the language to consider the

SPEAKER_02More delicate relationship and interaction between those two aspects. Now, obviously the brain itself doesn't have to obey our models and mathematical descriptions. So you find this coupling much more easily if you look at the brain. And you see that you don't have to call them representations, but neural responses or whatever incorporate both mappings or cognitive maps and evaluations or goals. For example, in that answer item system, once it was thought of as This kind of objective map of the world,

SPEAKER_02or I don't know, I'm not a hippocampus person, so I'm probably oversimplifying the story here, but the responses of the entorhinal system or the hippocampal system do not map nicely to the clear orthogonality between how we map the world and what we want to achieve in it. You see responses to both of these aspects of the animal's interaction or experience. And I think we should also have the theoretical language to look for these connections and describe them more explicitly.

SPEAKER_02And it shouldn't be something very new or different. It's just a shift in perspective, in my view. But we need the theoretical language and the kind of grounding to see that, to see this kind of gorilla in the room that we currently, I feel, tend to ignore. Because of the mathematical and theoretical languages we use. And the mind only stops. I think that's still a bit in development. I would say that's something I'm thinking about now, but we don't really have yet.

SPEAKER_02A concrete model or complete description of that. I do think though that the whole multi-agent or collective agency dimension will play a big role there. We actually do have a paper on that, hopefully coming out soon in topics in cognitive science, extending these ideas of Dharmakirti and the Teric States, currently only in a kind of conceptual way. We're still working on the mathematical description of this, but extending it to the context of collective agency,

SPEAKER_02where agents can stimulate the experiences of not just themselves, but also of other agents. So then the aesthetic state or the representations they form take into account experiences of multiple agents and preferences can be applied to these joint experiences, not just Preferences on my own experiences, but I can have also preferences on your experiences or Adam's experiences or collective experiences. And that also can help us transcend this fundamental tumbling stone,

SPEAKER_02which is this idea that we have a fixed and rigid self.

SPEAKER_01In terms of maybe both the ways of thinking about conceptualizing a mind-only ontology and also collective selves and agency, to what degree would you say, in Carl Fristen's work on the free energy principle, could that be considered to be models all the way up and all the way down? Would that be potentially an example of something that's like mind is fundamental? Also, I guess with Mike Levin's work also in terms of information bottlenecks all the way up and all the way down with quasi-agenticness about them, would that be like in the direction of what you would say would be like

SPEAKER_01a mind-only ontology or is that different?

SPEAKER_02Yeah, it's definitely in that space. And I've seen also work by Fiston recently on extending active informs for multiple agents, collective agency, collective remarkable blankets. They like to use those terms. I know actually that both Fiston and Mike Levin have been an Still are engaged in discussions with Buddhist thinkers. I just saw a panel discussion with Fiston. Jan Westerhoff, who's a famous Yogachara,

SPEAKER_02a mind-only scholar, they had a panel discussion, which was quite interesting. So it's definitely going into that same space of ideas. And Christian's ideas have also been applied in the context of Thomas Merzengel's projects on the minimal, phenomenal experience models. So there's a lot of richness in that interface. My feeling is that they're still not, at least in the kind of active insurance frameworks I've seen, the coupling between the normative and the descriptive is,

SPEAKER_02for me, still not explicit there. It probably can be bought out, but I feel that in the Darmakirtian or the Buddhist model, it's very explicit to begin with. And that's the direction I've tried to take in my work. But obviously, there's a lot of value and interesting work done in the... Active Influence and Michael Evans' work as well has been collaborating with the Center for the Study of Apparent Selves. And they also actually just published frequently a paper called Selves as Perspectives,

SPEAKER_02which reminded me of this paper by Laurie Paul, you mentioned, where selves are viewed as a way of seeing the world. And two people who have a similar experience are in some sense the same agent or something like that. I don't want to simplify that, but. I have

SPEAKER_00just this one more question, which is really just bringing the last question I asked you down to earth and linking it to earlier comments on the Benjo and Lacoon dialogue. In this paper, you talk about how the process of selecting And then you have a whole appendix where you formally explore one So this is where the question of understanding the way

SPEAKER_00that our technologies instantiate both our maps of the world and our values in the world either do or do not satisfy I think about people like Oxford economist, Eric Beinhocker, studying the conditions of eudaimonia or human flourishing.

SPEAKER_00And that's on one poll of this. And on the other poll is It is the question of limiting undesirable expressions of the agency of our machine systems. And so this is what I'm curious about really is like, how are you formally thinking through the ways in which we We have accidentally designed for ourselves and could more intentionally design our way back out of systems that confine our ability to transition between different telex states

SPEAKER_00that fix selfhood in a changing world that make us more like the kind of reinforcement learning systems that we've been designing. That are not capable of moving between different representations and deriving different goals for themselves. And then how might that kind of thinking improve AI governance conversations?

SPEAKER_02Yeah. I mean, I don't know if I can really speak to all of that, but what I can say is that I think that meta-awareness would play a big role in that. So psychologically, we often operate on a kind of autopilot and I'm very fond of Carl Jung's psychology and he has this notion of archetypes and the collective unconscious. And I think we often form these kind of archetypical The goal of becoming more self-aware or enlightened,

SPEAKER_02if you want to use the Buddhist kind of ultimate set point, is to realize what's driving our automatic responses and then By realizing that we can also make that more flexible and more amenable to change. And in this work you alluded to in the appendix, so that's from a paper we had on how teleic states can be transformed in order to have a more, we call it controllable.

SPEAKER_02So this comes from control theory, to have a more controllable representation. So we can choose our goals in a way that That makes our presentations more useful in a very general sense of useful. So not useful for a particular goal, but useful for being able to easily reach all the possible states So the controllability means the ability to switch between all possible states. And by linking the representations to the goals,

SPEAKER_02we can start to consider how selectively Collecting a certain goal would be helpful in creating a controllable representation. And that's what we try to make more explicit in this appendix. And I think this also catches back on the transformative experience point we discussed earlier. Because changing our goals is, in this sense, a transformative experience. It changes our presentations. And people often say that when they had a transformative experience, their value system changes.

SPEAKER_02That's maybe a way even of characterizing a transformative experience is to say that my goals have changed, my value systems have changed. And what we're trying to show in this appendix is that also means that our representations change and we could strive to change our representations in a way that's beneficial or gives us more controllability or more. It's also related to the notion of empowerment that Daniel Polanyi talks about, having a general useful criterion for representations through updating

SPEAKER_01our goals. If you have an agent where it's foundationally governed by intrinsic drives for empowerment, trying to have maximal controllability over the environment and maximal ability to achieve goals, would that then be likely to result in a telec state agent or would you need something additionally? And I guess one more thought in terms of like perception, so like the telec state framework is different than having perception being just like a impartial state estimation. How would it relate to something like the affordance competition hypothesis or

SPEAKER_01the idea of from ecological psychology, like our perception is naturally structured according to relevant affordances in different ways?

SPEAKER_02Yeah. I think I'll have to think about that a bit more, but what I could say that the telek states framework is at a different level than empowerment, for example, in that we don't assume that the I mean, we do have this other aspect of the work about controllability. So if you assume that what you want is controllability, then we show how changing your goals can help you. You achieve more controllability.

SPEAKER_02But I think that the fundamental idea is more basic than that. It just says, if you give me your goal, which we define as a set of preferences over experiences, if you tell me what kinds of experiences you prefer, that would determine what kind of representations you would form under the telec states. So empowerment could be one way of defining what do I prefer? I might prefer More empowered experiences or situations where I have more control.

SPEAKER_02But I wouldn't want to limit myself just to that because that's not... My focus is just to point out that there is this important connection between preferences and representations and try and characterize that rather than say what is the correct fundamental intrinsic motivations that we have. That would depend on the framework you choose. And actually, Buddhist philosophy would have an answer for that, I think, as well.

SPEAKER_02They would say that the goal should be alleviating suffering in yourself and in others. But how to translate that to a formal kind of description is still a way to go, I think.

SPEAKER_00Well, Nadav, thank you so much for joining us again.

SPEAKER_02Yeah, thanks. It was fun again. Yes.

SPEAKER_00Thanks again for tuning in. Every episode of this show takes weeks of research and post-production. Consider subscribing at michaelgarfield.substack.com like new and beloved patrons Emily Nguyen, Alison McDowell, and Ben Lockhart to join our selectors. I loved this book. And the conversation is a real zinger.

SPEAKER_00In the meantime, take care and remember, awareness, not attention. I was wrong about that. It's awareness. Awareness is our greatest natural resource.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Why Can’t AI Laugh at a Joke?

    Nov 6, 20261 hr 39 minAudio

    With AI on everyone’s mind, we have to ask: what does it mean to think? When does the brain actually create predictive models and rely on mental representations - and when is it really just a matter of inputs and...

    Credit the people on this episode

  2. Continual Learning: World Models in Natural & Artificial Intelligence with Adam Safron

    Jul 30, 20261 hr 18 minAudio

    What does it actually mean for a machine to “understand” the world? Are today’s auto-regressive LLMs truly reasoning, or is statistical text prediction fundamentally distinct from genuine deliberative agency and causal...

    Credit the people on this episode

  3. Visionary Ventures & The AI-Native Avant-Garde with William Morgan of Restless Egg

    Jun 25, 20261 hr 4 minAudio

    Today’s guest is William Morgan, the co-founder of Restless Egg, a London-based incubator and accelerator designed specifically for a new class of “artist-founders” whose work synthesizes art, technology, and commercial...

    Credit the people on this episode

  4. Rewilding Collective Attention: Bioregional Design for Life Online with Andrea Farias

    Jun 16, 20261 hr 27 minAudio

    ✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future ✨ Browse and buy the books we mention on the show at Bookshop.org ✨ Stream and download my music at...

    Credit the people on this episode

  5. Games & Metrics: Agency as Art & Artifice with C. Thi Nguyen

    May 26, 20261 hr 36 minAudio

    ✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future ✨ Browse and buy all of the books we discuss on the show at Bookshop.org ✨ Stream and download my music...

    Credit the people on this episode