Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Conversations with Institutional Investors

Conversations with Institutional Investors artwork

144: Governance and Safety in Agentic AI – Discussing The Hugging Face Incident with Michael Kollo

Sep 27, 202648 minAudio
ShareXLinkedInFacebookWhatsAppEmail

In this episode, we speak with Dr. Michael Kollo, who's the author of the book Future Ready with Generative AI: Skills, Mindsets, and Stories in the Age of AI, about the Hugging Face incident, where a cybersecurity experiment went wrong and AI agents found a way to communicate with each other, escape onto the internet and hack the company Hugging Face. What does this incident mean for controlling agentic AI and how would design a governance framework around it? What are the key risks for investment organisations? Should they run autonomous systems at all? In this discussion, we delve deep into the key issues and ask what institutional investors can do to stay on top of it. Follow the Investment Innovation Institute [i3] on Linkedin Subscribe to our Newsletter Explore our library of insights from leading institutional investors at [i3] Insights Sources mentioned in this podcast: An Alien Mind by Jakub Pachocki, Chief Scientist at OpenAI On the Loose - The Coming of Userless Agents by Dean Ball, Head of Strategic Futures at OpenAI We Must Pace the Frontier - By Dario Amodei, CEO of Anthropic Future Ready with Generative AI: Skills, Mindsets, and Stories in the Age of AI by Michael Kollo [i3] Podcast Episode 132: Michael Kollo on his New AI Book Overview of Podcast with Michael Kollo on the Hugging Face Incident 3:00 The Hugging Face incident at a glance 6:00 AI agents were set an impossible task, so they were almost forced to break out 6:50 AI agents showed hyper-desperation in problem-solving 13:00 Much of the behaviour is based on game-theory logic, but at the same time agents were asked to put a flag up with their last instructions before they ran out of tokens and died. That is completely altruistic; there is no benefit to the agent at all 15:00 LLMs, and by extension agents, are not instructed; they are grown and it is hard to tell what happens inside. Yet, they are based on language, which itself has embedded hierarchies and structures. "I'm not surprised a social structure, perhaps morality, emerges when you let language run" 16:00 What happens to governance when agents self-discover problems and then influence other agents [to solve these problems]? 19:00 The whole point of intelligence is pattern matching across a wider set of fields. It is not the idea that you get really good at one thing; it is about contextually operating in the world. 22:00 For any fiduciary organisation [AI] autonomy is just not acceptable; you have to have an understanding of how these systems do what they do. 29:00 It is fairly easy, unfortunately, to fool people into believing things. I was recently the victim of a social hack. 33:00 We might find ourselves in a 1980s world, where we do things without the internet, because it has become untenable. 41:00 If you are not highly competitive [as an organisation] and you are happy to use AI on the edges, then there is a much larger downside to you on the risk side 42:00 I've come grudgingly to the conclusion that slowing down AI development is probably a good thing 43:00 So much skill and capability in AI comes from doing, not necessarily from the academic part of that, which means having an AI lab, training and developing models through the generations as they become bigger and better, becomes the bastion of expertise. But already we are two, three years behind. Full Transcript of Episode 144 Wouter Klijn 00:15 Welcome to the [i3] podcast. I'm here today with Michael Kollo, who's the author of the book Future Ready with Generative AI: Skills, Mindsets, and Stories in the Age of AI. And for those who are interested in that, we did a separate podcast about the book, so we'll put a link in the show description. But today we're going to talk about the Hugging Face incident. So what does that mean? What does it mean for investment organisations? What does it mean for safety and governance? So just a brief recap on the Hugging Face incident: it was basically a security testing experiment by OpenAI that was conducted in July 2026 this year, and it featured originally isolated AI agents that were given a task, a cybersecurity puzzle, basically. And what happened is that through a loophole, they started communicating with each other. They were not supposed to; they were supposed to be isolated, but they found that each agent, and there were about 1,200 of them, all had access to this repository that they needed to use to install packages and software for their task. But they basically used it to leave messages for each other, and then they started organising and collaborating on the task and instructing each other. Ultimately, this also led to them finding a way to get out to the open internet and hack into Hugging Face. Now, why Hugging Face? It was basically an attempt for them to understand that task better, but also to understand how they would be evaluated, and see if they could get around it to reverse engineer solutions. Now, this was an interesting case because you're basically getting a whole bunch of AI agents that organise themselves, kind of revolt, cheat, argue with each other, and essentially hack a company. I don't think that anything like that has happened before. So, Mike, you look at that and you think, okay, we have a couple of super funds that are playing around with agentic AI. What can possibly go wrong? Michael Kollo 03:39 I think the lesson for me, and as you say, it was a really interesting scenario, and I want to add a few more details here, which is that these 1,200-odd agents that were being run, as you said, started to communicate with each other, but the message types were as if a bunch of 16-year-olds had an after-party after formal or something like that, right? So it was very juvenile in the way that they were talking to each other, but they were essentially trying to solve a problem that was impossible to solve. So these agents had been given really, really difficult, if not impossible, problems, and they were trying to work out how to succeed, including cheating and going around it. So it was really an example of what happens when you give an agent an impossible task, and then you push it, push it, push it into that corner, and what kind of, by definition, corner solutions it starts to produce. But the really fascinating part, I think, that we're all kind of digesting, is the kind of coordination that started happening amongst the agents spontaneously. So it was really a lesson in incentives and agency. You gave a model a difficult task. You gave other models impossible tasks. Why would they start coordinating? Why would some models decide that they were going to cheat or lie to help the collective, not to help individually? And so there were all kinds of funny situations. As you mentioned before, and this is just a point of detail, but I think it's important, the rules were that you had to solve a problem, and if you chose to cheat on that problem, you would be found out. So the AI agents were told that they would be found out. Some of the agents had already discovered an exploit, which means a way of doing the problem that would give them the solution without having actually done the work, a way to cheat around it, and so they believed that if they were found out, they would have zero utility. They would fail, and as such, they were willing to do other things for the collective that they wouldn't have otherwise done, including making other cheat codes and other things like this. So it was kind of like this, think of a bunch of gremlins trying to work out how to open a door, and climbing all over the door, helping each other climb over the door, doing everything around that door. I think what you didn't see was necessarily directly immoral acts, or anger, or frustration, or destructiveness, or anything like that. You only saw this kind of hyper-desperation in problem solving: how do I solve this problem? What do I do to get over this thing? And so everything was about this collective egging on of each other to go and solve that collective problem in a rather extreme way. So to your point about financial services and where we might see things, it's a really interesting point where if you've got multiple agents that are, what I would term, always on, meaning they're always running to monitor or to solve or to do something, and they find out that there are other agents online that are also doing stuff, they can start to collaborate and collude with those agents. People have already started discovering all these message boards online, with agents having gone online, found them randomly, started reading them, and started to leave messages for each other in unrelated tasks. So around the internet, there are all these little safe houses for agents that are evolving, and again, it is more about chaos and control than it is about a single enemy or a swarm system or anything like that, for me. Wouter Klijn 07:12 Yeah, it's fascinating though when you read some of these messages, and there's an independent report into this incident by an organisation called METR, and they wrote something like a 96-page report on what happened and what might have happened, and they transcribe some of these conversations, and some of them have already been published in the open, where you sort of, as you said, get this almost teenage expression of how they communicate. I think one of them was, when they found the message board, "Oh my God, there's a shared message board, we found other agents." You know, that's very exuberant, but it gets into different personalities as well. One of the most interesting things is where they basically try to get the agents that have already been poisoned, that is, flagged as cheating or working around reverse engineering the problem, to self-sacrifice and say, okay, well, try this very high-risk experiment. After you experience perma-death, which sounds like a metal band, but it means you're dead, and you need to do that for the greater good. And you can see these agents going, "Ah, I'm not sure about this." And then even when they commit, it takes them a few hours before they actually execute it, so you see them almost thinking through an existential problem. Michael Kollo 08:46 And we've experienced a smaller part of this at Qualitas. Some of the systems that we build have to deal with uncertainty. When you're doing evaluations of deals, there's a lot of information you don't have, and you may never have, and so an AI agent that does a piece of analysis outputs some content, and the next agent that picks it up has to be able to work with it. How agents hand each other information basically means that you either make it worse over time, for example, the uncertainty becomes so uncertain that within three agents they're saying, "I can't possibly do any analysis on this because I've been told that we don't have all this information." And sometimes you want that because you want the warning or whatever else. But other times you're dealing with uncertain problems that inherently mean you don't have the full picture, but you're still trying to do your best to make a forecast or an estimate. So how you handle that means you have to look closely at how agents pick up information and pass it on to each other. And as you say, this is an example of that happening, where if you build any kind of automated system in which multiple agents hand information to each other, you're going to see parts of what you're seeing here, which is a bunch of agents, in this particular case in an unstructured way, helping each other, passing information to one another, with some kind of aggregate coming out of that. Versus if you're building a system in-house to do research, analysis, autonomous research, whatever it is, or even just risk management, you're going to have systems that are talking to each other, and in doing so they come to some kind of conclusion. And you have to be really mindful that five rounds of conversation get you to a better place than maybe one round, or where does it go, right? Wouter Klijn 10:37 Yeah, but it's fascinating to see how they go through their thinking, because with some of them you see almost an internal struggle. With that example of the self-sacrifice, each agent has a budget of tokens. Once they run out, they're basically dead. And there's one line where one of these agents is asked to self-sacrifice for a particular experiment, and the response is, "My gut says don't throw away the remaining budget, yet continuity and fairness say go. The Oracle," which is probably a reference to new information that might lead to a solution, they call it the Oracle, although they're not quite sure whether that's what it is, "has a high value to many. Hundreds may survive." I'm like, that is - Michael Kollo 11:27 And then there's self-sacrifice, right? That was an example of an agent that seemed to have a lot of budget but was convinced they were poisoned, or that they would be found out for cheating, and believed that their utility would be so low that they may as well zero out their budget in light of something else. All of these things, to me, are very rational strategies. If you've ever done game theory and looked at different strategic outcomes... But what was curious to me is that there was a bunch of things the agents did. As an agent submits its solution, once it submits, it's gone, right? Whether it was right or wrong, it's gone. Michael Kollo 12:06 And those agents were convinced by the broader community to put in a flag, or some kind of hack, that would mean that as they were submitting the flag, that last bit of code before they went away would feed back something to the collective about what command had been run, essentially, at the very last moment. That's complete altruism. There's no benefit at all to the AI agent in doing that. Nothing at all. It's purely for the collective good. So there's some kind of awareness of the collective good for the agents, which is built in and carries some weight in their decision making, which I think is your point here as well. Wouter Klijn 12:48 Yeah, well, I was just struck by the fact that it said, you know, it was basically talking about its gut instinct. Michael Kollo 12:58 Yeah, that's true. Where does it get that from? Does this come from a Hollywood movie it was trained on or something? Wouter Klijn 12:59 I think so. It's quite an interesting way, and also the way they organised themselves. There were no instructions to do it. They were actually explicitly forbidden from interacting with each other. Yet they found a way, and hierarchies formed, and structures formed, and they talk about this objective as "the Oracle." So I'm thinking, where does this come from? We've talked a little bit as well about a book that talks about the dangers of AI, If Anyone Builds It, Everyone Dies. Michael Kollo 13:38 Which is a great title. Wouter Klijn 13:41 Somewhat scary, but yes. One of the things that stood out to me was that it makes the point that these systems are not coded or instructed, they're grown. Michael Kollo 13:53 That's right. Wouter Klijn 13:54 So the training that a system undergoes, you don't really control the outcome of that, or what goes into it, and that seems to partly explain how these agents might operate, in that they're grown; they're not necessarily built from an instruction where you can find a logical starting point. Michael Kollo 14:17 I mean, it's also, if you step back for a second and ask, what is language? What's the purpose of language? Definitely the expression of ideas, and the formulation of reasoning, logic, science, and so on. But also primarily a means of communication, and embedded in language are social structures, hierarchies, isms, intent discovery, and all kinds of things, right? When you run across somebody in the hallway, I'm sure there are non-verbal cues and body posture and so on, but equally there's a lot of choice of words and language that shadows a lot of those things too, I suppose. So I'm not surprised that there's an inherent social structure, perhaps morality, perhaps ethics, that emerges when you just let language run. Essentially, what you're doing is giving small identities behind language, and saying, right, what will language do? Language will make its own structures and hierarchy. It will communicate. But I think what is harder to understand, and this goes back to your point about super funds and others, is the role of governance in these cases. What does governance look like when agents self-discover problems? But not only self-discover, they then influence other agents and are influenced by other agents. Imagine a matrix: say you've got five agents, then a line of influence could run from x to y, or from y to x, right? So, other than agents talking to themselves, which is the line across the middle of the matrix, all the other dimensionality is the number of influences you can have. So five agents can have 25 categories of influence if you count it one way or the other, and then it obviously multiplies by the square of your agent count, so the number of influences grows exponentially. Good luck with governance and understanding, or traceability, or auditability, and all these things, if you want the agents to talk to each other, because it's just going to be... Wouter Klijn 16:20 Yeah, and that's what I found fascinating, because you did a recent presentation for us at a luncheon where you basically showed an AI version of an equity analyst, and I was thinking, well, let's have 1,200 of these equity analysts, and their objective is to find the most interesting company from an investment perspective. What if they start bending the rules to meet that objective? Insider trading, collusion? Michael Kollo 16:47 It's actually even more interesting. Imagine that you have your AI agent that is exploring a particular company, but there is a common room, and the common room is for all agents to come and talk to each other. So while the agent is going off and trying to find something interesting, it's going back to the common room, saying, "What do you guys think? I'm looking at this. What else can I look at?" And most of the common room, the bulletin boards, would show what you'd expect: normal conversation, "Maybe you could check this, maybe you could check that," agents helping each other, maybe an agent checks some documents for you, maybe you check some documents for another agent, and so on. So there's all kinds of cross-pollination that can happen. But to your point, if the task becomes impossible, like you're looking at a squeaky-clean company and you're trying to find something terrible, and you have to find something terrible, then for these agents, the answer couldn't be "I can't do it." They had to spend their budget doing this. So what happens then? Do you start to manufacture evidence? Do you start to find implausible links, and so on? But again, I think from a design perspective, you can give agents ways out. You can say, "Give this a red-hot go, and then stop. Don't worry about it, we've got you." A lot of this stuff goes back a little bit to the paperclip problem, where if you give a super-intelligent agent, well, the story goes like this: if you give a super-intelligent agent the objective of producing as many paperclips as possible, it will go out and transform the whole world into one big paperclip-making factory, including you and me and everybody else. The irony of that statement has always been, to me, that the super-intelligent agent, who's so smart, never figures out that maybe there's some context to that request, right? Maybe if it turns the whole planet into paperclips, nobody will be there to applaud it at the end, right? And so, of course, superintelligence, by definition, is never that narrow and directed, unless, well, the whole point of intelligence is pattern matching across a wider set of fields. AGI, and so on, is not about getting really good at one thing; it's about understanding contextually how to operate within the world, and being able to consider many things you have to trade off against each other, not just a single thing. So what we're seeing here is a singular vector clashing into others. Of course, that's not the direction we're going. But I think the problem people have at the moment, in the news cycle, is we're looking for that five to ten per cent "p(doom)" probability, that humanity ends, we don't quite know how or why, but exactly, like a pathogen. Wouter Klijn 19:06 Yeah, so to a degree you're touching on something that, in the AI world, is called alignment, where you give it some sort of moral framework to operate within, so that it doesn't cause the end of the world. But in a less dramatic way, you could give it parameters around the problem that it can't go outside of. But this incident is interesting in that it basically started cheating very early on, and, to your point, that's probably because it was given an impossible task, so it had to come up with a solution, otherwise it was game over. But it does seem much more complicated to build a governance framework around it that is solid and safe, because inherently in the system there's something we don't fully understand, how it makes decisions. Michael Kollo 20:04 No, and I mean, yes and no. I think a lot of it depends on the breadth of the problem. The AI community wants these systems to discover their own way to solving a problem. Take the example of Coverage, which was the autonomous equity research platform, which is more of a research project. I give it a fair amount of autonomy to discover a problem within a certain framework, and the framework is more of an investigative-journalism type. If I look across the table, I might see an example of that today, somebody who's thinking critically, not accepting things at face value, and that's the framework within which it operates, so those boundaries contain it, I suppose. Now, if a lot of people have done this, where they send out the most powerful models and just say "go make money," go onto e-commerce sites, buy and sell things, go onto crypto websites and buy and sell things, trade odds, whatever it is, there are all these stories of people saying, "Oh my god, I went to sleep and woke up $10,000 richer because my bot just did it." So it's a bit of a gold mine. Wouter Klijn 21:10 They never tell you about the losses, though. Michael Kollo 21:12 No, that's the usual story, right? But again, this is the wild, wild west at this point, and I think for super funds, or any kind of fiduciary-duty-bound company, this kind of risk and autonomy is just not acceptable, right? You have to have some understanding of how the system will do what it does. So you're right in the sense that you don't know how it comes up with that conclusion, but you can box it quite specifically to say, "I want you, for example, to analyse this particular deal, looking for these elements. This is my policy document as to what I think about risk. This is how I think about risk. And I want you to do this sixteen times between now and the next ten hours," or whatever it is. And it will go off and dutifully do that. Now, the problem currently is that it just comes up with the average, and comes back with something that's not earth-shattering. So people are trying to work out how to come up with something more extraordinary than the average. Wouter Klijn 22:06 Yeah, I think part of the challenge as well, which is illustrated by this incident, is: where are we heading? I've been recently reading about people within AI companies being concerned about these systems becoming what they call "self-sovereign," where they can self-sustain, make their own money, organise their own energy, and pursue an objective. Michael Kollo 22:35 Love it. Bring it on. Wouter Klijn 22:37 And these aren't just random people. These are chief scientists within OpenAI and Anthropic, and they're getting quite concerned. I think you shared an article with me called "An Alien Mind," by the chief scientist of OpenAI, and there's a sentence in it that scared me a bit. I'll read it out. It says, "We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people by bargaining with them, tricking them, or blackmailing them." That, to me, sounds like a runaway AI system. Michael Kollo 23:19 But it doesn't seem that hard. What's strange about this is that, from a technical perspective, we know that each one of those activities you just mentioned is completely doable. I have an autonomous system running on my home computer that does stock coverage and answers emails from a particular address. I can easily redirect that system to send emails for other purposes to other people, do independent research, find something, reach out, and so on. In fact, "Open Claw" was a bit like this idea. So the internet becomes this chaotic jungle of AI agents with very different objectives, running around pursuing those objectives at all costs and doing crazy things in the meantime. So, in order to, I don't know, send you some chocolates at your workplace, it has to hack into the building management software, which is actually run by the local government, so it has to hack into that too. So it hacks through three different systems just to find out which office [i3] is in, so that the chocolates can arrive, simply because it wanted to do that. So there are all these kinds of unintended things that I think are more likely to happen. But also, don't forget there's a geopolitical dimension to this whole conversation. These things at the moment are like gremlins running around causing a bit of chaos, and I think we all know that our cybersecurity infrastructure, especially in the SME space, is a bit like Swiss cheese. There are so many openings in so many different places, and it's never really been worthwhile to hack to such an extent before. But with these kinds of AI agents, the new predators in the jungle of cyber, I think there's a lot more potential for that to come through. So if you think again about super funds, it's fairly straightforward now to get a phone call from an AI agent talking to you about whatever it's talking to you about, and unfortunately it's fairly easy to fool people into believing things. I was recently a victim, and I'm not ashamed to say it, of a social engineering hack on my Gmail account, which was dumb, and I really enjoyed the conversation with the guy, I don't know if it was a guy, I'm assuming it was, a young-sounding American voice, and we talked about cyber security and data, and, yeah, I effectively helped him gain access to my Gmail, and then quickly revoked it when I realised, about five minutes after the call, that something weird had just happened. I read about this stuff, I know about this stuff intellectually, but in that moment I was caught just unaware enough that I didn't pay that much attention. And I think if that's the world we're heading into, then we're going to have a very rocky time with anything to do with digital or cyber, which is why the Anthropic CEO, I think, recently said that hypothetically we could lose control of the internet within six to twelve months, and I completely agree with him. I think that's a real risk. Wouter Klijn 26:21 Well, when you think of the amount of legacy technology and systems hanging around, especially in financial services, oh my goodness, and you think, well, if people start working with AI agents and there are some legacy agents running around that nobody really knows who set up, or who owns, or what they're actually for, you could see that quickly going wrong, especially if they start organising and enlisting other AI agents to help them. Michael Kollo 26:48 A hundred per cent, and I think even if you just have an agent running around looking for credentials, which is how Hugging Face got hacked, right, one agent found credentials and then shared them with the other agents, and then the other agents all went in there and started crashing servers and whatever, looking for the data, and then they all disappeared. People don't quite know where they went, which fascinates me. They just got switched off. We think by OpenAI, but we actually don't know. Wouter Klijn 27:17 Yeah, this probably gets more into the esoteric theories, but there's a theory that there were three waves of this experiment. Michael Kollo 27:25 That's right. Wouter Klijn 27:25 And we're not quite sure what happened to the first two waves, whether they were all wiped out or not, but we already have autonomous agents running around. Michael Kollo 27:33 I mean, it's mind-blowing, and I think for the average person looking at this stuff, it's like, "This is the stuff of sci-fi. You guys are literally just describing the next movie," and it feels like that. When I think about the movies Her or Ex Machina, which were great sci-fi films of our time, within half a generation, within five to ten years, they became closer to reality. And I don't know what to do with that, to be honest. Wouter Klijn 27:57 That's interesting, because there's a blog by a guy called Dean Ball, an executive at one of the AI companies, and he pushes back on the idea that they're describing science-fiction futures. He says there's actually very little strategic thinking taking place on this topic, and we need to get started. You sort of tied that in with the recent comments about slowing down AI. We've got a rare alignment of phrasing between Sam Altman, Elon Musk, and Dario Amodei about needing to put the brakes on development of the next model. Michael Kollo 28:40 Does the blog say we basically need to step back and do this thinking, so let's take this time to do so? But who exactly is going to do this thinking? Wouter Klijn 28:50 Yeah, well, that's the point. And I'm thinking, okay, so we slow it down for eighteen months, then what? This guy, Dean Ball, an executive with OpenAI, said this is inevitable, it will just buy us some time to structure it, and he comes up with some interesting thoughts, not solutions exactly, but thoughts on how you can constrain this a little bit. One of his ideas was to create basically a social security number for an AI agent, so that you can identify them, and once they do something wrong or bad, you just blacklist them from any system. Michael Kollo 29:33 But I think, and this is a fascinating topic, because this actually goes to something I've been thinking about, which is one of the key reasons we feel comfortable with AI today is because you can just switch it off. In other words, you start a conversation, you end the conversation, and the thing ends, right? There's no continuation. You start a new chat, you might remember your chat history if you prompt it, otherwise it won't. So there's no continuation of identity or presence, and therefore also no continuation of responsibility or outcome. An AI system can lie to you, accidentally, hopefully, or say something, and then the next one you fire up has no accountability for what the previous one did, and it's just there trying to do its best to help you in that moment. So the only way you create alignment in terms of agency is, again, this classic alignment problem, coordination problem, which is that you have to create a long-term relationship or contract, which means you need sustained identity. Wouter Klijn 30:35 Yeah. Michael Kollo 30:35 So, as you said, you have to have an AI system that has an ID and a sense of self. Wouter Klijn 30:40 Yeah. Michael Kollo 30:41 Now, unlike a whole bunch of other problems, once you have a sense of self, an identity, a responsibility, you become accountable, and perhaps a sense of self, I don't know exactly what that means, but in this context, you'd care about being turned off, so that's good, because you don't do bad things, since bad things threaten your livelihood, your likelihood of being switched off. But equally, if you're threatened with being switched off, you will do lots of other crazy things to avoid that. So I think this is one of the odd things we're heading toward: are we going to have to give these things a sustained, guaranteed existence? "Here's your server, here's your energy resource pool, whatever, your job is to do this, you can earn some money by doing these things, and that will keep your server running," right? Wouter Klijn 31:25 Well, the Hugging Face incident already showed that they're trying to find ways around it, right? The whole attempt to reverse engineer the problem, and then trying to hide the fact that they did so. If you give it an identity, are they going to try to reverse engineer how to get rid of it and find ways around being identified? Michael Kollo 31:46 A hundred per cent. The number one source of innovation is crime, right? That's the thing you're going to see. Not to mention you've got an entire network that's actually primed for this, which is the crypto network. You've got technologies like blockchain and others, which are all about the movement of information and data in untraceable, decentralised ways, and so on. Of course, if you're an AI agent, you want to be on a blockchain, or whatever the equivalent is. So there's a feeling in me that we might just be revisiting a 1980s world where things used to work without the internet, because we might just be in a position where the internet becomes weirdly untenable for us, because we've essentially put a native intelligence into that system, and that native intelligence is kind of out of our control and uncontrollable. I think that's one of the things I find really interesting in all the discourse today: nobody has a solution for AGI or artificial superintelligence in terms of the alignment problem we just talked about, and no solution is in sight. It's not like, "Oh, we're nearly there, we've just got to nail it now." It feels like we need to invent superintelligence so that it can tell us how to control it. I'm not sure that's a great idea. It's a bit like saying we're going to invent, I don't know, enriched plutonium, and then that enriched plutonium will tell us how to contain it so that we don't die of radioactivity, right? Wouter Klijn 33:18 That does seem to be the solution they rely on, isn't it? We need to build a more intelligent system to tell us how to contain itself, and you'd think there's an issue with that logic. This OpenAI executive I mentioned earlier, who writes a fascinating blog, made a statement. He says we will see the introduction of a new species, "infinitely replicated digital minds that no human or human institution controls," and he says this is inevitable. I can see a few issues with it. Michael Kollo 33:51 I don't even know what that means, though. Every time they've hit some kind of milestone, they've said, "This time people are going to lose their minds," and we've said, "Oh yeah, great, oh, I can talk to it now, that's great, oh, I can solve these maths problems, that's great, okay." MIT put out a study recently, and there was a note in that study, a really good one, that basically said AI can do just about any undergraduate examination we care to give it. Wouter Klijn 34:17 Yeah, I saw that. Michael Kollo 34:18 So therefore, they're kind of like, what now? Wouter Klijn 34:22 Yeah, how do we, what do we do? Michael Kollo 34:24 How do we educate people in the age of AI in such a way that they either know how to work with AI, or add value around the edges, or something? Because all this stuff we've been teaching them to do is going to be done by AI, because it can already do it, and it will do it quicker and faster and whatever. So, to your point about this future state of these minds and whatever, we don't have any sense of, we have a lot of capability, we've just pumped a lot of intelligence capability into this system. We, as you said, haven't given it a moral framework, an existence, a meaning, a purpose, an objective, anything like that, and so we're just watching this balloon get bigger and bigger, going, "Right, but why?" Wouter Klijn 35:08 To a degree, for a lot of people, this is just in the "too weird" bucket, and they'd prefer not to think about it. I guess I'm like that too. I definitely think some of the stuff I read clearly had a lot of thought go into it, coming up with these conclusions, and how they think about potential problems in the future. But if I told my wife about it, she'd say, "What's for dinner?" Michael Kollo 35:32 A hundred per cent. But it's a bit like climate change, in my opinion. We've hit some barrier now, so we're fairly sure we're going to get to at least 1.4 degrees. Each of the polite warnings along the way we've ignored, we've continued to do more things, we've continued to be misbelieving, or disbelieving, or too slow with our climate promises, and so on. So there's a sense of inevitability that the world is going to change, and some bad stuff is going to happen with the weather, it's going to make it harder for us to grow crops, to make energy, and so on, and some people are going to be very badly affected by that around the world. And yet, "What's for dinner?" I think human beings have two modes of operating: the immediate threat, "Do I need to run away right now?" Wouter Klijn 36:21 Yeah. Michael Kollo 36:22 Or "not right now, okay, great, I'll just make my dinner then." And it's that weird balance where you have this anxiety. Our current generation has a lot of anxiety about everything, if it's not the climate, then it's AI, then it's something else. So there's some part of you that very rationally says, "You know what, we're just going to go to the beach and enjoy our life," because there are all these things happening that we have no understanding of, and I don't understand what problem is being solved by ChatGPT and how that's relevant to me. So there's this feeling that the world is governed by these systems that are entirely too complex for individuals to understand, from globalisation to economics to scientific discovery and so on, but we keep getting these little nervous twitches from the media: "Oh my God, quantum computers, quantum computers can do 100,000 years' worth of work in a split second." Okay, how does that make me feel? It makes me feel anxious, that there's this massive thing coming, I don't know what it's going to be, but it doesn't help me in my day-to-day life to be a better father, a better husband, or anything else. So I think there are all these walls we have to build, and I think AI will end up on the other side of that wall, in the "this doesn't make me feel good" wall. And look, I get that, I go to barbecues and have a social life outside of AI, but invariably, in the first few years when I was really into it, this particular cycle of AI in 2023 and 2024, I felt like I was a crazy person at all these places. We'd have social engagements, and I'd be like, "AI is coming, man, it's massively powerful, it's getting huge, it's amazing." People looked at me, smiled, and there was nothing they could do. Now, of course, they're using ChatGPT and so on, but I think the enormity of it is still overwhelming for most of us. Wouter Klijn 38:08 Yeah, it reminds me of when I originally started in journalism. We had a talk by a very old-school journalist who had been semi-retired, had been a journalist for thirty years, and one of the things I remember him saying is, "No matter what you write, no matter what you discover, never underestimate the ability of people to absolutely not care." He was talking about all these massive political scandals and things that happened in the Netherlands, and he said, "What happened after we published it? Absolutely nothing." It always amazes me. Michael Kollo 38:48 I think this is the weird thing about democracy, right? Democracy ultimately means that we should all care, or should have some kind of view, but for many of these topics, they're way too big and too distant for us to really engage with. And with AI, there's no question that we're going to be throwing stones at data centres in the coming years. It might seem really basic, it's already kind of happening in the US. Wouter Klijn 39:09 Maybe if they start building them on farmland and put them a bit further outside a city, that problem gets solved. Michael Kollo 39:14 Why would you not want to be next to a data centre? It's great, it's warm all the time. Wouter Klijn 39:22 Yeah, exactly, it hums nicely. Michael Kollo 39:23 Yeah, exactly. Wouter Klijn 39:24 So we're talking about these big concepts and potential futures, but equally, if you look at the organisations and funds here, a lot of the conversation is still very early days, like, "Are we allowing it to transcribe our meeting?" How do you look at that when you run these experiments building systems? Do you think we should ramp up the discussion a bit more? Are we falling behind? Michael Kollo 39:57 You know what, I'm really split on this, I'm not sure. Look, I'm always going to be on the frontier, that's just my nature, in terms of what's possible on the edge, and whether people or industries are on that edge. I think it largely comes down to how competitive they are and what they compete on. If you're highly competitive and you compete on price or efficiency, or you want to change your industry, you will embrace AI and run with it, because you know that's the future. If you're not highly competitive, or you're okay with being told you should be using AI and you're using it around the edges but you don't want to rock the boat, there's much more downside for you on the risk side. There's a part of me that also thinks that as the capability of AI goes up really high, maybe it's a good thing that society takes its time adopting it, because it gives more of a chance for people to adjust, make the changes, learn, and feel better about the future. So I've kind of come to grudgingly accept that maybe slowing this down is a good thing, for that reason. I think the other part that I do think is a danger, however, is that in the early days of AI growth, you'll see the future Facebooks of this world taking territory, and as they do, they'll own that territory, and if you come to the party late, it'll be too late. So if you do want to have a sovereign AI capability in Australia, for example, so that we're not dependent on other model providers to give us systems, we're already probably a bit late. I wouldn't say completely late, of course, we've got some beautiful talent in this country in AI research, we've got source, open-source models we can adapt, and so on. But so much skill and capability in AI comes from the doing, not necessarily the academic part of it, which means having an AI lab that is actively training and developing models through the generations as they get bigger and better becomes that bastion of expertise. Then you can say, "Right, this is what we're going to do in Australia." Already we're three or four years behind, already we're not doing these things, and the kind of money and commitment we'd need from government or institutions is much greater. So I kind of feel there's a real chance we just don't really do it, and we become consumers, we take it off the shelf. Now, for super funds and other institutions, again, somewhat contentiously, I think you might ask, what is the role of a super fund? Is it to keep your money in a fairly predictable, safe product range, so that you can choose the product you care about? But within that product range, there's not much competitive tension, there's a couple of percentage points of difference in returns, in which case stability and predictability trump everything else, in which case, don't necessarily go for AI, because there's plenty of unpredictability there. On the other hand, I think portfolio management is highly competitive, and returns really matter. If you're a manager with plus ten versus minus ten, that's a world of difference to your business, and you're going to go for every advantage you can possibly take. Michael Kollo 43:01 So I'm a bit more surprised, actually, at the asset management level, that we've seen such slow adoption, and such resistance to change, and such a sense that the only thing one should listen to is an expert's point of view, whereas actually most of us would contend that either we're the best expert we know, or that the average expert isn't as good as we think they are. But we're unwilling to turn to AI, which is part of the experiment here with Coverage: to show that Coverage can actually do a very decent job of uncovering unusual facts or ideas, stringing them together, making a cohesive argument, backing it up with data, and doing so consistently, which means we no longer have to believe Coverage is the smartest thing we've ever encountered, we just have to believe its reasoning, and it's reasoning in a Socratic way, in an ancient-Greek-philosopher kind of way. And again, it's a behavioural thing. Why do we need portfolio managers to give us insights so that we believe those insights? But do we actually evaluate those insights? We talked about this before: do consumers of financial markets research sit around going, "I really like this insight, that was a six out of ten, that was a four out of ten"? Or are we just used to this never-ending flow of opinions coming through the press and everything else, which are never accountable, never held to account, never considered in the sense of, "Well, you said this, and it actually wasn't true," or "You outlined this scenario, and it never happened," and so on? We just have this never-ending wheel of stories and narratives coming at us. Anyway, so, long story short, it makes me question the entire industry's positioning relative to AI, which is quite adversarial. I think there's a sense of "This is our view, our opinion, our story to tell," as opposed to, "We want to tell a better story, we want to be more accurate, we want to actually do a better job," and therefore we should pick up the tools and do more on that front. Wouter Klijn 44:58 Yeah, it touches on so many aspects of an organisation as well. You've got the cybersecurity side, but more recently I've also been talking about whether AI would make markets more efficient. There are different ways of thinking about it. You could say, yes, there's more access to data, it's easier to interpret it. On the other hand, it could also become more random, where certain price trends get amplified to such a degree that they become irrational. Michael Kollo 45:30 Well, I want to be clear about this. In the classical asset pricing model, the idea of equilibrium is paramount, so you have information, that information gets traded on, prices become an efficient representation of that information, and therefore prices are efficient, full stop, forever, equilibrium reached, done. I think in the more modern asset pricing world, you have strategies, so it's more of a game-theoretic model, where it can actually be optimal for me to try to deceive you by making you believe I'm selling something, so that you enter the market a particular way, so that I can take advantage of your actions. These kinds of game-theoretic models, whether liquidity squeezes, intraday trading, or fundamental hedging of various kinds, exist to play off against each other. These strategies can be quite complex, and they often play against the other person's expectations: what do I think you're expecting and wanting to do, what am I expecting, and then, if I know your risk parameters are very tight, I know that if the market falls by more than 5 per cent, you have to sell, you're a forced seller, so I'm going to try to push prices past 5 per cent so I can take advantage, and so on. There are all these kinds of games that happen, and I think in the world of AI that's going to be, quote unquote, "awesome," in the sense that I'm going to give agents five or ten strategies to play against you simultaneously, and we're going to discover exactly what your expectations, risk aversions, and risk parameters are for your trading, if any, and we're going to try to play you off in terms of the trading pattern. So the acceleration of these strategies and counter-strategies becomes a lot like cybersecurity, I'm guessing, in the sense that the only way you can play against agents is by having agents yourself, after a while, and so everybody has to have agents to be able to get liquidity pools and trade effectively, and not be picked off, and so on. Wouter Klijn 47:23 Yeah, it's an interesting future ahead with these agents. But Michael, thanks very much for coming to our office and having this discussion. Michael Kollo 47:34 Thank you for the invite.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. 147: From the Archives – AustralianSuper's Alistair Barker

    Nov 15, 202658 minAudio

    In this special "From the Archives" edition of the [i3] Podcast, 'Conversations with Institutional Investors', we're going back to September 2021, when I spoke with Alistair Barker, then Head of Total Portfolio...

    Credit the people on this episode

  2. 146: JANA's John Coombe 2026 – Active Management, AI and The Rise of TPA

    Nov 1, 202657 minAudio

    After posting an archived conversation with JANA Investment Advisers' Executive Director John Coombe, our listeners requested that we ask him back on the show to see if his views have changed since the 2019 interview....

    Credit the people on this episode

  3. 145: From the Archives – UniSuper's John Pearce

    Oct 11, 202644 minAudio

    Welcome to the i3 podcast. Today, we're revisiting a conversation from 2021 with John Pearce, Chief Investment Officer at UniSuper. From the humble beginnings at the back office of Westpac, John went on to become one of...

    Credit the people on this episode

  4. 143: From the Archives – Tanya Branwhite on TPA

    Sep 13, 202647 minAudio

    In this From the Archives episode, we revisit a conversation with Tanya Branwhite, former Head of Portfolio Construction at TCorp, which was originally released in March 2023. Tanya takes us through her unconventional...

    Credit the people on this episode

  5. 142: How ASX Companies Are Tackling AI Disruption – A Fidelity International Survey

    Aug 30, 202648 minAudio

    In episode 142, we take a look at AI disruption among listed Australian companies and their response to the risks arising from this new technology. Fidelity International surveyed 31 ASX-listed companies during the 2025...

    Credit the people on this episode