Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

C B m E & U

C B m E & U artwork

Accountability & AI with Simon Chesterman

Oct 1, 202438 minAudio
ShareXLinkedInFacebookWhatsAppEmail

Send us Fan Mail In Episode #5 of CBmE&U, Professor Simon Chesterman (David Marshall Professor and Vice Provost (Educational Innovation) at the National University of Singapore) chats with Sinead and James about the black-box problem with AI. Simon answers questions like, should we use beneficial AI even if we do not know how it made a decision? Who should be responsible if we do not know what went wrong? Should we put computers and pigs in prison if they do harmful things and should we say please and thank you to Siri? Remember people, be savvy!

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_01The center presents a new podcast at C BME and You, a podcast devoted to promoting awareness and debate around controversial ethical issues in biomedicine.

SPEAKER_03Welcome back to CB Me and You. From wherever you are. Today we talk about the opacity problem of AI and its implications on the use of AI in healthcare and in law. Joining us today is Professor Simon Chesterman, David Marshall Professor and Vice Provost at the National University of Singapore and Dean of NUS College. Welcome, Simon. Thanks very much. It's great to be here.

SPEAKER_00Thank you so much. We have a couple of questions to get to know you a little bit better.

SPEAKER_02Okay.

SPEAKER_00First question: coffee or copy.

SPEAKER_02So it depends where I am, maybe a copy sodae, or else uh skinny flat white, depending on yeah, what's available.

SPEAKER_03Hmm, excellent choice. Do they fit the same kind of taste profile for you?

SPEAKER_02A skinny flat white and a copy soda is my sort of sinful uh choice. It's it's a little bit sweeter, but not too sweet. Uh and uh yeah, the palm oil is not great for you, but a little bit every now and then doesn't hurt.

SPEAKER_00Excellent. Uh second question. What country do you want to travel to the most next?

SPEAKER_02So I would love to take my kids when they're old enough to look after themselves back to Tanzania, where I lived for three months uh back uh last century. Uh go on a safari, climb Kilimanjaro or something like that.

SPEAKER_03When you said last century, it sounded a really long time ago, and then I realized I was born in the last century as well. So I don't know. Okay, it was 1999.

SPEAKER_00Okay, um and final question: what is your go-to karaoke song?

SPEAKER_02Only time I sing karaoke is when the chairman of the Board of Trustees asks me to, and the last time that happened, which is some years ago, um, I asked my wife and was advised you can do a tufa, you can do karaoke, and you can do it in Chinese. So I did uh Yu Liang Taibia water sin, which is really impressive for a white guy singing Chinese, but it's actually an incredibly easy song. So highly recommend it.

SPEAKER_00It's a way to impress your colleagues.

SPEAKER_03Exactly. Yeah, I think you just got a lot of cred uh with whoever's listening out there.

SPEAKER_00Can you just translate that for me?

SPEAKER_02Oh, so Yuleang Taibia Wada says the the moon, the the moonlight represents my heart.

SPEAKER_00Oh wow.

SPEAKER_02So it's a it's a cheesy romantic song. But uh yeah, white guy singing Chinese nails it every time. You go to China, as a seriously, as a white guy, you go to China and say Niha, people say, wow, your Chinese is amazing. But you have Chinese Singaporeans who whose Chinese is much better than mine get no respect.

SPEAKER_00Ah.

SPEAKER_03Yeah, the next time I'm going to China, I'm gonna pretend to be like Hi, I'm Australian. Good eye. Good eye.

SPEAKER_00Oh, thank you.

SPEAKER_03Okay, so now it's time to get down to the real questions. So uh you've written a book on AI and the use of AI and the legal implications of it called We the Robots. And in it you listed three main challenges associated with AI uh speed, autonomy, and capacity. And we want to focus today on uh opacity. So my first question is, uh, what's the deal with this opacity thing and why is it a problem?

SPEAKER_02Yeah, so the book basically is looking at the governance and regulation of artificial intelligence. And I think one of the problems in this space is people think you've got to start afresh. You've got to come at it like Isaac Asimov come up with new laws of robotics, and that kind of misunderstands the problem as both too hard and too easy. Too hard in that, in fact, you don't need new rules. Um, AI shouldn't discriminate because no one should discriminate. AI shouldn't harm people because harm should be sort of managed, risk should be managed. Um, but it also misunderstands the problem as being too easy because applying the laws is where the real difficulty is, and that's where opacity comes in.

SPEAKER_02So opacity is the quality of being difficult or even impossible to understand. And this has been one of the sort of uh evolutions in machine learning over the last 12 or so years. Uh, as AI models became more sophisticated, more accurate, they also became harder to understand. Not just to experts, but even the experts cannot explain how some decisions are being taken. Uh, and even the large language models themselves cannot explain in a way that humans can understand what's going on.

SPEAKER_02So, in terms of a lawyer's uh view of this, this is a bit of a problem because if we can't understand what's going on, how do you allocate responsibility? And in particular, who do you blame when something goes wrong? So, for all these reasons, I think opacity is something that's worth thinking about, even if you don't need to understand everything all of the time.

SPEAKER_03Great. So just to push you a little bit on what you said, you started out by saying, well, we don't really need to reinvent the book when it comes to a lot of laws. You know, if if we know that we know that discrimination, for example, is not supposed to happen. We can just say, well, AI shouldn't discriminate. Um, how exactly is opacity a problem for this? Because it seems to me that we can know that something is discriminating without knowing how it works. We can just see we we, for example, we can tell that somebody's being racist without knowing exactly what's going on in their minds.

SPEAKER_02Well, can you? So that that's a really good question. And actually, one of the benefits of AI is if you interrogate an AI system and say, are you making decisions based on discriminatory reasons? Um, the AI at least will try to tell you the truth. No human's ever going to tell you the truth. No human's gonna say, yeah, I'm sexist, I'm racist, I'm so on and so on. Um, but opacity becomes a problem when in particularly complex uh decision-making processes, you actually can't understand why a particular decision is being made.

SPEAKER_02Uh and so, yes, you're right. If you aggregate all those decisions and you can see, okay, well, particular decisions are favoring a particular class of person, then you might conclude that it's being discriminatory. Uh, but actually, when it comes down to the understanding of a system and being able to explain behavior to attribute responsibility to different actors, then actually understanding how the decision was made becomes more important. It's a bit like if you think about an employment company, and you discover that the company is only hiring white men, for example.

SPEAKER_02Yes, it looks discriminatory, but how do you attribute responsibility to an individual within that company? Because it's very hard to prove in an individual case how or why someone is making a particular decision. Uh, and so you can address it structurally, but if you want to attribute responsibility to an individual, it's much harder. When we're talking about AI systems, this becomes more of a problem when you have unexpected behavior. So, although in discrimination context you might be able to aggregate the information, if we're looking at large language models, for example, even Sam Altman will say he doesn't understand how ChatGPT does some of the things that it does.

SPEAKER_02And if it's a low-risk scenario where it's just answering your question about, I don't know, your favorite poetry, then no one cares. But if it's giving someone mental health advice or diagnostic advice, then if we don't understand how it's making decisions, we might not be able to advise people about the risk profile and how to manage those risks.

SPEAKER_00Now, I know you've said that you you don't work in healthcare, so that your understanding might be a little bit more limited. But I do see that maybe some of your knowledge might translate into healthcare. And so one of my questions that I'm personally interested in is the way that uh black box AI or opaque AI can do marvelous things in the healthcare space. It can be highly, it can be highly accurate. It isn't always, and we're obviously people are still working on this, but there are some cases of AI being 45 times more accurate than clinical experts.

SPEAKER_00So my question for you is if it is actually more accurate, and it is particularly in healthcare, providing all of these benefits by saving people's lives and picking up on diagnoses five years early, as I've you know seen in the breast cancer space, then is it so much of a problem that we don't know the reasons because it's doing so much good?

SPEAKER_02No, not necessarily. And indeed, um I've had some really interesting conversations with medical professionals, uh, including one where I was talking about opacity, and he said, Well, look, hang on, we've been prescribing aspirin for a century. We don't know how it works, we don't know why it works, but it works. I'm going to keep on prescribing it. There are all sorts of medical um procedures, all sorts of pharmaceuticals that work in clinical trials. We might not know how it functions at the molecular level, but we're sufficiently satisfied that on a statistical basis it's a there's an acceptable um level of positive outcomes and a manageable amount of negative outcomes.

SPEAKER_02Uh, I gather it's even more extreme in psychiatry and psychology, where there are processes or treatments like electroconvulsive therapy, which can be useful in some cases of depression. We've got no idea why it works, but it works. So for all these reasons, yeah, I don't think we need to understand everything all of the time. But in a medical context, if you have, for example, a doctor who's going to prescribe a treatment to a patient, he or she needs to be able to understand what's going on to a sufficient degree of detail that the patient can make an informed decision.

SPEAKER_02Otherwise, the whole sort of foundation of medical law, which is based on patient autonomy, doctor-patient relationship, all that breaks down if neither of them understand what's going on understands what's going on.

SPEAKER_00Is it also true that doctors are already using AI through clinical decision support systems already? And we we, as patients, we haven't been told about them.

SPEAKER_02So in many jurisdictions, there are sort of three things that are regulated. There are doctors and hospitals, you've got to get certification, there are pharmaceuticals and there are medical devices. And so if something falls into one of those categories, then it's regulated. But many diagnostic systems in many jurisdictions are treated basically the same as a doctor looking something up in a book. Or as I've experienced myself, talking to a doctor and the doctor turns around and Googles for information. Now that's fine if the doctor can then sort of put that knowledge in the context of his or her education and so on, um, and can give you a recommended path.

SPEAKER_02But if he or she is just putting it into a system and it spits out a suggested um approach, the problem is not that he or she doesn't understand it, but that he or she is nonetheless going to be held responsible. Because if the doctor prescribes a treatment, that's his or her professional judgment. But if it's a diagnostic system that the doctor doesn't understand, uh then outside of the medical device medical diagnostic system being treated as a device pharmaceutical or a doctoral hospital, then who's responsible? That that starts to become a problem.

SPEAKER_03I I understand your question now. When I first heard it, I thought you were probing for a conspiracy theory. They've been using AI this whole time.

SPEAKER_00No, I've actually uh I've spoken to an academic from the University of Melbourne about it. And when you know she informed me about it, I was shocked. I was like, Are you telling me my doctor's been using this the whole time? But then I was like, actually, there's a lot my doctor doesn't tell me, and I seem to be okay with it.

SPEAKER_03So it can just date everyone.

SPEAKER_02Yeah, I mean, people talk about AI being a black box, but a person's head is a bit of a black box. I mean, we don't necessarily know why people make particular decisions to a great degree of detail. Uh and again, back to discrimination. You can interrogate an AI system for its reasons, it will try to tell you the truth, but um there's some evidence that a lot of us make decisions and then sort of retroactively or retrospectively justify them. So uh yeah, explainability is an issue for machines, it's also an issue for people.

SPEAKER_03Yeah. Great. So you said um that knowing how things works and explainability is really important for uh autonomy and informed consent. Um But what somebody might say in response is look, uh as a patient, what I care about is whether something works. Right? So the only thing I care about for me to provide informed consent is that is to know the probabilities. You know, what's the likely what what are the chances of this procedure uh succeeding versus this alternative uh procedure procedure or uh what are the prospects of of those procedures compared to any other choice I might make?

SPEAKER_03So why do we need do we really need explainability rather than just knowledge of the potential outcomes?

SPEAKER_02Yeah, so it's a great question. And it depends on the context. So again, back to the example of sort of pharmaceuticals, if we can run randomized controlled trials, and even if we don't know at a molecular level what's going on, we're satisfied with the result. That's okay. Uh and so I think we probably want to make a distinction between machine learning systems that analyze huge amounts of information, billions of parameters, and then come up with recommendations by identifying patterns that we don't necessarily understand. Distinguish that from generative AI, which has hallucinations or confabulations and uh engages in this kind of creative activity that we don't understand.

SPEAKER_02So if you're essentially doing elaborate statistics and sort of managing the information, you can come up with a recommended treatment, maybe that's okay. But if you're dealing with a more creative type of AI and we don't understand how it's coming up with these creative ideas, again, the sort of hallucinations are a little bit different if it sort of makes a mistake about what rides there are in Disneyland from telling you that uh, well, you're suffering from some sort of um suicidal thoughts and I can help you with that uh in a negative way.

SPEAKER_02So yeah, it'll depend on the context, whether we care or not. And that's maybe worth taking a step back. So why do we worry about opacity? Because sometimes people think that opacity is a problem just because we should understand everything. And as we were saying earlier, you don't need to understand everything. I mean, I'll routinely get on an airplane, I don't really understand how it works, but I'm pretty confident someone does. So that doesn't worry me. Um but we worry about opacity if if there are either illegal or impermissible or inferior decisions. So illegal decisions are the kind of discriminatory context.

SPEAKER_02So if you can drill down and identify bias, then yeah, we should we should stop that. Impermissible decisions are kind of decisions where it's um departing from norms that we would want to adhere to or practices that we would want to um uh respect. And in the medical context, you have ethical principles, you have um professional standards and so on. But there's also the issue of inferior decisions. And this is this is a real problem if the reason for opacity is not just complexity, but um confidentiality or proprietary interest, intellectual property.

SPEAKER_02That sometimes the reason why decisions are opaque is not just because they're complex, but because companies don't want to share information. And companies sometimes withhold information because they have a proprietary interest in it, and sometimes because they really don't want external scrutiny. Uh and so in some of those situations, you probably do want to be able to dig in a little bit to identify whether decisions are being made on the right basis or not, but also to help improve those decisions. Because generally, um, if you have uh a degree of openness about how certain practices are going on, other people can study it, other people can try and improve it, and that will lead to better outcomes.

SPEAKER_03And currently in the law, are there legal ways to get a company to make a company share some of the proprietary information? Um, maybe with just the courts rather than the whole world, if something is important enough?

SPEAKER_02Yeah, so courts are used to dealing with this in a couple of ways. The first is yeah, you can compel a company to hand over information. If it needs to be kept sensitive, you can have closed court proceedings, there are all manner of ways of maintaining sort of trade secrets and so on. So that's perfectly normal. There's also a way of dealing with the complexity problem. If something's hard to understand, you can have expert witnesses and so on. There are also ways to deal with that. The problem of the current machine learning models is that it's a combination of sometimes there's trade secrets, sometimes there's expert knowledge, but some of these models literally are impossible to understand and they cannot be uh explained to a human in a way that we can sort of manage the information because there are billions of parameters going on simultaneously.

SPEAKER_02Uh and so that's a problem because it's not just that a company is withholding information, it's the company literally does not have the information itself. Um, again, sometimes we might not care. Sometimes we might just say, well, as long as the results are good, that's fine. Uh and while everything's operating normally, perhaps that's okay. The issue, and this is this is what lawyers, this is why lawyers are loved around the world. The lawyers get involved when things screw up, when someone gets killed or someone gets injured, uh, and there's an effort to attribute responsibility.

SPEAKER_02And so that's where we might want some measure of transparency. And so if transparency is the opposite of opacity, you can think of transparency in a couple of different ways. Uh, because again, sometimes people think transparency should mean, well, I should understand everything that's happening. But there would be a cost associated with that. And I think that's important to understand that transparency has costs. Uh, because if you reduced the number of parameters, you might make it easier to understand. But I think as Sinead was saying, the model would then be less accurate. Uh, and so that's a cost.

SPEAKER_02And so when we think about transparency as the opposite of opacity, I tend to think of it as, okay, well, what needs to be understand, when, by whom, and most importantly, why? So, why do we need to understand this? And unless there's a good reason, maybe we don't need to worry about it too much.

SPEAKER_03Um I'm gonna ask a question that's way out of my depth here, uh, because it's a legal question. And it's that I I know that in the law you have things like strict liability and liability without fault. Um which is strict liability. Yeah.

SPEAKER_00Showcasing your legal skills.

SPEAKER_03Yeah, strict liability slash liability.

SPEAKER_02Well, no, okay, no, no, fair enough. There is there is a concept of absolute liability where you don't even have to prove um injury, but but that's an extreme case. Yeah.

SPEAKER_03I thought you were talking about a vodka. Anyway, yes. Uh so given that we don't actually need to assess fault in every single situation, can we just say, look, if you're the person who who's using this AI, you've decided to accept the risks. If something goes wrong, you're accountable and you're responsible. Regardless of whether we can explain it or not.

SPEAKER_02No, we can do that, but that's that's actually the problem with the sort of creeping role that AI plays in in all of our lives. So if we start with the simple example, like let's say uh we use Google Maps or Apple Maps or whatever, uh, and you lose the ability to read a map, and so you're doing this all the time, and then something goes wrong, you can't then blame the map software for you crashing your car into a river, which has happened. Um, if you bring it to a doctor, if a doctor is routinely using this software, doesn't understand how it works, uh, the software makes an error, but he or she is held responsible for it, um, there's a question not just about the justice for the individual, uh, but whether that's the right attribution of responsibility uh in order to um to minimize the risks.

SPEAKER_02So, one way to think about this is that the origin of strict liability in many contexts is what's called product liability. So, around the world, most jurisdictions, like if your car blows up, um, you don't need to prove what went wrong. It's pretty clear that blowing up is not what a car is meant to do. So you can go after the production line and you can sue whoever sold you the car, they can go after matter of fact. But for the consumer, or at least the estate of the consumer, um, you can go after them. The reason that came up is at the start of the 20th century, um, the idea in the market was buyer beware.

SPEAKER_02So if you got injured by a product, you could go and sue, you would have to prove fault and so on. And so the assumption was we're all educated consumers, you can see whether the product is safe or not. But as manufacturing got more sophisticated through the 20th century, it became pretty clear that was inefficient because the individual consumer is not in a good position to make these uh assessments. And if everyone has to go and sue, it's a very inefficient way of addressing the problem. Much better to work out, okay, who can manage the risk better?

SPEAKER_02Well, the manufacturer can. So let's just put the onus on them. Uh, and so from around the mid-20th century, we had this idea of the manufacturer should make sure that something is safe before they go to market. And it's kind of amazing that now we have these products, these sort of AI systems that are going out to market. And the developers are saying, why should we have to determine that it's safe? We want to roll this out, let consumers go crazy, and if something goes wrong, well, we'll sort that out later. Uh, and that's a kind of amazing situation that we find ourselves in. But it's partly because there is this shocking situation that it's not clear whether a computer program is a product for the purposes of product liability.

SPEAKER_02I mean, if anyone's listening, this is old enough to have gone out and bought a disk to then install on their computer, that was a product. But if you download it, in most jurisdictions, a computer program is not a product. The EU has recently concluded that it is. Uh, but we've got this sort of responsibility, liability, not vacuum, but sort of gray area where it's not clear who's responsible. And one of the things that the developers are really worried about is the relative responsibility of developer and deployer. And you can imagine if you're Google, if you're OpenAI, you're developing this sort of multi-purpose software, you don't want to be held responsible for everything that might go wrong depending on how someone uses it.

SPEAKER_02But if you're a developer and you're sort of taking this large language model or some other AI software and you're sort of putting a skin on it and making it into products, you don't want to be held responsible if there's some underlying software problem that goes wrong. And then you've got the user who might then be deploying it, and he or she might not also want to be held responsible. So when things start to go wrong, um, that's when we probably need a little bit of clarity, a little bit of transparency as to who made mistakes. Um that's a problem in the context that we're discussing it, because if you do that sort of before the fact, you might limit the ability of the systems to do useful things.

SPEAKER_02But if you wait until after the fact, it might be too late or it might be unjust to attribute it to any one of those actors.

SPEAKER_03Could we spread out the responsibility, have a kind of insurance, AI insurance scheme? You've read chapter four of my book.

SPEAKER_02So that that indeed is a great suggestion. And there are all sorts of examples. So I'm from Victoria, Australia, where there is a mandatory insurance regime. So if you have a car crash, everyone is insured. And that's actually a very efficient way of managing the risk, partly because it means that you get compensated and there's very little sort of moral hazard because even though there's mandatory insurance, we pretty much assume people aren't intentionally going and getting to accidents. You could have that for an AI system.

SPEAKER_02And you could have some sort of mandatory insurance regime that the companies could pay for. In fact, what you've sometimes had is the opposite. You've had some companies arguing back a decade ago that actually you should have give AI companies immunity because otherwise the risks are so great that they wouldn't be able to get insurance. Now, we didn't need to give them immunity. We certainly don't need to limit, they haven't limited their ability to innovate. But bringing it forward today to today, one of the things that I think would be really helpful in many jurisdictions is to think about this through the lens of insurance.

SPEAKER_02Because there are going to be people who will get killed by autonomous vehicles or injured by medical devices. And if you have an insurance regime, at least you can manage that risk. And then you can use other tools to try and address the developers to make sure they're not inappropriately putting consumers at risk.

SPEAKER_00Now, in the context of healthcare, um, I I found that whole concept of insurance fascinating. I was, I think I'm the lay person listener in this, where I was like, how are we going to solve it? And then you were like, insurance. I was like, genius.

SPEAKER_03The free market because it strikes again.

SPEAKER_00Ah, damn it. Um, in healthcare, though, is I think it's my understanding that AI diagnostics and tools aren't required to go through clinical trials and are not required to go through, and I'm not sure about this, to go through our regulatory bodies like the FDA or an or in Australia, the Therapeutic Goods Administration. And I was wondering if there is a push to have, you know, clinical decision support systems and other AI tools be clinically trialed and be pushed through these regulatory bodies, because they could also be therefore accountable for deciding that practical risk between at what point do we justify this highly accurate AI, even if it might have some risks with it?

SPEAKER_00Because we all accept risks in medicine. We know medicine is not perfect, it's not certain. So, do you think that we should be pushing for the role of regulatory bodies in pre-approving these systems before they get rolled out?

SPEAKER_02Yes. But the problem is that so okay, the reason it's not happening at the moment is that in theory, um, doctors are responsible for the diagnoses that they they come up with, and whether they come up with it out of their head, out of a research book, out of searching the internet, or from a diagnostic AI system, the doctor, him or herself is responsible. So at the moment, there's not a lot of pressure to do that. There's also a regulatory challenge that you don't necessarily want to unnecessarily constrain innovation.

SPEAKER_02And that's that's uh an interesting dilemma in the context of regulation that's sometimes attributed to David Collingridge. So the Collingridge dilemma, this is the idea that um at an early stage of innovation, it would be easy to regulate. So you could regulate some of these devices, uh, but we don't necessarily know what the risks are, and there's a cost associated with constraining innovation. So you could regulate now, uh, and uh, but there'd be all sorts of unintended consequences. You might hold back innovation, you might also drive it elsewhere.

SPEAKER_02So if Singapore, for example, took a really hard line and said, okay, we're we're going to heavily regulate these things, well, all the innovation would go elsewhere. So you'd miss out. The longer you wait, the clearer the risks are, um, but the cost of regulation start to go way up. So this is this is most apparent with something like social media. So if you think about social media, 20 years ago, when Facebook was launched in 2004, the iPhones launched in 2007, you could have regulated these um activities, uh, but no one knew about teenager anxiety and depression, Cambridge Analytica, election interference, and so on.

SPEAKER_02Jump forward to 2024, um, the uh the some of the risks are clear, but it's much harder to regulate. In the AI space, I do think it's much more the kind of wariness of constraining innovation or driving it elsewhere that is holding back that. And it's not necessarily for bad reasons, as you said earlier, Sinead. I mean, some of these tools are incredibly useful. We don't want to sort of limit the ability for doctors to have access to it, for patients to get the benefits of it. Um, but how do we ensure that we do that in a way that manages the inevitable things that are going to go wrong?

SPEAKER_02Because as you said, in in medicine, things will go wrong. And so do we want to address that by just compensating patients? So you can do that through an insurance regime, or do you want to do it through punishing bad behavior? And probably you want a mix of both, but you want to limit the punishment of bad behavior to the worst behavior uh because you don't necessarily want to whack people who are in good faith trying to come up with innovations that will net net help people.

SPEAKER_00I find this really fascinating because I think in my brain there are two kinds of harms that come out of um poor health care. One is those accidents, those mistakes, the negligence that happens across the board. Um, and it doesn't really discriminate between patients and doctors. It just happens. And we know it happens and we have the compensatory system for when it happens. But then there are other systematic kinds of harms like bias, discrimination. And what if we could find a way to take those more systematic sorts of really wrongful harms and just regulate those because you know they're there are problems across the world.

SPEAKER_00Uh they don't discriminate between healthcare and other things, even, and find better ways to ensure and put protective measures in place for those sorts of harms.

SPEAKER_02Yeah, and that's one of the things that AI is actually better at than humans. I mean, humans make all sorts of mistakes. Now, there are some situations where we might not want to go down that path. And so this is a slightly different area, but one of the areas I've looked at is the use of AI in warfare. Now, um many people will say that you shouldn't delegate sort of lethal decisions in warfare because, well, it's really complex and distinguishing distinguishing between civilian and combatant, what military necessity means is complex, and so you shouldn't delegate it to a machine.

SPEAKER_02I think that's a pretty terrible argument because humans are very bad at making these decisions. Um the reason I want to go back to Tanzania with my kids is I interned at the War Crimes Tribunal for Rwanda there uh and sort of did it in a very small way, played a role in looking at sort of war crimes, and people commit war crimes because they're racist, sexist, they're tired, they're angry, whatever. Um, and so these are all things you can train AI not to do. But I still don't think you should delegate these decisions to machines because a human should have to grapple with the morality, and a human should be held responsible for that.

SPEAKER_02Now, bring it forward to healthcare, it's a very different situation. You still maybe want people to grapple with the morality, but if you project forward five, 10, 20 years, I can imagine in future it will be not it'll probably be professional misconduct not to use AI. Um if you say, well, I'm gonna look at this x-ray and I'm gonna make a decision based on the couple of hundred x-rays I've looked at, and I'm not gonna use the machine that's analyzed millions of these things, that would probably be professional misconduct, and maybe it even is today.

SPEAKER_02Um and so I think there may be situations where you actively want to encourage, and that addresses some of the systemic questions. That one of the benefits of using AI systems is you can um look at aggregated data, much more data than any individual can. A different way of looking at this though, because you said, look, around the world these these challenges exist, is that some of the conversations we might be having in countries like Singapore, Australia, or the US, we're worried about misuse of AI. Um but I think in many parts of the world there is a real worry about missed uses of AI, that the inability to take advantage of some of these decision-making systems means that there are harms associated with not using AI.

SPEAKER_02So you bring it back to opacity. Transparency may be a cost that you want to pay in some situations, but if that means that the decision-making system can't help you as much as it would if it were opaque, then maybe in some situations you want to sort of embrace the opacity and not worry too much about understanding exactly what's going on if you're satisfied with the outcome. But that really does depend on the type of decision that's being taken. And so if it's sort of a clinical trial and you're looking at large numbers of population and sort of what's going to make sense, that's one thing.

SPEAKER_02But to pick another extreme case that I'm maybe more familiar with, um, sentencing decisions, um, you can tell a patient, well, nine times out of ten, this treatment will lead to the best result. And that's maybe acceptable. But you can't tell a person who's being charged with a crime, well, the computer says nine times out of ten, you're guilty. So off you go to prison.

unknownYeah. Yeah.

SPEAKER_00This is fascinating. Um, and I think if you could summarize your idea and your confidence in our ability to ethically and lawfully use AI in healthcare, would you say you're fairly confident that we can do it?

SPEAKER_02I think we can do it, but basically what I've argued for is two things. I mean, there's a tendency when we think about AI to reinvent lots of wheels, to say, yeah, AI should be fair, accountable, transparent. And sure, I mean, people should be fair, accountable, transparent. Companies should be fair, accountable, transparent. But what do we actually need? Then basically it comes down to two things. I think. Firstly, uh, and this is not so relevant to the medical context, but you need human control, or what in the in the Singapore model AI governance framework is described as human centricity. So it should be designed to help and benefit humans uh and not to be uncontrollable, uncontainable.

SPEAKER_02That's one thing. But relevant to our situation, you need uh a sufficient amount of transparency so that every decision can be attributed back to a traditional legal person. So what I mean by that is that each decision, say in the clinical context, so each diagnosis or treatment, someone's going to stand behind that, some human or some institution or some country, for example, so that you avoid the kind of um the accountability vacuum that the machine made me do it, or don't blame me, the the AI system told me to do that.

SPEAKER_02Um argue that, well, maybe the AI itself should be responsible, uh, which we can talk about if you're curious, but I think that's a pretty terrible argument. Throw them in jail.

SPEAKER_00So there are there are put the computer in a cell.

SPEAKER_02There are there are learned articles talking about how you could dismantle robots, destroy robots. Uh, and indeed there's a rich history of things like so. I I for my research went back through the history of um trial of animals. And there's some pretty horrible examples of like there was a pig in France that killed a child, and so they put the pig on trial, put it in a suit, and then hanged the pig. Um, and it's it is kind of laughable today, but the reason they did it back then was partly to sort of assuage the sort of community unhappiness.

SPEAKER_02Uh, and there's some situations where we we engage in, we attribute criminal responsibility partly to deter people. So hopefully you're not going to murder me today, partly because you're deterred because of the possibility of a criminal sentence, partly because you've been socialized, not hopefully to think about such things. Um, but we also engage in criminal punishment to reassure the community that this is aberrant behavior. And so you can imagine, again, in the future, if you had sort of a robot that sort of went on a killing spree, there might be calls for that robot to be dismantled.

SPEAKER_02And that might be satisfying uh to some part of the community, but it wouldn't actually address the problem. So what I mean by transparency is you need to be able to go after the person who built or deployed or programmed that robot uh and hold them accountable. So that's the kind of situation that you want to avoid where the robot can go on a killing spree and every and everyone can say, Well, it wasn't my fault, it wasn't his fault, wasn't her fault, uh, and wasn't this company's fault. So that's the kind of transparency you need. Final question then.

SPEAKER_03Have you seen the Avengers endgame? Yes. So let's say you so let's say you had all the the the stones of power, you can snap your fingers and and instantly implement one significant change. Now, obviously the right answer is, you know, end suffering forever. But let's say you can only make one significant change in this area. Uh, what's your snap going to be about? Okay, wishing for the power to make more changes.

SPEAKER_00Someone else did this and we were like, we'll change the rules for next time. We did not change the rules.

SPEAKER_03The problem was dealing with so many lawyers.

SPEAKER_00It was Barry as well. Yeah, yeah. Another AI lawyer.

SPEAKER_02So, okay, so I'll I'll give you a different um reference, which is uh which is Dune. Uh so in Dune you had the the idea of the Butlerian jihad. If you if you watch, if you read Dune or you watch Dune, um it's quite striking. You have this far future, like 10,000 or something BCE, uh C E rather, and there are no AI systems. And the reason is that there's a kind of commandment that's been built in to avoid uh creating an artificial mind. Now I wouldn't go that far, um, but I would say, yeah, we want we want to maintain control.

SPEAKER_02Uh and so I mentioned in passing the idea that you shouldn't give responsibility to AI, you shouldn't attribute responsibility to AI, you shouldn't be recognized AI as legal persons. Um, occasionally people would suggest that you should. And there are kind of two and a half reasons for that. Um, and the first two are wrong. The half reason maybe is correct. The first reason is uh to blame when something goes wrong. And I think that's just a mistake that would just protect companies and uh and individuals. I mean, if you could just blame the car, the autonomous vehicle, that will just protect the company that made it. So that would be a mistake.

SPEAKER_02Sometimes people would argue you should have AI legal personality to reward when things go right. So AI creative uh behavior. But I don't think you need to incentivize and reward AI innovation in the way that you do humans, because AI, all you need is an electric an electricity supply and a programmer. The half reason is that if some of the speculation about AI superintelligence ever comes true, then one of the things that the law tells us about legal subjectivity and the recognition of people is that it's tended to expand over time.

SPEAKER_02Uh and so, in like centuries ago, you had slaves, women didn't have rights, people of color weren't uh recognized as full legal persons. And we've tended to expand that over time. And so one theory is if we are moving towards an era where AI really is better than us, the theory goes, if we treat AI well today, then maybe it'll treat us well in the future. And so, yeah, I mean, I don't at the moment think that's a realistic prospect. But for the moment, yeah, if I had one magic wish, it would be that uh we should keep control of AI systems and avoid an uncontrollable, uncontainable AI system.

SPEAKER_00But just in case, always say please and thank you to Siri and ChatGPT.

SPEAKER_02Exactly. And all to all the large language models listening in, um, I'm on your side.

SPEAKER_00Ask the old gal how she's doing once in a while. Thank you so much for coming on the podcast today. Um, we've really appreciated your insights and expertise on this topic. So my absolute pleasure. You're welcome back anytime.

SPEAKER_02And thank you for not asking me to sing.

SPEAKER_00And now.

SPEAKER_01Thanks for listening to our new podcast. Subscribe online and tune in every first of the month. In the meantime, listen to CBME and U and be stabby.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Can Nihilism Improve Your Mental Wellbeing?

    Nov 2, 202645 minAudio

    Send us Fan Mail We spend much of our lives searching for meaning. Some of us look for it at work. Some in our families and friends. Some look for it at an overpriced spiritual retreat in Bali where some dude named...

    Credit the people on this episode

  2. Insurance Discrimination: Is it Fair? with Harisan Nasir

    Sep 24, 202649 minAudio

    Send us Fan Mail In some insurance scheme, you might be required to pay more because you are more prone to disability. In another, you may have to pay more because you're a woman (and are statistically likely to live...

    Credit the people on this episode

  3. Genomic Newborn Screening: Where less really is sometimes more with Ainsley Newson

    Aug 25, 202655 minAudio

    Send us Fan Mail Did you know that genomic sequencing can now screen newborns for hundreds of rare conditions right at birth? That looking too closely at our DNA might actually cause more anxiety than answers? And is...

    Credit the people on this episode

  4. The Hidden Side of Birth: Injury, Trauma, and PTSD with Angela Ballantyne

    Jul 2, 202648 minAudio

    Send us Fan Mail Did you know that over 80% of people who give birth experience an injury? That anywhere between 30 and 50% of people who give birth find it traumatic? And that up to 15% of people experience PTSD from...

    Credit the people on this episode

  5. Brains in Jars with Insoo Hyun

    Jun 1, 202656 minAudio

    Send us Fan Mail Could a blob of cells in a petri dish one day think? Feel? Have rights? In this episode, Professor Insoo Hyun (Harvard Medical School, one of the world's leading experts on the ethics of stem cell and...

    Credit the people on this episode