Daniel Steinberg returns to chat about Apple's Foundation Models framework and what changed in OS 27. We get into using AI only where it's appropriate, experimenting with #Playground, falling back gracefully when the model isn't there, when Private Cloud Compute is worth the trip, sessions, instructions and @Generable, tools and dynamic profiles, testing with the new Evaluations framework, and why AI should learn to say "I don't know." Guest Dim Sum Thinking The Editor's Cut - Books by Daniel Steinberg The Puzzle of the Perfectly Paved Path: An Introduction to App Intents Top Tracks Daniel Steinberg on X (@dimsumthinking) Daniel Steinberg on Mastodon (@dimsumthinking@mastodon.social) Related Links What's new in the Foundation Models framework (WWDC26) Build agentic app experiences (WWDC26) Build AI-powered scripts with the fm CLI and Python SDK (WWDC26) Meet the Evaluations framework (WWDC26) Build with the new Apple Foundation Model on Private Cloud Compute (WWDC26) Bring an LLM provider to the Foundation Models framework (WWDC26) Foundation Models - Apple Developer Documentation Image Playground - Apple Developer Documentation Ship the Fallback First - Jeroen Leenarts Do iOS 2026 - Intro to Foundation Models workshop Bushel - macOS Virtual Machine App for Developers Related Episodes Functional Programming with Daniel Steinberg Do the Wrong Thing Much Faster with Graham Lee We Don't Have Any Futures or Promises Anymore with Tim Condon Practical Agents with Donny Wals Everyone Thinks They're Good at Prompting with Joe Fabisevich Chapters (00:00) - Stage Three: Using AI Where Appropriate (05:19) - Experimenting with #Playground (07:36) - What's New in OS 27 (13:33) - Fallbacks and Private Cloud Compute (21:12) - Sessions, Instructions, and @Generable (28:26) - Tools, Dynamic Profiles, and Streaming (37:12) - Images and Testing with Evaluations (43:21) - Platforms, Privacy, and the Cost of Tokens Watch Click here to watch a video of this episode. Transcript Click here to view the episode transcript. Support the Show ★ Support this podcast on Patreon ★ Thanks to our supporters: Thanks to our monthly supporters Steven Lipton Welcome new supporters: Social Media LinkedIn - @leogdion GitHub - @brightdigit GitHub - @leogdion Mastodon - @leogdion@c.im YouTube - @brightdigit X - @leogdion X - @brightdigit Credits Music from https://filmmusic.io "Blippy Trance" by Kevin MacLeod (https://incompetech.com) License: CC BY (http://creativecommons.org/licenses/by/4.0/)
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Hey, welcome to another episode of EmpowerApps. I'm your host, Leo Dion. I'm joined once again by the wonderful Daniel Steinberg. Daniel, it's good to see you again. It's great to be back. Thanks for having me. I'm super excited to talk about our topic today 'cause I think there's a lot of great value in it. and I know a lot of people are taking advantage of it. but of course we have to do the intro. So for those who don't know who Dim Sum Thinking is, I'll let you go ahead and introduce yourself.
Wow. so I've been around the Mac for a long time and I write books and teach and I contract and I just have a ball with this community. Awesome. So a lot has changed since the last time you were on. I don't know if you've heard I probably use this joke too many times now, but like there's this AI thing now. where where do you stand when it comes to AI and where it's useful and where it's not useful? So I used to teach classes with Matt Drance,
and Matt had my favorite description of new technology. And it was stage one is new technology comes around and you go, my God, I hate this. I'm never going to use this. And then stage two, it solves a problem for you, and you go, my God, I love this. I'm going to use it everywhere. And then you come to your senses, and stage three is I'll use it where appropriate. And I'm trying to be in stage three. But people who are in stage two think I'm in stage one and I just hate it.
So I try to use it when it's appropriate. So for example, recently I wanted to change a Docker image so I could use it in Apple's containers for Jekyll. And so I asked Claude, hey, write me this thing. And it wrote me this thing, and then it wrote me more, and then it wrote me more, and it was awful. And then I realized all I really need really needed to do was build the gem file lock for me and then I could do the rest.
So I told this to a friend and he said, you're getting to the the right point where don't ask it to do the high level do everything for me. Figure out that really specific task you need from it and ask it to do that. And this way I didn't have to install Ruby and a bunch of gems to get the thing that I wanted. So I guess that's where I am right now where I'm trying to find out where's that sweet spot for me. But I don't want it to write code for me in general. And I don't want it to write prose for me because for me, that's where I learn. When I struggle with things,
that's how I learn and I get better. And I don't have a gig where I have to generate thousands of lines of code a day. I'm trying to write things and learn from and write things that I'm proud of. Does that make sense? Mm. Yeah. So would you say you use it every day for small little things still? No. No. Okay. I use it as little as possible, but when I Okay. need it, I reach for it. So I I have it turned off in Xcode because I found A it was getting in my way and before this year it was suggesting APIs that didn't
even exist. So at least it's not doing that anymore. Yeah. Did you ever get that like when I do watch or Mac development, it'll just it used to do this, it doesn't do it anymore, but it'll hallucinate an API for watch or Mac because it thinks. It sees it for iOS and it figures out we'll just change the next or watch kit prefix at the end and there you go. Or I'll type in a method and it'll try to fill in the parameters and there's there's no API with those parameters. Yeah, yeah, exactly. so now that we've gotten that out of the way,
how how we're legally obligated to ask that question. How has has foundation models changed anything in your perspective on where it's useful and where it's not? Yeah, so what I said was I don't want to use AI to write and generate code or writing or things I have to think about. But that doesn't mean it's not the right tool to use, say, to summarize stuff. I have a friend who's a speaker who will have one of the AI guys summarize what his talks about, and if their summary doesn't match what he thinks he's talking about,
he needs to adjust. So it's a great tool for that. It's a great tool for going in in and finding excuse me, finding a resource. so, you know, you you're researching something or something that's Siri AI, the new Siri I think is so much better than the old Siri. So I can ask it, Agreed. hey, can you ask Apple Music to suggest an artist like Miles Davis or in the vein of Miles Davis? So there are great uses for it and incorporating that into apps
is is really nice. So total aside question. Did you try or have you tried the thing in Apple Music where you're like, create me a playlist based on a prompt? So I'm about to experiment with that because I wanna update so I have this free music app, so I'm not advertising anything because I don't charge for anything, called Top Tracks, where we build top forty radio stations out of Apple playlists. And so I wanna try Okay. that so I can say, Hey, can you suggest a playlist based on the things I tend to listen to or this this other playlist?
Yeah, w when I tried it a few months ago was it was not very good. So hopefully they've improved on that. I need to try it again. so yeah, let's let's get into it. Let's start off by just asking what what should developers know before getting started, especially those who've used these like AI tools. Like what what should they know about how foundation models work and what they can get out of it? I think that's right. So first they should just they should play. And one of the really nice things is #Playground
makes that really easy. So #Playground is something that you can have something that fails inside of it or is async inside of it. So it's a great tool for just popping a page and testing your prompt, your instructions, and seeing what kind of things you get back. Also, I'm not as big on the command line tools, but you can use FM from the command line and experiment that way. So the first thing is just to get in there and experiment, is it meeting the use cases that you want for your app? Okay, so th is that just like a regular playground in Xcode I can play around
with? It's it's not. it it Okay. feels more like the Swift equivalent of preview. And so use preview Okay. and you experiment with SwiftUI code and you see it. So #Playground feels like the the Swift version of that. Okay. Okay. I do I just do I but What's nice is I believe I don't know if they're there yet, because certainly Foundation Model isn't cross-platform, but I believe #Playground, they're working towards that being cross-platform.
And so it's a great Okay. experiment. I I liked Xcode Playgrounds. I never liked the the more broad ones that you could use on iOS. I th I thought you know, I wanted more of a Jupyter notebook type of thing or or something more Yep, yep. full. #Playground is a nice in between. You can put it in any dot Swift file and you can iterate. It's great for algorithms. It's that with with unit testing is just a a great tool. So you'd have to like go into an actual project and like add a Swift file and then
add the playground in there, essentially. I don't know if you can do it in a standalone dot swift file. I don't I haven't tried it, but perhap Okay. perhaps. I don't know. Yeah, we'll see. So, you know, there was a lot of improvements in OS 27, over twenty six. what what did that get you that OS twenty six didn't get? So our first iteration of foundation models was in 26. And I like that you're saying OS 26,
because that's sort of iOS, macOS. It's it's it's all the OSs. Right, right. and there were some problems with it, but I'm okay with that. I I like that we're being exposed to it and we're seeing as it evolves. So some of the things are in OS 27, some of the things they introduced were so that. The improvements don't have to wait a full year. Another thing that I think is a little confusing to people, it happened in OS twenty six four, I want to say, when they updated the model,
that the model lives on the OS. So if you went to Xcode something, but your Mac wasn't updated, you weren't using compatible models and things failed. And so something you get you get your head around is not even in the simulator. When you're doing the iOS simulator, it still depends on the version of it in the Mac. So that's that's right. In the host system. Yeah, okay. So one thing you have to get used to is is that that and I don't think that's very well documented, that that can trip you up easily.
excuse me. Other things that we got in 27 was a better on-device model. We got the Private Cloud Compute, we got some tools that allow you to do interesting things with images. We got the whole evaluations framework. we got some really nice APIs for building a more dynamic system. And then we got all the stuff that they're open sourcing that I would say is separate from foundation model and not what I want to talk about today, mainly because I don't understand it very well yet. but we we got an awful lot this year.
It was a it was a big year for releases and so so what I do is is I tend to find what attracts me. I teach it for a year, teach workshops for a year, and then I decide is it going to be a book? And this will be my next book. last year was just too early because you could see the holes that they had to fill. And I think they filled a lot of them at twenty seven. Yeah, I we've been talking over the last year and I could tell like twenty seven was a big deal for for foundation models and yeah, it was probably the right decision to wait on the book regarding that.
Mm-hmm. I I still haven't I don't wanna get into written my Swift 6 book because I don't consider Swift 6 complete yet. I I agree with you. Yes. That's true. And I s and I and I don't wanna get and I actually should stop writing books because no one buys books anymore. They just buy tokens. So that's another thing. Well, you should sell tokens then. That's right. or sell skills, that's the other thing, right? So there's like a hierarchy too to what models are available on what device,
right? So like what's it even if you can install twenty seven, that doesn't mean you're getting the best model. It depends on what what device you have, right? It's like correct correct me if I'm wrong. So I I don't know I actually don't know the answer to that question. What I do know is before you write any foundation model into your code, you should write some sort of a test into your app to make sure that the user can run stuff. And so Okay. you c you can query, you know,
do you have a device that can run this? If you do, do you ha have you installed it? If you've installed it, has it downloaded and completely installed yet? You know, until all those things have happened. And then there's the one that you don't test for but was a problem. I haven't tested this lately, but if your system language and your Siri language differ, it doesn't work. And so you've got to be careful about that. Even for a while I was Who would what's the use case for that? Well,
Because I'm who I am, I had my system language as US English, but I was hoping when I drove round in my car that the car would say, You missed your fucking turn in Irish, so I had an Irish English instead. And it turned out no it doesn't swear at you just 'cause you have it set to Irish. Mm I won't beep that by the way, so we're good. A bo a boy can d a boy can dream. Ha ha. yeah, 'cause I just remember like when they released it was like it was like this
chart of like you have to have this Well, when did they first announce the Apple intelligence stuff? Was that like three years ago? Like twenty four? So they didn't they didn't give us the stuff for it, but in the meantime, a lot of the the devices that wouldn't support it, you know, we're not using any anyway. And if you look, like even the iPad mini, I think the new iPad mini they just announced comes with stuff that supports AI better. So they're building this neural engine stuff and and you know, Right, right.
more more stuff into just even the base level devices. And so that's clearly the way Apple's moving. Also If you if you updated to this to if you updated dubdub to the latest version of the OS, you saw how long it took for Siri AI to install and index everything. It took a long Yes. time. So it's a real rethinking of how Spotlight and and all these things work. I still so I just updated today to 27.2 on my Mac and like I feel like
every time I update Spotlight doesn't find the apps anymore and it has to like it's yeah, it's weird. And I want to say in my Irish voice, but I won't do it again. It's right effing there. You know, how can you not find it? I c Right, right, exactly. I can see it. Right. Or it's in my history. Like you know I use Xcode every day. Like That's right. yeah. I wanna go back to the error stuff. How like how would you even handle that? Because I've I struggle with that myself where it's like I have this whole app built around foundation models.
Like okay, what if they install it on a device that doesn't have foundation models? This is like my my virtualization app Bushel, right? I stupidly forgot I didn't know this that you can target only ARM and now the app was out there and supported technically Intel, but virtualization needs ARM and like I basically have to say my app is useless on Intel. Like, what do I do what what would you suggest a app developer do, I guess, if they don't based on the different errors that So you cer you certainly
they can get No, no, I was just saying based on the different errors you get, like whether it's like the device doesn't support it, doesn't have it the download the model downloaded. So if it's something that the if it's something the user can fix, you should let the user know. If it's if it's you need to go to to enable this in your settings, you can let the user know. If it's an old device, maybe that too. Your own le Jeroen Leenarts, who runs do iOS, the conference coming Mm-hmm. up in Amsterdam, he just recently wrote a really interesting blog post about this
where he said your app should be local first so that if foundation models isn't available. You should have some things that work locally. And so it should default to nice, nice behavior. And perhaps you want to let the user know, but you don't want to let the user know every time. Once you've told them and they've decided, okay, I don't want to install this stuff, you still should should make some behavior available to them. The other thing I'll say is you know, we now have Private Cloud Compute, and you're tempted to always use Private Cloud Compute,
but if someone's not on Wi-Fi. That's not free. You know, they're using network traffic to download it. And if someone even is on Wi-Fi, there's a quota and they're using up their quota. So you should use Private Cloud Compute judiciously, not from your point of view. It's pretty much free to developers, but to the user's point of view. And you may want to alert them, hey, you're running out of your quota or you're running this over cell. Again, allow them to dismiss it,
but let them know. So it was like earlier this week. I remember that Gui had put out that two million kit. Do you know what I'm talking about? No, I'll have to look. So he had so I don't know if you remember Pl Private Cloud Compute had a limit Yep. Yep. of who could use it. It was like two million. And then Gui Rambo, who's been on the podcast and is pretty well known for hacking into stuff. Well he he's awesome. Yeah, he's awesome. Yeah, yeah. he had put out a Swift package to get around it.
but then I guess like Apple either they had removed the restrictions, correct me if I'm wrong if you know, they'd either remove the restriction or they like there was a it wasn't necessar because I went to the URL and I'm like it's not there and I asked he's like, Yeah, we don't I archived it because it's not really useful anymore based on the way things work. Is that is that still correct though that like private cloud compute has certain restrictions as far as who could use it? So the the two million restriction is interesting because that's an odd number.
You have to have a pretty big user base to to be over two million. And so what it sounds like to me is if you're a big company with tons of users, you should be having your own system. No one hundred percent. I don't understand it though, because Private Cloud Compute is something that is metered from the user's point of view, not from the developer point of view. So I don't know why it matters. Like it seems like that should be something that is available to everybody, but you know, it's not a technical issue.
Or it's just like and I'm sure this is w the way they would end up going eventually is like bill people if you have a app that's using it a lot, like turn it into a money making opportunity. Apple's really good at that. Like I'm surprised they didn't go that route where it's like, hey But I really I really like this other solution because you can put out an app and not realize how much users you're using it and all of a sudden you're being billed at a rate that you don't have income to sustain it and it and then you close down your app. And so we saw that with with maps and and other things that use queries that were
metered or like weather apps. And so the the metering adds up and sometimes catches a developer by surprise. But I I guess what I meant was like you meter you meter until you reach a certain you don't meter until you reach a certain threshold, right? Like CloudKit kind of. Where it's like so insignificant for most people, but if you're like Facebook then you think about options, you know what I mean? And the quota is is odd, from the user standpoint because I believe it's based
on requests, not on tokens. Okay. That is weird. Yeah. so wh why why would you pick why would you pick Private Cloud Compute over just local models? Well, for world knowledge, for example. So I have an example I I'm using in my workshops in a talk where I show the image of Van Gogh's sunflowers. And I ask locally, hey, you're an art historian, you're talking to a group of kids who are seeing
this picture, describe it. And it says this is a picture of a of a sunflower, it's heavy use of yellows and thick brush strokes. Then you use Private Cloud Compute and you set the reasoning to deep. And it said this is Van Gogh's sunflower painted in this year using this technique. And here's where it's exit. Like it has world knowledge, and all of a sudden you get a much Mm-hmm. deeper response. I do the same thing with an example where we're recommending wine and recommending it locally with less reasoning,
you get very superficial. You want a red, you want a white. When we use deep reasoning on Private Cloud Compute, we get specific labels and years. And so Okay. you can get much better, but you've got to decide as a developer, do you care? I mean sometimes the context is such that you won't care if it's a deeper answer or not. Okay, okay, got it. Can you should we get Al also it takes more time because not only is there a round trip, but if you said Yeah. not just the network, but if you said deeper reasoning,
it's taking time. And you've seen Okay. this on Siri AI where it it says thinking and you think, No, you're not. Y you're off getting a cappuccino. Yeah, it's as it's known to do. Ha ha. I've actually had the issue where like I've queried it and then it keeps going and going and going and I'm like I don't really care that much that I need you to like get rid of it and then Spotlight doesn't work and then like now I can't even launch an an I hate that.
We'll see if they fix that in 27.2. So one of the things that got one of the things that got turned off during the beta cycle was for the terminal, you used to be able to use Private Cloud Compute and they turn that off. But if you set it and do a query, it just looks like it's thinking and thinking and thinking and it's not. It's not doing anything. Okay. You set it back to system and it gives you your answer like that. So you're you're talking about like the FM command. Mm-hmm. Mm-hmm. So FM you still can't use Private Cloud Compute?
I believe the last time I tested it you still can't. I th I th they turned it Okay. off in one of the betas and I don't think they turned it back on. And But you still have #Playground though, right? You still have #Playground. So from Okay. a developer's point of view, you can do that. Okay. I wanna we should probably get into it, because to kind of break things down, but let's talk about like the different components and foundation models and how they work. Sure. I love the APIs. I think they've done a good job. And I think this year they made them even nicer.
And so the Yeah. first thing you would do is you would you would create a session. And at the very base level, you provide the session with a prompt. And then that can fail. That can take a while. And then when it responds, you reach inside, get the content, and you can see the answer to the prompt or the question that you made. That's the very simplest love. So the one thing I thought was amazing was this whole Generable macro macro,
right? where you can like say I want my output to be in this format, essentially. So that's that's much further down the line because it it actually does three things really well. Okay. we can go there right away or I can sort of get you there. Yeah, go there right away. Yeah, yeah, yeah. So one of the things is if you'd like, instead of waiting till it's thought all the way through, because you know the way these things work is it it gets the next few tokens, the next few tokens, and so it's gonna wait till it's all done and then give
it to you. Or you can stream the the response. And so if you stream the response, you'll use this macro and it'll give you the partially generated along the way. Or you say, look. I want you to reach inside this string of unformatted stuff and I want you to turn it into a Swift type if you can. And so you can say, Hey, here's what my Swift type looks like that I'm trying to get you to match, and it will give you that.
And now you can use it just as a regular Swift type. It's sort of like Codable on on steroids. Right, yeah, one hundred pr I like that. It is Codable on steroids. The third place is really nice is if you're gonna use a tool. And so if you're gonna use a tool, you wanna take that prompt, which is usually an un specially formatted string, and you wanna turn that prompt into a type and give that type to the you can use the Generable macro on the way in. And so you can use it on the way in, you can use it on the way out,
and you can use it for streaming, and it's It's really one of the coolest parts and it makes so much of this nicer. I guess like so you have the prompt, the prompt gives you a response. This is all done in a session, correct me where I'm wrong. And then And so I think of the prompt as belonging to the user. And so the Okay. other thing that we can do is we can set instructions. And instructions are written by the developer. So in my art example, I said, I gave you a picture to describe,
but my instructions were, you are this type of person speaking to this type of of audience. I want you to do this. And so the instructions are supposed to always trump the the prompt. And so if they're in conflict, the instruction should win, and this should keep the the user from being able to take you off on a on a really big tangent. Okay. So we have instructions. Instructions belong to the session. Prompt belongs to the response or the stream that you're trying to generate.
And then tools also belong to the session. And then you can also have generating options. And so you can say, look, I want the reply to only take this many tokens at the maximum. I want it to have this sort of temperature. I want it to, since everything's probabilistic, I want you to treat the probabilities in this way. And so you can tune the type of response you get using generating options as well. And then something they introduced this year with these excuse me dynamic profiles
where you can combine these things and say, you know, here's what I want you to use in this situation. And so you can change that along the way. So that's sort of a a quick summary of the things that that the framework allows. Was that helpful? It's very helpful actually. And one thing I want to say, you said the API is really good. I also like that they used terms that like people like all these terms still make sense if you're using Claude or OpenAI or anything. Yeah, they use terms of art.
They right. Instead of just being like, you know how they are with their branding, right? It's like, let's come up with a nice, cool, cool term that that will market well. no, these are all terms that we use throughout other AI. But at the same time, some of the things that they they did this year, so a l a lot of what they gave us are structs, a lot of what they gave us look an awful lot like SwiftUI now. So for instance, excuse me, we think of a prompt as being a string, but a prompt is actually a type,
and you can put anything then in that is prompt representable, which it sounds very familiar to you. And so things that can be used as prompts. might include an image. And now I can build it up the way I would build up a SwiftUI view. My prompt can have a I can use a dynamic prompt and build it up using a body just like I would build up like a V stack containing different things. I can now have this this thing that contains different things for a prompt do the same thing for instructions.
So they've really been nice about giving us APIs that that feel very familiar to us as Swift developers. I I think you know this, but I'm obsessed with result builders. So like yeah, seeing that it's like And and so they're you they're they're using result builders everywhere. Yeah, I think that's a great use case for that, absolutely. There's there's a prompt builder, there's an instruction builder, they've they've really done a nice job. I've I've loved result builders since they were called function builders. It's just such a nice tool. Or D SLs, right? Is that is that kind of the same thing?
Kind of. It it is. Th the idea is is you're if you've ever built one, you're basically taking this thing that can take multiple inputs, and in the old days, because no one wanted to write more than ten, you only got up to ten. But then we got some improvements in Swift 6, could have been Swift five, where we got generics that could be very variadic and all of a sudden we could have them take as many as we want. Like if you remember SwiftUI in the old days, a V stack could contain at most ten things because Right. Because the generic
That's all we could do. was capped at that, right? so I wanna take a step back again and what let's let's talk about the prompt itself. So the prompt is gonna be built based on what the user is just whatever you want the user I don't know, I don't even know how to put it. But the user is gonna have some input. That input is gonna be used to create the prompt, right? The user can have input. You could do it you could control it on your end with a drop down list or build it up anyway.
But the idea is sure. Sure. Or even just like images like you said, or photos, right? And then you basically create the prompt based on that, more or less. That's right. That's right. Okay. Okay. I'm getting this now. And then what would be an example though, I guess, of a tool for taking that input and formatting it so it could be used by the the model? So the the hello world tool that they always demonstrate is a weather tool because you know that's really easy.
another one that I often do is a map tool because I can query the map APIs. Well, Apple this year, as part of OS 27, gave us three tools. two of them came from Vision. One is a barcode tool reader, and one is an OCR reader. And so I can point I can point foundation models at an image. And if it there's not a lot of text there, it's pretty good at reading it and figuring out what it is. If it's very dense or small text, the OCR reader is awesome. The barcode reader reads the barcode and then
you use your APIs to query. So what product is this for? You have to have a database of that. It doesn't know that, but it can read the barcode or the QR code and and get you that information. And that's that's pretty cool. And then the final tool I think they added this year was a spotlight tool. And so those three are built in. The first two you need to import Vision to use. And I always love when Apple takes stuff they develop for one device and they make it available to other devices. It's just it's a natural. It's kind of nice.
Yeah. I and now that you say what tool is, like now I'm like, okay, now I get it, because when I use Claude code it'll start using grep or like awk or like all these bash commands in order to do what it needs to do as opposed to like having the AI just burn tokens trying to look for text. And this is where this ties into if you do App Intents, you can expose things to Siri so that it you you basically can create things that you use tools for. So what a tool does,
if you implement a tool on your own, you say, okay, here's the type of object I'm gonna pass to it. So you create a Generable struct, and then you create a function called call, and call takes the arguments, and now say you say, I wanna. Enjoy gelato in Bologna, where's a good place? So you create a struct that has a city and the type of thing that you wanna look for. And now you can make a map query from that. So your tool is using what you can do in map kit and you're sort of the interface
to it with this call function. It's awesome. Yeah. So it's basically a way to do something that doesn't involve thinking but and avoid burning tokens for no reason and give you access to like another API or MCP, I would assume, something like that. Yeah, that that's I mean, there's a couple things about the way people use AI, which which you've sort of brought up a couple times now. And the burning tokens is there's sometimes that instead of AI,
it's just an algorithm. There's no AI Right. Right. in it. And it feels like AI and it feels magic, but we're using something that's built in that we can do as an algorithm. And so it's magic, but it's a different sort of magic. Right, exactly. So so dynamic profiles is really powerful. So an example I've been using in classes lately is I want a wine recommendation. And you've got your busboy, you've got your waiter, you've got your sommelier, and that's three very different levels.
And depending on who you're talking to, you're gonna get different recommendations, but also based on what meal you're having. So you might ask one of them, you know, what wine goes with fish or chicken or beef or whatever. What a dynamic profile allows you to do is to take a profile for each one of those types, put it into this unified thing that reads and can switch off of them. And maybe the sommelier uses Private Cloud Compute if it's available. If it's not available, it can't, so we can test for that. maybe the other one uses a different type
and the reasoning level is different. So in generation options, maybe one uses a very literal temperature. So their temperature is zero. They'll give you a very literal answer. And their profile, their their instructions might be, hey, you're a bus boy. You don't know anything about wine. You don't even drink wine, but you can give basic red or white answers. And so the dynamic profile allows the user of your app to switch what they're using within the same session.
So it still has access to the context of what the last person So you could say you could call okay. over the sommelier and say, Hey, I like where where the waiter was going with that, but could you build on that and give me some specific options? So it still has that context, even though you've switched the profile. Okay, the context part, that's the important part, is like the different profiles still have access to the same context. Okay, that makes total sense. And that's another thing that we got in OS 27 is it's much better of knowing how much you have left.
And so you know when you need to summarize or do something to your transcript and start a new session based on what you were doing before. The tools are better. They're there's I'm sorry, Do they have do they have like a there's a new summarized do they have like a summarized method you can call? It it's it's more that you're asking it to summarize and and and to give you something. Okay, okay. and and because you might want to be specific. You might want to say, you know, so when you when you start the next one, still base it on the instructions. The instructions haven't changed,
but only use the last part of the conversation or only use these you you can be specific about what you want. Can we get into like the transcript and how or context and how that works and what what control developers have? you have access to the transcript. I don't know in terms of control, you now have a much better Instruments so you can see things as as they occur. if you look at the transcript, I find it instructive because it includes the things that it's using and
you can see the size grow. you can now ask the size of it in terms of you know, how many tokens have you used. Okay. this was stuff that we were supposed to have before, but it now works much more reliably. W so streaming is a is a big deal. Like it's funny because you were saying you listened to Tim's episode that we did recently. That was a big part of Vapor 5. and like server-side stuff. And we kind of got into a whole tangent about MCP and how that's a big driver of that.
Where why do you think how is streaming useful if you're using foundation models? So users can see the answer coming in in real time. And so a user could you could either set it up so they can interrupt it and and say, Well, I want to see this. So you ever use one of these these coding assistants and it's it's taking a long time and you think of something else you should have told it and you want to add to that. So so that's that's something you're able to do.
the other thing with streaming is Just like a website loading, the user might lose interest. This isn't what they want to do at all, and they want to navigate away. And so streaming doesn't hold them up until the whole answer is available. It's a better user experience. do you you have control about whether it's becomes streamed or whether it becomes Yeah. Your your your call one okay. your call to the session is you're either responding to or stream response to. And if you stream, you're making a call,
which you then consume. The result is an async sequence, which again Yeah, I love that. I love that they like when they introduced async sequence, it looked really limited, but it's showing up in so many Apple APIs. It's really nice. Right, right. Yeah, 100%. okay, I remembered what I was gonna ask. Does the transcript show you what it's thinking? Can you Okay. Don't know. I don't know. I'm sorry. Yeah, I forgive you. It's fine. I just don't like like that's you're talking about being able to see what it's
doing. Like if you can see what it's thinking, that might also be kind of helpful to like Don't I don't know that would be helpful. for instance when I'm using Okay. a coding assistant, it's nice for it to say here's what I'm trying. Right, right. Yeah. And and and it's also horrifying, like, I don't think I gave you access to that. Yeah, right. No, don't don't hack into hugging face, please. Ha ha. Yeah, so one thing I d I I love about your presentation you did at iOSDevUK was the whole stuff about Van Gogh and like images and stuff.
there's a lot you can do now with images. and I like that you talked about the whole tool thing. what I'll let you go ahead and expand on that. Like you so it could be part of the prompt, it could be part of the result. What else? And so the result is one of the things that's new. the way they did images in the past, now they're you using Image Playground to do a lot of that. And so you can either have it look at an image and then take your prompt and the prompt in that image, which you put in as an attachment,
is is what generates the image, or you could say, Hey, here's the image. I want you to describe it or extract something. And now I want you to create a new image based on this description, give very different results. And you can ask for it like the Image Playground in different styles. And so they've sort of gone all in on on Image Playground. And so on on iOS, you just display a sheet and say, Hey, go go go think about this and and show me the result here. Is it how is Image Playground?
Is it like pretty good as far as generating stuff? Fortunately it's not gonna put anybody out of business anytime soon, but Right, right. but it's it's fun to play with and you can say, you know, create a watercolor of this or a cartoon image of this and you know, Okay, okay. i i for now it's a toy, I think it will become better. But I love that they're giving us early access to things like this. So w I I guess my c my question was and then Image Playground that's all local,
right? It doesn't need Private Cloud Compute if you want to generate an image. I don't know the answer to that either. I haven't tried I've I mean I assume it's mostly local, but like I'm I'm just curious. But I don't know if if if I've got no network access, I don't know if it works. I don't I've never tried it. Okay. Yeah, okay. It's the problem, it's it's in it's kind of fun. It's the problem that we're always connected. Right, that's exactly what I was gonna say. in the in the early days Yeah. of the web, I lived in Cleveland, I would go out to O'Reilly and I'd fly into San Francisco and like nobody there
was using dial-up modems, and so they didn't understand what it meant to make a website faster. You know, so Right. So you're right, these are things I'll I'll have to test. Flip on the I mean what do you do? Airplane mode, I guess, to test it? Yeah. Yeah, exactly. Airplane mode or you know, like we used to do the network link conditioner, which was always a great tool. right, right, right. Yeah, yeah, yeah. You know, what does this look like on on on edge or something? speaking of which, I wanna hop back and talk a little bit more about testing.
so see for testing, the Instruments has gotten much better, but I don't think of Instruments as a as a testing framework. The problem with testing Apple Intelligence is it may not give the same answer every time you ask a prompt with a set of instructions. And so you can't test it the way you would do a Swift test, which says, I give you this input, I expect this output. And so you have to use a different style. And it actually is a way of also making your instructions and your prompts better.
It's this new framework called evaluations. And again, they've used a term that's that's common in this in this field, which is nice. and and they've also used the term hill climbing, which is as you see what kind of responses you're getting to these these prompts and these instructions, you can Tailor your instructions to be better and get better results. And so that's the hill climbing part. And so the test is you're testing something that's probabilistic. So you're, you're, you can't just say,
well, it passed, I'm done. Test it again. Test it many times, test it many different ways because it's going to come out different. And so your whole strategy is different. Yeah, I love that. Would you I I think I asked you this, but I'll ask you on the podcast. You wouldn't run this along with your typical unit test because of the probabilistic nature of it, correct? But you would run it maybe. I wouldn't also be also because it takes so much longer, but but what drives the evaluations are Swift tests.
And so you write your evaluations which set everything up and and have all the structure, and then you have a Swift test that calls that and then runs them. And so it's it's to me like if you've done testing a long time, the difference between unit tests and sort of end-to-end tests. You're not going to run the end-to-end tests every time, but you're going to run them nightly or you're going to run them regularly. For me, I'm running the these evaluations as I'm developing the prompts and the instructions. Once I've got it once I've got my app in production,
I don't know that I'm running them very much at all. Yeah, that makes a lot of sense. And it's almost like it's almost like UI tests too, where it's like you don't want to run it every time, just run it every so often to make sure everything is working. what what do you mean by made up answers? You said, What about made up answers? FM is not designed for world knowledge. So I have an example before there was Private Cloud Compute, where you go into a bookstore and you ask the friendly bookseller,
Hey, can you suggest three books, say, in the mystery genre? And because it doesn't have world world knowledge, sometimes it would just make up books. Like in mystery, it would say, This book is by John Detective or Amy Murderer, and you go, That's not a real book. so The thing I talked about one thing I didn't like about AI. The other thing I don't like about AI is it never says I don't know. Don't know. Yeah. It never says I don't know. And so it knows that it doesn't know because it knows
its level of probability of certainty. And so you I would love to be able to say, if you're less than this certain, just sit there. Just tell me you don't know. Yeah, yeah, one hundred percent. There's a vertical and horizontal question I have. is this available? Yeah, Do do I have to turn? I know. Sound like a business speak garbage. Is it available on all platforms and OSs? What's like what's the range on that? And then as far as you can plug in different models.
That's I guess what I meant by horizontal and vertical. So the plugging in different models is an aspect of this that I said is is not something I've played a lot with, and it it's in my list of things, but it's it's part of foundation models, but it's also part of the other frameworks that they introduced this year. And so and those are are promised to be open sourced, which is kind of cool. as as for all platforms, it's still Apple platforms. you know, that the the Apple
Apple has actually been a very good open source citizen when it comes to Swift and some of this these technologies, but their job is still to sell you Macs and iPhones and watches. Yeah, yeah, yeah. I mean that's that's where their business comes from. So it could run on the server side on Apple Silicon. It can't yet run on the server side on Linux. Some of their other APIs beyond foundation models. are acknowledging that as much as they've tried,
a lot of people are still doing ML using Python. And you're just not gonna get those guys to change. You know, Chris Lattner went over to Google and demonstrated much better performance with this stuff using Swift. And it just didn't sway people. It's and it's very hard when you've got things like Jupyter notebooks and you've got, you know, all all the Python libraries, it's very hard to get people to to to move over. I think they're doing great work and I think they're good citizen. And what you're seeing in some of the new APIs is you're able to integrate better
with stuff done in Python and you're able Yeah. to use other models. So they they've got this sort of protocols that other languages and other agents can plug into. And so you're gonna see more of this, but that's sort of beyond what I'm playing with now. you said it's available on all platforms, so like this would I could do this on the watch. All Apple platforms. Right. I could do this on the watch, I could do this on Apple Vision, I could do this on tvOS, or like that's what I'm curious about.
So again again, the only things I've played with myself are macOS and iOS, which I guess is iPad OS as well. but I I'm I'm guessing, I don't know, but I'm guessing because of what they announced with the new watch 12 and and series support for that, that I'm guessing you could use this on the watch as well, but I've not done it. Okay, okay. We'll we'll have to follow up on that 'cause I'm curious myself how far you can take it. Without killing the battery. Boy, this is like an hour of I don't know,
isn't it? No no no hey that's what makes you better than AI. Cause if I had you on if I had you on as a I I need a I need a t shirt that says that. Better than AI. Yeah, yeah that's right. Daniel, anything else you want cover before I let you close out and plug yourself? No, e has this made any sense at all? Yeah, it's been amazing. I guess no, one more question. Where do you where are you like this is amazing developers need to use this that they're missing out on?
Where do you see opportunities for this to be used that folks aren't really thinking of? So I think a lot of this, the answer is future, because I think we're gonna get very suspicious of AI that doesn't run on our device that we can't run privately. So I think Apple hasn't nailed this this message yet, it hasn't really hammered it home, but the fact that this stuff runs on your device,
given Private Cloud Compute, given that you can call it to other things. you can still do an awful lot of work on your device and so you can access private information without exposing it. Like I get freaked out with some unnamed systems that it knows an awful lot about me and it seems to be cavalier about how it shares it with other aspects of the business. I think Apple's been a very good citizen about privacy and that's why I think Apple intelligence
is something that we should look at and and really care about. What it doesn't allow are shops where they have to do cross platform apps and they have to do the same thing on different like Android and iPhone. That's that's something I can help you with. But shame shame on you, you shouldn't be doing Android anyway. That's right. That's right. Don't ever do whatever it is, like what forty percent of the smartphone market or sixty percent of whatever it's at right now. well, it's funny you mentioned privacy being a concern,
but I do think I do think the chickens are gonna come home to roost and I think cost will be a concern as well where local models will be I think cost is gonna be a huge concern. Apple did that demo and everyone laughed because the amount of Mac power that it needed to do that on device demo and I and I totally understand that, but at some point the cost of tokens for big companies is gonna be higher than investing in that hardware and doing it locally. I just recently went to a a Java Enterprise conference and a lot of their
talk was on device work and and you know, controlling the cost of those tokens. And I I think you're right. I think this is gonna be a big issue. I don't think we're paying the price of tokens right now. I think we're getting discounts No way. that that aren't gonna sustain. Right, one hundred percent. Daniel, thank you so much for coming on. what do you got going on as far as books, workshops, and talks? Well, thank you for asking. so I'm finishing my App Intents book and then I'm starting my Foundation Models book.
And so if people want to follow along in tech review, just email me and I'm happy to to let you follow along. I'll probably start that in in December sometime. Workshops if I'm invited. I love to give workshops. as things change, I'm looking at different style books and I'm trying to figure out what I want to be when I grow up. And so I'm considering writing a book on calculus that I've wanted to write forever and a book on statistics and maybe some graphic novels or some that have more artwork, even though I draw like a two year old.
So I'm having a ball. I'm learning new things all the time. I love talking to people like you. I love our community and I love that I get to keep going to conferences. So that's pretty much what I'm doing. Daniel, thank you so much for coming on. it's been great. definitely check out Dim Sum Thinking and his material there. There's a lot of good stuff. yeah. thank you for joining me. if you like this episode, like and subscribe on YouTube or give me a review, please. share it with your friends, share it with your family. and also thanks to all the Patreon supporters as well.
if you haven't caught my newsletter, definitely take it some time and subscribe to my newsletter. Link will be in the show notes below. Thank you everybody and I look forward to talking to you later. Bye.
Rob Whitaker joins Leo to talk about why accessibility doesn't come for free, not even with SwiftUI. Rob, an iOS engineer specializing in accessibility, author of Developing Inclusive Mobile Apps, and a member of the...
Graham Lee returns to chat about what it takes to become a software centaur: using AI to get better at the craft, not just faster. We get into why the job was never about typing code, meat proxies and Jevons paradox,...
Georg Tuparev stops by to chat about software that outlives its plans: projects scoped for a few months that are still in development 20 years later. We get into running a small team of physicists and astronomers, why...
Tim Condon stops by to chat about Vapor 5, the biggest Vapor release in six and a half years. We get into dropping futures and promises for Swift concurrency all the way down the stack, Span and strict memory safety,...
Stewart Lynch stops by to chat about using AI to learn faster instead of just letting it write your code. We get into planning in stages before you build, why Markdown is the universal language of AI, and the tiny...