Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

ADSP: Algorithms + Data Structures = Programs

ADSP: Algorithms + Data Structures = Programs artwork

Episode 213: NumPy & Summed-Area Tables

Dec 20, 202435 minAudio

In this episode, Bryce and Conor chat about NumPy, summed-area tables and more. Link to Episode 213 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Twitter ADSP: The Podcast Conor Hoekstra Bryce Adelstein Lelbach Show Notes Date Generated: 2024-12-10 Date Released: 2024-12-20 NumPy Scalar Functions Monadic Pervasive Functions (Uiua) Numba Python frompyfunc Summed-area tables Summed-area tables in BQN (Tweet) BQN ˘ (Cells) Leading Axis Theory Softmax llm.c Convolutional Neural Networks in APL Intro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons - Attribution 3.0 Unported - CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

ConorMy first point, it's a shadow of APL. Uh, my second point, though, is that uh it's still pretty great. You know, as far as Python goes, if you program in numpy, your Python code is faster. NumPy is faster than Python because at the end of the day, numpy is implemented in C. Welcome to ADSP the podcast, episode 213, recorded on December 10th, 2024.

ConorMy name is Connor, and today with my co-host Bryce, we discuss NumPy, summed area tables, and more you know, you mentioned um in the last episode that you've been doing more and more Python.

BryceUm, and I also have been doing more and more Python the past uh six to eight months. Um and I I don't remember it I think we may have talked a little bit in a previous uh prior episode about uh sort of my search for understanding about what is Pythonic. Uh and I I sort of feel like I've now uh uh I've now I've now grasped Pythonic. I now have a sense of what it is. But um what I uh what I want to talk about before we get into this algorithm is is NumPy. And I want to get your thoughts on what do you think of uh numpy?

BryceBecause for me, um learning numpy coming from C has been at times it has been both refreshing and frustrating, and I wonder if you've had a similar experience.

ConorWith sorry, learning Python or learning learning numpy?

BryceLearning NumPy specifically.

ConorSo let's time box this to two minutes so that I don't go on forever here. My first comment is Oh, and we should say what NumPy is. NumPy is the number one popular library, probably in the world, because Python's one of the most popular languages, first, if not if not first, second. And it is an array library, which means that you've got your uh np, which is typically you go import numpy as np. np.array gives you access to a multidimensional array where you can do APL array language like things with these arrays, such as uh they're called different things in different languages, scalar or pervasive operations.

ConorSo you can for scalar operations like plus, times, minus, you don't need to map these operations, the number of maps over the dimensions of your array. So it like I love to show the Haskell example where if you have a list of lists and you want to add one to every number in that list, you have to go map, map of plus one over your list with some parentheses in there uh in order to get that to work. Uh whereas in a numpy, you just go plus one. And it automatically, using a technique called broadcasting, which is just uh what they call um it's their form of rank polymorphism, it and that's just a rank polymorphism is just a fancy word for being able to apply these operations to arrays of different ranks.

ConorSo integers or scalars have rank zero, lists have rank one, matrices or tables, however you want to call them, they have rank two, a cube has rank three, and it goes all the way up. I think there's a limit in NumPy. I don't know what it is. I think in APL or J, I think the limit was sixty-three dimensions at one point. Which you don't typically end up doing stuff with sixty-three dimensions. But you know, when you're designing a language, there's no reason to really limit it if you can uh design it generically. Anyways, uh that's what NumPy is. Uh what are my thoughts and feelings about it? It is a a shadow of the power of array languages.

ConorNumPy is basically just an APL implemented in Python. I was about to say poorly implemented. I don't want to say poorly. It's just like when you're in a language where all functions are words and you have operator uh overloading, but only to a certain extent, uh, it's what you end up with. You can't do a a a ton better in Python than NumPy. But you lose a lot of the elegance and power of a well, but but but you don't you don't get composition, right?

BryceBecause everything is sort of eager. You can't really do like lazy composition of things.

ConorFusion is what we should say there. You don't get the the fusion of operations. So there is um there are libraries uh out there that try to do this kind of stuff, but um numpy does not. So and that also a lot of the array languages as well do not fuse, which is one of the critiques of both numpy and array languages, is whenever you do plus one times two, and then you wrap that all in a summation, that's three different operations. Two, uh one addition, one uh multiplication, and one reduction. And a lot of the times that ends up in creating whole new matrices.

ConorSo like you can imagine that instead of doing some in place transform in C, I'm doing like a copy if before each of those, and then like then doing my transform. And like I that very sadly, we don't we've already had this conversation. We mix up the underscore copy versions, like the semantics of some of our algorithms, like sort uh is in place by default, but doesn't give you a set of output iterators. Anyways, we've already had that conversation C in C. My first point, it's a shadow of APL. Uh my second point though is that uh it's still pretty great.

ConorYou know, as far as Python goes, if you program in numpy, your Python code is faster. NumPy is faster than Python because at the end of the day, numpy is implemented in C, and C is faster than Python. So a lot of the stuff, maybe paradoxically or um unergonomically, if you take a list in Python, convert it to a numpy array, and then do your operations on that numpy array and then convert it back to a Python list, your code's gonna be like an I don't know if it's an order of magnitude faster, but it's definitely a multiple faster than your Python lists, which means that like if you're looking for performant Python code, become a numpy expert.

ConorEven if you're not doing scientific compute, it's still a great library to do these kinds of you know basic generic programming operations with.

BryceAnd there's also there's also this JIT compiler numba uh which uh can take uh Python code and JIT compile it down to native uh to native code. And in pr in particular, numba is numpy aware, so it can take your numpy code and turn it into code that is going to be uh in a lot of cases as fast as if you'd written something in C. So uh there's one particular thing in NumPy that uh there's a bunch of like little quirks that um I think have just been like stylistic differences between C ⁇ and Python that I've sort of adapted to.

BryceBut there's one thing that it really is a difference in approach that still bothers me, and that is around like user-defined uh uh reductions in scans. So it let's say that you wanted to do like, you know, uh just like a plus reduction, you know, a reduction with just like operator plus on an array of like ints. You you can just do np.sum, right?

ConorYes.

BryceUm and if you want to do a scan, you can do np.cumsum. Or uh there's a longer version spelled cumulative sum. But what if you want to have your own, you want to provide your own operator? Um like if you want to do something like m like uh uh a max reduction, uh there's okay, there's np.max, but maybe you want to write like your own like you know quirky, you know, uh uh operator. Like how would you do it?

ConorSo there are two different ways. Technically, all of the equivalents of higher order functions like reduce, scan, outer product, which I believe outer product is called outer, and as you mentioned, uh well actually I think the generic version of cumulative sum or cum sum is called accumulate, which I've highlighted in several talks and lightning talks, because I think it's uh when I first discovered that I was like, are you kidding me? R reduction in C is what they call a scan in Python.

ConorAnyways, um, so you you all of these higher order functions, I think they have some mechanism where they have binary operations where you can call those higher order functions as methods, which are just manually implemented. So if you want to call, it's it makes more sense where when you don't have basically convenience names, so some is what they call a plus reduction. But if you want to do an outer product on like the max binary operation, there's something where you go like operation.max and then you can call as a method dot outer.

ConorAnd that is the way you do an outer product on a uh implemented binary operation that NumPy provides for you. So that's probably not the answer you're looking for. That's answer number one, though. The second answer is if you want some binary operation that is not provided by numpy, and it's some bespoke thing, which is a, you know, whatever, a couple binary operations combined, uh, the way that I have done it in the past is you need to build up that binary operation, typically in the form of a lambda. You then pass that lambda to this function called frompy func that I think numpy performs, and it basically uh is a function that will take an arbitrary binary expression and then have you you're you're then able to pass that um to whatever like higher order function you want.

ConorSo it's like you go numpy dot frompy func, you pass it your your uh arbitrary binary expression, and then you call like dot outer or something. I can't remember the exact order, but it involves like building up a custom lambda, passing it to this from py funk, and then like using that, which is coming from like C where you just have generic algorithms that you can pass generic operations. It is it's painful.

BryceIt's like taking a yeah, and so part of that is so in numpy there's this notion of a universal function, which is what from py func gives you. And a universal function is like a a uh a scalar function that numpy transforms or vectorizes into something that can be applied to arrays. And it's cool because these scalar functions with they get um they they use the numpy broadcasting rules. And so the broadcasting rules are like, oh, like what happens if you do like if you call something with like a matrix um and a vector, or like you know, like a 2D array and a 1D array, like like how does that work?

BryceAnd there's these this you know complex set of rules that's I think fairly intuitive for how this broadcasting works for different shapes of things, um, so that so that you don't necessarily have to have all the things be the same shape. Um and like one of the reasons why they have these universal functions where like for writing your own element-wise operations is because uh writing writing it as a function and then passing it to this decorator, um uh uh this vectorized decorator, uh gives you more efficient code than you would get if you just wrote a for loop that iterated through the entire array and then did your thing manually.

BryceSo basically it's like to avoid doing like raw loops. But the thing that I just don't understand is why why can't we just have the sum function take another operator that's you know a function or you know, like a lambda, and then under the hood it can call like the vectorize on it. Like, why is it that I have to go in, I have to go and do like the from pi func, something like that, and then I have to call like dot outer, dot you know, reduce or whatever on it. Like it just it just feels very odd. I would I would prefer to do the C thing where it's like, oh, I want to do a reduction, like I can just call dot sum and it'll just do it with like you know, operator plus.

BryceBut if I want to have my own operator, I can just call sum my array, comma, the function I want to call it on. Like that just seems so much more intuitive to me. And I I it just is a little bit mind-blowing to me that that's not a thing. And I think it just it just illustrates the difference in in approaches.

ConorSorry, wait, I didn't understand the you said you want dot sum to take a custom operation? Or did you mean to say like dot reduce?

BryceBecause like dot summary No, no, dot like yeah, I want like dot sum to be able to take a custom operation. I don't want to do this thing.

ConorLike what what does the operation do?

BryceUh no, no, no. The operation would just be like your reduction operation.

ConorWell, so I'm confused. Why do you want to open like because sum to me is like hard-coded to plus reduce. So why do you want to be able to customize the plus?

BryceOkay, maybe maybe then then numpy should have a top-level thing called like reduce. I guess the th the thing that I'm getting at that's odd is if if I'm doing a sum with like operator plus, I call a top-level algorithm thing. But if I want to do my own custom reduction, I have to create my operator, and then I don't pass my operator to some algorithm. What I do is I call the dot reduce method on on it. And that is like a difference that I don't want. What I want is I want like a unified API where there's like there's a thing called like numpy.reduce, and I can call it with you know array, and if I call it with no uh operator, it it implicitly uses plus, or maybe we decide that we don't want to have implicit things, it should always take an operator.

BryceUm, but to me it's that's just more natural. It's just to have it be like a top-level function uh that's like reduce like thing like operator. Instead of what it is right now is it's like um it's like operator.reduce of the thing.

ConorI don't I don't know enough about the Python and NumPy implementation to know if there's any specific reason. Because I agree.

BryceYeah, I suspect there is.

ConorIt is a more ergonomic API, like every language. Uh I shouldn't say every. A lot of languages, including functional languages, C, etc. The way higher order functions work is you pass it the the operation as custom behavior. You don't build up something and then have like a pre-implemented set of higher order things that you'd then go, you know, dot and then call that method. Um but my guess is there's a reason that they they do that and it has to do with it. I think so too.

ConorYeah, but I I'm totally on board. I agree with the reduced thing, uh, definitely. If it was possible.

BryceSo I'm gonna use a remaining time to talk to you about a very cool algorithm that I've been uh messing around with, which is um uh the problem that I first was presented that this algorithm can be used for is a summed area table. Have you heard of summed area tables before?

ConorNo, only from the messages that you sent me, but I never googled it afterwards. Uh I just imagined at some point you would be explaining this to me either via talk or via podcast.

BryceSo so the idea with a summed area table. So so you know, a partial sum, you know, you take an array and you get you get all the partial sums up until the end of the array. Um, summed area table is like that, but just in like 2D. Um so it's it's used a lot in image transformations, um, where uh basically you do uh a partial sum for each row, and then you do a partial sum for each column, such that the um the last element, you know, the the element in like the MN uh row and column, uh like the bottom right, if you think about it as an image, like the bottom right of the image, um that that last element will have the area of the entire thing.

BryceUm does that make sense?

ConorYes, it makes exact sense. I've got BQM pad open. I've already got my five by five matrix of ones. Do you want me to share this? I mean I can walk uh and then I'll stop sharing. Don't worry, we're not gonna lose the listener too bad. So we've got our our five by five matrix of ones, which is five characters, and we do our plus scan with two characters, plus, and then a backslash. And then we want to, after doing this, we want to do under transpose. God, you gotta love under. And then you want to do the same thing.

ConorUm, and then you're done. God, that is that is gorgeous, folks. That's gorgeous, folks.

BryceYou know what's what's so beautiful about what you've just written, Connor, is that the without even realizing it, you have done the the trick to this algorithm that is essential for uh uh implementing it performantly on a GPU, which is the transpose. Because let's think about how we would do this on a GPU. So there's two different scans here. So first we're gonna scan the rows. Okay, so what we're gonna do is first we're gonna let let's assume that like the rows are contiguous in our matrix.

BryceSo first we're gonna go and scan all the rows. Uh so so let's assume that that how we're gonna do this in GPU is that like each thread or maybe each thread block will load one row, which is contiguous in memory, and then it is going to uh uh scan that row. Um and so it's gonna load a contiguous chunk of memory, that's good performance-wise, and it's gonna scan it and then it's gonna write it. But then, okay, what do we do? So now we have to scan the columns. The columns aren't contiguous, and also the current, like we we've loaded in the row to this thread block, and so now what we would have to do is we'd have to do a strided load of the columns, which is gonna be a lot less efficient because we'd basically be gathering from a bunch of different uh locations in our contiguous array, because it's gonna be a strided load.

BryceAnd so what we can do instead um is we uh we load each row, we uh scan each row, and then this is assuming that this is not in place, but that we're writing to a new output. So then instead of storing in uh the original locations, what we do is we do a transpose. We store the rows transposed. And the reason we do this is that we're then going to do a second pass.

BryceAnd in the second pass, we're gonna do the scan of the columns. But because we've transposed the data, even though the columns originally were not contiguous in memory, because of the transpose, when we do the second pass, the columns are now contiguous in memory. And so we can again do the fast thing. And if we do this with two different kernels, what we end up is in the first kernel, we have load rows do a contiguous like load of memory, um, like for the thread block.

BryceUh the entire thread block reloads a contiguous chunk of memory, then it does the scan, and then it does a uh a scatter store. So it does the transposed store. Now that writing to non-contiguous memory, doing a strided write to memory like that, yes, it's not as efficient as doing a strided store, but stores are always substantially cheaper than loads on a GPU because the store is you can do it asynchronously. I don't have to wait for it.

BryceSo I do this store, and then this kernel ends. And then I start the next kernel, which is going to uh uh to load those columns. And the reason that I that I that I do the neck that I do this in two kernels is because I'm I have to have a global barrier after that first scatter store. Um I gotta have everybody wait uh uh until those stores are you know uh uh are done and are propagated, and it's just you know easier to do that with a second pass.

BryceUm and then the second kernel I load the low memory locations that were originally the rows. So I load from the output this time, which is where I've stored these temporaries, and I load um uh the the transposed uh data. So now I'm loading columns, and then I scan those columns, and then I do a transpose store again, which untransposes the matrix, giving us the correct result. And I think this is a very cool algorithm.

BryceUm the thing that got me very excited about this was one, this is an example of an algorithm where I think that the optimal case might be um a two-pass algorithm. Some people have pointed out to me that there's another approach where um you don't do a scan of the entire column or the entire row and column. Well, we'll we'll get there in a second, but but the I'll say this. The two-pass version of this algorithm that I just described, I think is quite an elegant algorithm, which is usually when I look at multi-pass GPU algorithms, I'm like, I like have to hold my notes.

BryceI'm like, I would prefer to do this in a single pass. But I think the two-pass version of this one is so elegant. But I was talking with uh uh Bradley Dice, uh, I think one of your former co-workers uh on the Rapids team about this, and Bradley said, like, oh, this is like a multi-dimensional scan. Um and I was like, that's interesting. I hadn't thought about it like that. But he's right, it's like a multidimensional scan. This can be generalized to multi-dimensions. But then he pointed out something else interesting. In the the scan, the parallel scan algorithm, which uh we implement this thing called decoupled look back, you have this dependency graph where where you, you know, if I'm the nth uh chunk of data, I depend on the zero to n minus one chunks of data.

BryceUm and that dependency chain is like pretty um non uh it's pretty boring for the 1D case. But Bradley pointed out that for the 2D case, it's got this like wavefront property that like if I'm like at like n m, I depend on like this graph of um all of the, you know, of the all of the the chunks that are from 0, 0 up to uh uh up to my chunk in m.

BryceAnd he pointed out that oh, this sounds like once you go to the multidimensional scan, this sounds like it's got a lot more interesting dependencies. Like it it's got this sort of dependency graph, and maybe this is something where you want to explicitly have a dependency graph, especially when you get to multiple dimensions. And I just I thought that was very interesting. Um It sort of reminds me of some of the um there's some linear algebra algorithms uh that have a similar like dependency set, I think maybe like Jacoby um uh kernels, uh but uh but I haven't had a chance to really look into that part of it yet and whether that intuition that this is just a multidimensional scan and that it's got this more interesting dependency graph, does that mean that there's some clever single pass algorithm that you can use to implement this?

BryceUm but uh it's very cool. And this summed area table, it gets used a lot in image processing. It gets used for things like uh depth of field uh transformations. Um I'm working on writing like a little a little example in CUDA Python that uh that's gonna do some some quirky, you know, so like it'll actually like be an example where you can give it an image and then it'll do some image transformation. But uh but I I'm curious about what this looks like in higher dimensions too.

ConorI mean, if I'm not mistaken, the dependency at M and N, it's just the sum of the rectangle formed.

BryceRight, right, right.

ConorAt any point in time.

BryceYeah.

ConorUm but I guess you could get there if you're doing like a decoupled look back kind of thing. You can get there by looking at different points.

BryceYeah.

ConorYeah.

BryceAnd it presents interesting scheduling problems because um you you want all of you want all of your dependencies to be running before you um uh in a scan. Um and in a 1D scan, that's pretty straightforward because on GPUs today it's not guaranteed, but the rasterization order, the order in which thread blocks are executed, happens to be monotonically increasing. So on GPUs today, there's this implicit uh guarantee, which is that not at all prop promise, not at all documented behavior, but happens to be the case that if thread block n is running, then you know that thread blocks 0 to n minus 1 have either run to completion or are currently running.

BryceUm but if you did build a single-pass multidimensional algorithm for this, um you would need to uh think a little bit more carefully about rasterization order, because I don't know if that implicit guarantee works if you're doing like a multi-dimensional uh thread grid. Um but if you linearalize the thread grid, which often happens even when you're doing multidimensional stuff. So even when you're doing launching a CUDA kernel that um that works on multidimensional data, sometimes you'll end up launching it with a 1D thread grid for a variety of reasons, like usually because you're doing something like row-wise or column-wise.

BryceUm but you'll you'll you'd have to be careful with this multidimensional scan to ensure that you have this monotonic progress that uh uh for performance, you'd you'd at the very least, you'd want to you'd want to ensure that um all of your dependencies have started running before you've started running.

ConorYeah.

BryceThe the BQN solution here is is, I think, quite elegant because you did more or less the sort of thing that I described where you you did the the two scans and then you you did the transpose there.

ConorUm and then do you note that I've added a second solution, which it's it's is it ironic? I'm not sure if it's ironic, but it's funny that I came to the first one with the tr with the under transpose first. And that's just because I've been really falling in love with the under modifier in BQN.

BryceAnd what what does under do?

ConorSo under applies a transformation, then does some operation, and then basically unapplies that original transformation.

BryceOkay, so it's like it's like it's it's like transpose it, do this thing, and then untrans.

ConorOh so the classic way to use this is if you want to do a reverse scan, you do an uh a plus scan under reverse, and then if I I give it five numbers. So this is uh the number zero to four, zero, one, two, three, four. And then if I if I if I just do a plus scan, I get zero, one, three, six, ten. But if I want to do a reverse scan, I either have to add some other modifier or higher order function, which is what they do in functional languages. Like Haskell has uh scan L and scan R, one from the left, one from the right.

ConorBut we have under, so we just go under reverse, that reverses it, does a plus scan, and then re-reverses it. So it basically does a plus scan from the back. And we we also deal with this in C with reverse iterators, right? If you want to do a reverse scan, you just you use a different type of iterator. But like this under applies to a ton of different things. The one that I've really fallen in love with recently is using it to basically modify certain parts, like filter out. So say I've I've got my array, let's just do this, uh, an array of five ones.

ConorIf I want to add one to like a certain subset of my array, how do you do that? You know, you could do some scatter-gather kind of thing where it's like, well, I gather the elements.

BryceOh, like a filter.

ConorUh filter, but then you need to reconstruct the original, right?

BryceUrrr, yeah, yeah, yeah.

ConorSo like you it's it's a non-trivial thing to do. Um and like in functional languages where you don't even have mutation, you might end up having to kind of like split your array based on these values, mutate the values you want to, and then reconstruct it. And then reconstruct. Under enables you to apply any kind of filtering, compaction, selection. So this is saying get the first and third index and then apply this unary operation to the indexes or the values that you've retrieved, and then like unapply your selection. So just like anytime you're using Oh, which is like it's just beautiful.

BryceThat's very fascinating. And so like so it's basically like what you're using it for here is like saying, like, give me a view of this thing that I can modify in place. Oh, that's super cool.

ConorAnd like, and so and just in general, like I have used unders with under with reverse and transpose and things like this, but only since doing advent of code over the last couple weeks have I really discovered that uh it's a super common pattern that you want to either build up a list of indexes or build up a uh a compaction mask, a filtering mask, and then you know, get that view into your data, apply some transformation or replace it with some value, and it's just like it's a very idiomatic thing to do, and it's uh incredibly nice.

ConorUh sorry, you were gonna ask a follow-up question though.

BryceUh yeah, yeah. I was gonna say, so tell me about the second solution um that you have to the summed area table, which does not use the transpose.

ConorUh right. So the first one uses under transpose. The second one basically it's just a row-wise scan. So this is the cells modifier, which for a matrix uh converts your operation to operate on rows instead of columns. So by default, a plus so if we get rid of this, the plus scan is on the columns. Yeah. And then if we add back the plus scan, use the cells modifier. For a matrix, this says do this row-wise, which for a bunch of languages you said it's called cells. It's called cells, yeah.

BryceUm I have to say, uh, I don't know, m maybe it's just that uh latent exposure to uh to uh BQN uh through you has been rubbing off on me, but this is one of the first times that I've that you've shown me BQ in code where I sort of intuitively can very easily tell what it's doing, and it like just clicked very immediately. Um like uh I at no point in your explanation did I get lost. Um and usually it takes me a little bit more mental energy to understand what it's doing.

BryceVery cool.

ConorBQN has been my favorite language for a couple years now. Although I've I've played around with uh Jell-O and We Waz as becoming number one, but having used it like more seriously over the last couple weeks from Avent of Co. but also just like the last half year, it is like the gap that it has put on other languages. Like there are just so many things that I've realized are incredibly cumbersome in all languages except for array languages.

ConorLike as as soon as you're doing something with a grid or a map where you have coordinates and like you're checking whether you're going in or outside the bounds of some maps, like all these graph traversals, and it's just so, so intuitive. Like you can add a delta to a position, and you can add a list of deltas to a position in order to move around. And like all of that in uh in a normal language, you have to manually adjust your X and Y coordinates and manually check each whether each one is outside of the grid.

ConorAnd it's just uh I gotta figure out a way to to make this, you know. I you know, I need to get paid to write BQN, is what I'm trying to figure out how to do, basically.

BryceWell, you know, it's it's it's funny you mention that because uh in a in a future episode, uh I so I I've because I've been working on Python and uh this one particular project I've been working on, I've been uh I finally, it's seven years at NVIDIA, I finally had to learn something about machine learning kernels. Um and there's actually like some of the algorithms are uh pretty cool um and intuitive once you get them. And I was just thinking as you were showing me this, like, oh, it would be really cool to implement like flash attention in BQN. Um and uh I I I want to show you at some point uh this uh some of these other machine learning like layers and algorithms like uh uh yeah, softmax is pretty cool um uh algorithmically, but like even like flash attention.

ConorI was literally just thinking like I don't know what the formula or the algorithm is for flash attention, and we'll save it for another episode, but like I did look into doing some machine learning stuff in BQN or APL one time, and I was just like, and literally I was thinking about softmax because softmax is like it's ridiculous. Like it's two operations one after the other.

BryceIt's like it's like Yeah, it's so simple. Yeah, like they make it, they make it sound yeah.

ConorIt's like a division with a floor or something. I can't remember the two operations, but it's like this is nothing, this doesn't need a name. Like, and Relu is like the same thing. Like a lot of these terms are just like composed binary operations, which is literally like two symbols in like so the APL version, it's like a couple lines of code.

BryceHave you heard of like lm.c, which is like a simple implementation of GPT2?

ConorYeah, yeah. Didn't we do a refactoring of that into CUDA? And we we I'm not sure if there's a link online to that, but well, if there is, we'll find it.

BryceUm I I think uh I think I bet you could write like lm.c and BQN and like have it fit on like you know a slide.

ConorProbably. Minus the you know, GPU acceleration.

BryceBut uh Yeah, but that that's that's uh I mean what w when you have it expressed in a rail in array language, I feel like it's it's pretty straightforward to figure out how do you lower it to a GPU.

ConorYeah, maybe.

BryceYeah. But uh but yeah, I I bet it I bet I bet it would I bet it would be very elegant. I bet it'd be very elegant.

ConorYeah.

BryceOh you have me excited about PQN. What are you doing to me?

ConorYeah, well they there's a paper out there called Convolutional Neural Networks, an APL that was published at a workshop a couple years ago. Link in the show notes if folks are interested. Uh so there has been some exploration of this stuff. But yeah, anyways.

BryceIt's funny because I, you know, I've been at uh we've I've been at NVIDIA for seven years, and I mostly have worked on like the compute HBC stuff, and uh uh I think I think some people who are at NVIDIA who don't work on like the AI stuff, like to some degree, I wouldn't say there's resentment, but you sometimes have this feeling is like, ugh, like the stuff I work like this other thing gets all the attention, and the stuff that I'm really passionate about, uh, you know, which is just like general compute, doesn't get as much attention. And it can get to some people. And this is the first time in my time at Nvidia when I've like really had to look into machine learning kernels.

BryceThey're actually pretty cool. They got some interesting algorithms in them. Um and uh and uh I just like I I'm seeing out in six months, and and uh uh I hope my entire world does not become consumed by writing machine learning kernels. But I don't know, maybe I'll have fun with it. We'll see.

ConorI mean, uh if we're all out of work in ten years, you know, I'm just gonna write BQN programs full time for fun. Uh I I know what my hobby will be if uh the AI takes over the world, but we've still gotta we're still a ways away from that, I think. Be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day.

BryceLow quality, high quality. That is the tagline of our podcast.

ConorThat's not the tagline. Our tagline is chaos with sprinkles of information.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Episode 313: C++26 Reflection vs CRTP, AI, Kids & More with Zach Laine

    Nov 20, 202647 minAudio

    In this episode, Conor and Bryce chat with Zach Laine about C++26 Reflection vs CRTP, AI, kids & more. Link to Episode 313 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP:...

    • Apple Podcasts

    Credit the people on this episode

  2. Episode 312: The C++26 Contracts Fiasco

    Nov 13, 202652 minAudio

    In this episode, Conor and Bryce chat with Zach Laine about the C++26 contracts fiasco. Link to Episode 312 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP: The Podcast:...

    • Apple Podcasts

    Credit the people on this episode

  3. Episode 311: Career Changes, GPUs & More with Zach Laine

    Nov 6, 202628 minAudio

    In this episode, Conor and Bryce talk with Zach Laine about his career and the career changes he has made. Link to Episode 311 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials...

    • Apple Podcasts

    Credit the people on this episode

  4. Episode 310: Leaving NVIDIA for... ?

    Oct 30, 202624 minAudio

    In this episode, Conor and Bryce talk about Bryce's next chapter and where he is leaving NVIDIA for. Link to Episode 310 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP: The...

    • Apple Podcasts

    Credit the people on this episode

  5. Episode 309: Default Construction & CppCon 2026 with Jason Turner

    Oct 23, 202629 minAudio

    In this episode, Conor and Ben chat with Jason Turner about default construction in C++. Link to Episode 309 on Website Discuss this episode, leave a comment, or ask a question (on GitHub) Socials ADSP: The Podcast:...

    • Apple Podcasts

    Credit the people on this episode