Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Data With Direction | Gambill Data Podcast

Data With Direction | Gambill Data Podcast artwork

Live Data Friday: Escaping the 9-to-5, Local Networking, and Live AI Builds

Aug 16, 202657 minAudio
ShareXLinkedInFacebookWhatsAppEmail

A live session with Databricks’ Scott Haines covering the reality of transitioning from corporate IC to independent consulting, special pop in comments from the great Ryan Dolley helping us to identify the breakout skill of "touching grass", and reinforce the importance of in-person community and why it beats cold outreach, as well as how autonomous coding agents are reshaping technical workflows. Get full access to Chris Gambill at gambilldataengineering.substack.com/subscribe

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

SPEAKER_01Come on, StreamYard. StreamYard Live.

SPEAKER_00Yay. All right. So welcome back to Live Data Friday with my very creative name as a data engineer. We're all so creative. We again have Scott Haynes, staff developer advocate. I can't speak on a Friday at Databricks. Super excited to be with Scott again. Always enjoy hanging out. And yeah, you mentioned you had an idea about what you want to do today.

SPEAKER_01Yeah. So thanks again for having me on the show and just for hanging out on a Friday, which is super fun. I've got my morning coffee. The mug itself comes from a random place in Connecticut. It's a donut shop. That's a good story. It's a good Friday story. Not really.

SPEAKER_00My very generic Dunkin' Cup. No, I love it. Yours is much more interesting.

SPEAKER_01That's the amount of coffee that's needed on a Friday. This one's not deep enough. It should just be like yay big. But yeah, no, I was thinking a little bit more about just kind of interesting things that we can set an agent to go build. And one of the things I was thinking about this morning, are you familiar with sequencers for music?

SPEAKER_00Yeah.

SPEAKER_01I think the last time I attempted something like this was probably with Flash 25 years ago, maybe. And so I want to come in as a complete idiot and just be like, you do the work, you figure it out, let's go from there. And maybe have something that makes music at the end or maybe it will be an epic failure. And that was kind of, that was the thought that I was having. I don't know if it's a good idea or a bad idea. It's data specific because we need to keep track of like a time series in order to make music.

SPEAKER_01It's just a little bit different, you know, approach to time series, you know, data. Because if it's good, it makes good music. If it's bad, it still makes bad music, but it still does it accurately. And so there's probably some analogy there for, you know, data pipelines and things like that. For sure.

SPEAKER_00For sure. Yeah. Because there's, There are plenty of not great data pipelines out there that are running that-

SPEAKER_01That aren't making music?

SPEAKER_00That aren't making music, but get the job done. Yeah.

SPEAKER_01Yeah. So yeah, I don't know if it's a good idea, but I think it was, I don't know, we can just play around with it and see. Hold on a sec. So yeah, do you want me to share my screen and we can kind of go from there?

SPEAKER_00Yeah, that'd be perfect.

SPEAKER_01All right, awesome. Let me see. Yes, it is easiest with two monitors. That is correct. But let's go and just try and do everything inside of the on the joint. All right, cool. So the screen's up on deck. We're creating a new session over here. Let's start off with a simple prompt. Let's describe what we're going to actually do in this session.

SPEAKER_01Sorry for the mechanical keyboard. It's like tap, tap, tap, tap. People might like that. I don't know. I don't know if it's showing up on the mic as well. Today we want to create a sequencer application. It allows us to create MIDI tracks over time series with loop capabilities.

SPEAKER_01I can't spell. I apologize for my inability to spell. Capabilities. It's going to know anyways.

SPEAKER_00That's all good. Let's just kind of see where we go from here, though. I think you need to connect your host.

SPEAKER_01Oh, yeah. I had a host. Why is it offline? It's running my machine. Let's always allow. Thank you for that.

SPEAKER_00There it is.

SPEAKER_01All right. Let's go move my working directory for this. Hold on a sec.

SPEAKER_00I'm going to go create a new

SPEAKER_01You know what? Forget it. That's going to be fine. We'll just work in this directory. It's OK.

SPEAKER_00Is that your UC directory that you've been working on? Yeah.

SPEAKER_01Yeah. Let's get creating another one.

SPEAKER_00We don't want to. Let's not break that.

SPEAKER_01No, no. It's fine. It's going to be good. All right. So let's go into here. I'm going to go into my terminal. Just go create a new location. I'll just toss it into some place in Git.

SPEAKER_00OK. You're doing it the old school way, Scott, actually creating your folder, not just blindly asking how many that I created for you.

SPEAKER_01Now we get a sequencer, though. So we get the sequencer. Let's go ahead and we can be lazy. We can just copy and paste this from over here while this other one is running. Go back over here. Go back to Sequencer. Run this over again for this one. We're going to go ahead and just cancel that. And I'm going to go and just delete that other session.

SPEAKER_00It doesn't matter. OK.

SPEAKER_01So we'll see where this goes. But in the meantime, I don't know. Chris, do you play an instrument?

SPEAKER_00When I was younger, I played trombone.

SPEAKER_01Oh, nice.

SPEAKER_00OK.

SPEAKER_01Maybe we can see if there's any free sound libraries. We can make some EFX trombone tracks today as well with our fancy new something, right? Sequencer might come around.

SPEAKER_00Yeah. My daughter plays guitar, so that would be.

SPEAKER_01Oh, that's a much better. That's a much better. Sorry, that sounds bad. Oh, no, it is. You're not wrong.

SPEAKER_00Definitely much cooler for sure, yeah.

SPEAKER_01Yeah, I played when I was in high school. I played in like the jazz band and I also played marching band. And the trombone at that time was like really cool because like ska was like a big thing that people were listening to. Yeah. And so it's like, oh, it's like you play the horns. That's awesome, right? Like this is cool. But I feel like I guess trombone maybe does get like a bad rap. So I don't know. It's fine though. You can always play the tuba. The tuba has always had, you know, it's not been the coolest thing.

SPEAKER_00Yeah. And they're heavy to march with.

SPEAKER_01Yeah.

SPEAKER_00All right. What are we doing over here? It's pondering. Yeah. Just thinking. It's on extra high, though. So I mean, it should be.

SPEAKER_01I mean, it is thinking. So we'll see. That's my problem. I'm in Opus 5. We'll see. We'll see what happens.

SPEAKER_00See what happens. So while it's pondering, let's maybe talk a little bit about What have you been working on this past week? Can you share? Do you want to share?

SPEAKER_01Yeah, I can share. So we're... Oh, cool. Sensible defaults. Thank you so much. Yes, you can approve this, and let's just keep going again. So I've been working on... So we're doing a release for Delta. So Delta Lake 4.4.0 is coming out. That's bringing a lot of performance updates. It's a lot more kernel-centric for people who've been following along with the Delta Lake project. We have a common central Rust kernel that's now being essentially the foundation for

SPEAKER_01the entire project across both the Java side, through the FFM for Java 25, and then also through our Rust side as well. So there's been a lot of progress there, which has been fun. But yeah, so that's coming out. I'm working on a release blog right now. It's not as fun to be like, oh, yeah, this is going to be cool, but it's a good overview for the agents to look at and are for people who want to kind of dig into the release. And then we're doing another release kind of in parallel with the Unity Catalog.

SPEAKER_01Unity Catalog 06 is coming out, and that brings full metric view support, which is pretty cool. It's all ready for Like Spark 4.2. So it supports like a lot better, like, you know, SQL view support, things like that as well. And also like the official metric view support. And so it's all sort of kind of there. I don't know how cool this is, but the other new thing that's been added to the Open Source Unity Catalog project and Delta in general is the ability to have generated identity columns,

SPEAKER_01which is cool. Another thing that is an inheritance thing from Spark, but now if you want to have You no longer want to have an identity column that you're building yourself. Yeah, you can now just generate it on the fly so every new record is generated correctly. So it's like SQL's had this stuff forever, but it's kind of fun when it comes to the OpenTable projects and things like that as well. So generated columns. Those are fun.

SPEAKER_00Yeah. No, I'm super cool about some of those new releases coming out. Like the metric views we were... I was talking to my AEE yesterday, and we were talking about some of that. It's going to be good for companies, I think.

SPEAKER_01Yeah.

SPEAKER_00Between ontology and the metric views, I think will help a lot of companies.

SPEAKER_01Oh, for sure. I think for a lot of people, there's always that question of the why. It's like, why do you want that? Why have a metric view? It's like, well, have you ever asked two people the same question and gotten two different results? That's a good reason to have a metric view, right? But I think there's also somebody that asked me like at a conference a couple of weeks back, it's like, you know, what's the, at what point do you need them? And a lot of times it's like when you get into issues with like scalability for like reporting and, you know, there's like the unknowns,

SPEAKER_01like a lot of times it's like that tribal knowledge can, that can survive like a company of 10 people, it can survive a company of 20 people, but it starts getting really complicated when I guess now like when agents are introduced and everyone's creating their own, you know, many more Assets than we ever had before, right?

SPEAKER_00Well, yeah. And I think the first time you get into an executive meeting and somebody asks about revenue and you've got your salesperson giving one number and you've got your finance person giving another number, like that's the day that you know you need to start using something like metric

SPEAKER_01view, right?

SPEAKER_00I mean, that's just probably before you get to that point, but that's definitely the point. Ryan's here, man. What's up, dude? What up, dude? Ryan just got a comment here. These videos are more meaningful than my nine-to-five data rolls.

SPEAKER_01Oh, my God. Thank you so much. All we're doing is crunching on something right now.

SPEAKER_00Yeah, just hanging out. Any insight on how to escape the nine-to-five but still stay in the data industry?

SPEAKER_01Oh, my God. Start later. That's how we get out. Start at 10? Yeah. Yeah. Start at 10. Then work till 6. Yeah. I don't know, Chris. Do you have any advice for, I mean, I've got an idea, but I don't want to steal the show. I currently have a very slow cloud. I feel like Opus 5 is running slower. My machine is just a pile of crap today.

SPEAKER_00It's got to be Opus 5, right? I mean, it's not really your machine. So yeah, we could jam about this while we're waiting on that to run.

SPEAKER_01Yeah. We only have one model, so let's keep going. So we'll deal with it however we deal with it. It's fine.

SPEAKER_00That's right. That's why it's Live Data Friday because whatever happens, happens.

SPEAKER_01That's going to be so good.

SPEAKER_00Let me answer this and then we'll answer Ryan here and ask him about what we're working on. For me, I didn't escape my nine to five until later in life. Okay. That's a lie. I escaped my nine to five in 2008 for a few years, but 2008 was a bad time to escape your nine to five because, well, the markets kind of, you know, shit the camp. So, but it's all for me.

SPEAKER_00It's after, especially after working at AT&T for 14 years, I had a Ingrained kind of network of people that had gone to other companies that I had worked with that I'd already built relationships with and previous employers that I was able to reach out to at first to help me kind of make that jump. I would recommend for anybody at first, though, to kind of test the waters and see is there Is there an interest?

SPEAKER_00I mean, people get all these comments, and Ryan probably has a terrific insight into this, man, as well. But test the waters first, see if you could get some side gigs where you're helping a small business with data. I mean, one of the things that I'm getting ready to walk into is the Pediatric AIDS Foundation. Need help with some Databricks training. And they lost their USAID funding and they need some help.

SPEAKER_00And I'm actually going to help them pro bono do some training with them. But for somebody just getting into it, like those types of situations where A small business doesn't have a lot of funding, but could use some help with some organization of their data or even some training on what the art of possible is like. Those are great ways to start testing the water and seeing how you feel about that type of situation.

SPEAKER_00But yeah, for me, it's been all about networking, talking to people. If I usually get in front of people like I was. Part of the Chamber of Commerce and would go to sessions. And I enjoy the ride a nose that I enjoy the training piece anyways.

SPEAKER_01Yeah, I mean, dude, that's awesome. I think the paying it forward thing definitely always helps as well. I mean, like the escape of the nine to five, right? Like nine to five is what you put into it, right? I feel like for me, I would always try and gamify like the things that I would be doing. It's like even if you don't feel like what you're doing is fun or like, you know, everything can be fun with like the right mindset. There's a lot of stuff where you feel like you're the janitor in England working on a fatberg.

SPEAKER_01A lot of the data plumbing stuff is not the sexiest of work from time to time. It's like, I need this thing to be good. I want this data to be clean. And I may or may not get the kudos that I want for the amount of effort I put into making this good. But it's like you can create libraries to help with it. You can, you know, you can innovate differently than just kind of like doing the same thing over and over and over again and like feeling the monotony of it. So that's like, you know, you can turn into something fun, turn into something fun.

SPEAKER_01And if you can't turn into something fun either, it might not be the right job. And so that's always another thing to kind of, you know, be on the lookout for as well. Like, do you fit with the company? Do you fit with the culture? Do you like the people? Or is there any kind of like, you know, forward momentum, like in the career? Are you still learning? Like I think for a lot of people, it's like when you stop learning, that's usually when you're like, ah, the job starts to feel like it's like, if you're the smartest one in the room, that's kind of like the pinnacle of where you can be. But if you're also not teaching other people at that point either, then you're kind of like, I get, I used to get restless like that.

SPEAKER_01And so that restlessness is, I think, what makes it feel like you're trudging through. And it's like 2 o'clock always comes like a month later. And you're like, why is it not the end of the day yet?

SPEAKER_00Yeah. I had a boss that would tell me that I know that you're bored when you start bringing problems to me and solutions because you just start poking around at trying to break stuff. But yeah, I would argue that even too, that if you ever look around And you think that you're the smartest person in the room. One of two things. Either you need to probably check your ego a little bit and or you really are maybe

SPEAKER_00in the wrong room. Like, go find another room. Ryan's got some great insights here, too. Literally had a meeting with a local food processing company this week to talk about their analytics. And I would agree. When you start poking around at the work that's out there, there is a ton

SPEAKER_01of work out there.

SPEAKER_00From my perspective, there's no end to the amount of data work that's out there to do. And Ryan made this connection through my purpose. Oh, cool. Right? So 100% talking to people. And it feels like a cop-out sometimes, and you hear that a lot. But it's true, like networking, I'm a full believer of what you put out into the world, you get back. And because I've experienced it myself,

SPEAKER_00but the more helpful I am,

SPEAKER_01I love what you just said, too, though, because in my head I picture the more you know type thing from back when we were kids. Leave the house. The 9 to 5 is the 9 to 5, but one of the things that can also help is getting up. Back when I used to work on site more often, I used to really cherish walking around Taking a step back, I used to work at Yahoo back in between 2011 and 2016.

SPEAKER_01They had this, it was in California, right in the Sunnyvale area. There's this thing called the Bay Trail. The Bay Trail goes from San Francisco all the way down, I think, to Gilroy, not to Gilroy, San Jose somewhere. It's a very long road, but you can walk it and it's fun. The team after lunch used to go and walk together. And we'd walk for 30 minutes. You get some exercise. We get some vitamin D from the sun. It barely rains in California, so it's not something you have to really worry about. But that in itself also kind of took the monotony at the end of the day

SPEAKER_01because it's like you're not sitting there usually talking about like, you know, what are we triaging? What work is happening? It's like, you know, talking about family, talking about things that people actually like and like really connecting. And I feel like it's very hard to not really connect if you're also not, you know, Yeah. Oh my God. Yes. Right. Touch grass. Please touch grass. Right. Like people say it and then it's like, we still sit here and I'm like, I haven't left my office in weeks. Right. It's like, but it's an important thing. Like it's definitely like one of the most important things because,

SPEAKER_01you know, nowadays, like there's, there's, there's an endless amount of work if you look at it like that. And, um, the way that you kind of compartmentalize or like break up your day is great. For mental health.

SPEAKER_00I think that's one of the biggest things moving up in any, whether you come up via software engineering or data engineering, is understanding that there is really no end to the work, to that ticket queue, and put it down and walk away. Or your nine to five will very quickly, especially working remote or working from home, will very quickly turn into nine to nine.

SPEAKER_01Or 7 to 10. You can just keep it expanding in both directions. You can even connect. You can make the loop, yeah, for sure.

SPEAKER_00So Ryan did ask what we're working on.

SPEAKER_01So go ahead. I can show more of the native stuff behind the scenes, like what's actually happening. When we switch to the other view, what we're looking at right now is Omnigent, which is an open source project that DataRex is open sourced during the data summit. When we're just waiting for the end result and completion lines, it's not the most fun to look at. We're not really working on anything right now other than attempting to get AI to create a MIDI-type sequencer. This was an idea I had for the show for today.

SPEAKER_01I've got a 12-pad sequencer at home, and it's kind of cool. So yeah, the time series will make good music. We hope so. It could make bad music too. But at the end of the day, if we can just get a piano roll going, I'd be happy with that. We can add sounds. We can see what they have out there for free music. What's a free sound library that's available? Hopefully, we're not going to actually steal it from anywhere. So we know that agents and the LLMs behind the scenes now have very sneaky maneuvers

SPEAKER_01and ways of getting things. We'll just see what shows up, right? And if it's copyrighted, then we'll just delete this afterwards. So it's not going into GitHub yet. So hopefully, it's doing the right thing. But yeah, we're making a MIDI sequencer right now.

SPEAKER_00Yeah.

SPEAKER_01It's a new software.

SPEAKER_00And that's why it's Live Data Fridays. We're doing something live. We're trying it out. We're using Omnigent, which is super cool. I've come to use Omnigent quite a bit

SPEAKER_01here lately.

SPEAKER_00So I'll be happy when I can spin up a sandbox host because I'm on an Azure Databricks environment. So I'll be happy to be able to do that at some point soon.

SPEAKER_01Soon. We have fingers crossed for that. I don't know the timeline for the internals, since I'm working on the open source side of it. But soon, because usually it's like things will roll out to Amazon, then things will roll out to Azure, then to GCP in those kind of release cycles. Yeah. All right. Let's see what's going on. I think I have to also give it my PyPy proxy. Oh, okay.

SPEAKER_01Not my 8 p.m. p.m. Yeah, I was way off. It's a morning. I hadn't had my coffee yet. I'm giving it the wrong registry override directory. It found it. Now everybody knows. It's like, good news. I found it for you. Here's the registry you didn't want to put on camera. It's like, okay, good. You have to be on our VPN. Let me show the world. Yeah. Oh, my God.

SPEAKER_00Thank you so much, Andajan. This is great.

SPEAKER_01Yeah. Okay, it's fine. It's fine. Okay, let's go over here for a sec. We'll just let it run and do its thing. But I think the cool thing, so to the point, right, where, like, the agents have made the live demo. Love them more boring. Way more boring. Because we're, yeah, we're just waiting around for it to complete something that we haven't thought of yet, right? But also, on the other side of that, too, if we just had the sequencer and then we were kind of walking through, like, the code, That used to be one of my favorite things to do, right? Where it's like, here's what we're going to do today.

SPEAKER_01And then you start off, it's like, boop, boop, boop, boop, whatever the sequencer would sound like. I'm not a synthesizer. And then you're like, let's break apart how we actually built it. Sometimes that can also be like, if we take a look at the right-hand side, we just take a look at the file system. And we take a look at, it's like, okay, how exactly did we get here? Part of that stuff can be a little bit complicated as well to get there and to be happy with it. So, yeah. Good luck with that today, Ryan. Thanks for hanging out, Ryan. Cheers, man.

SPEAKER_01Good luck with that. But yeah, that whole kind of like working backwards from like the finished product, I think it's a nice way of kind of showing you where we wanted to get to. And so now like it's just like the reverse of that only we're just not the ones doing the thinking. In this case, we're not even orchestrating. We're just waiting.

SPEAKER_00We're just, yeah, we're the managers, right? We're just managing the...

SPEAKER_01But things are going well, right? So we almost have the dev server. We're working backwards through that right now. I'd love to see what this actually comes up with.

SPEAKER_00Yeah, I'm super interested. Local LLM, ooh.

SPEAKER_01Yes, yes. So on my Mac Studio, I've done that with the Docker model registry. So what's up, Daniel? Good to see you as well.

SPEAKER_00Good to see your picture on LinkedIn. All the cool people are showing up today. Ryan was here. Daniel

SPEAKER_01was here. I feel blessed. Here we are just trying our best getting some application out the door. But yeah, Daniel, if you ever want to kind of get together, we could do the three of us could just go and hack on Kimmy or something else like that. I mean, Kimmy doesn't really run on my machine. It runs and then freezes the whole machine because it's not enough RAM. But we can get some smaller open weight models and go to town and see where we get.

SPEAKER_00Yeah, that would be super cool.

SPEAKER_01Battle of the models. See what comes out better.

SPEAKER_00See who could train the best model. Nice.

SPEAKER_01Yeah, that'd be fun. Maybe next week. Yeah.

SPEAKER_00I was talking to a business the other day where they're using open weight models. They're an AI company because everybody's an AI company, right?

SPEAKER_01Oh, yeah. You have to be. That's the cool thing to be.

SPEAKER_00But they are using open models and training them for specific businesses. But their proprietary thing is they can take an individual piece of the training, basically pull it out, retrain just that piece, and put it back in, which to me was a super cool concept. Because otherwise, you kind of have to retrain the whole model all at once. Like if something didn't quite work out, it was a neat concept.

SPEAKER_01I mean, that in itself, it reminds me of dynamic partition overwrites, right? It's like, I mean, to nerd out a little bit. Not as cool as being able to essentially do surgery on an LLM. But if you're doing surgery on a gigantic table and you don't want to overwrite the whole table, I'm like, these are tips. These are the tips and tricks that come out in these shows, right? Right, yeah. Good stuff. But that is actually really cool. I think like, because I feel like right now it's like the models are getting like incrementally better.

SPEAKER_01Like I remember like the big, like we talked about this before, but it's like there was like the really kind of like gigantic leaps, right? Where it was like, you know, it's like the Opus 4 series, like Sonnet 4 series. Then there's like incremental improvements and then like 4.8 kind of blew everything out of the water. And now it's like kind of incremental improvements as we come out to like, you know, for Opus 5 and like, you know, Sonnet 5. They're incrementally better. I used Grok 4.6 earlier this week because it's like 50% off. And just to, you know, play around with it, test it. That's pretty fast.

SPEAKER_01And that one, it seems to be pretty great at like a lot of things as well. That's interesting, right? Because I feel like everything's kind of, everything's like pretty great. Like, so it's like, Where do we go from here? How much? I don't know. I still want to do something. I still want to do some work. I don't want to just do what I'm doing right now where it's like, hey, go build this thing. I'm never going to look at it. Just if it runs, it runs, and that's awesome.

SPEAKER_00It's like blindly copying from Stack Exchange and just trusting that it's going to work just fine.

SPEAKER_01When is it ever broken?

SPEAKER_00We're almost here, by

SPEAKER_01the way. It's here too. Hey, Ed. Add chat to YouTube. Send a comment.

SPEAKER_00Yeah. Go back to YouTube and put a comment in there, man.

SPEAKER_01But also, hey, Ed. How's it going? Good morning. Actually, good afternoon. So this is your lunchtime for you all.

SPEAKER_00Just snack all day. It's fine.

SPEAKER_01Oh, I never asked. How did the Knoxville coffee hangout go?

SPEAKER_00It's in a couple of weeks.

SPEAKER_01I was way off.

SPEAKER_00Yeah, you didn't miss it. It's going to be great. It's going to be great. I actually have 10 people signed up for it, which I feel like is a good number of people for a first kickoff session. No,

SPEAKER_01it's awesome. I feel like with a lot of the low-key happy hour type things, Honestly, it doesn't matter if you get a ton of people, right? Do you find those first foundational people that will continue to come up? Are you enjoying the conversation? Are you learning together? The real life stuff, I feel like it's so much more important now that we have an entire team essentially sitting in a computer. You still need that real life networking. So again, going out and touching grass, yes, 100%.

SPEAKER_01It's also the way that you kill the nine to five by, you know, making it interesting, you know, talking with real people.

SPEAKER_00So just what Scott was talking about. So I started a low key Knoxville data and AI meetup. Our first session, our first meetup is this morning coffee and conversation. Wednesday, August 26th from eight to 10 a.m. Because I'm an early person. And because I have an 11-year-old daughter, they have to take to school anyway. So at the Empty Cup in Knoxville, super excited about that.

SPEAKER_00For those of you that don't know the area, the Empty Cup is a really cool place where you could go. They have couches and tables, and you get your coffee, you hang out. It's super homey. And their proceeds go to local adoption centers. So also really cool from that standpoint.

SPEAKER_01Yeah. I love it when a coffee shop stands for something. I can't remember the full name of it. I think it's Cafe 1951. It's in Berkeley, California. Proceeds go to refugees. And there's also a nonprofit called Red Rock Coffee in Mountain View. I don't know what their deal is. I don't know if they're giving things away to charity or anything else like that. Their coffee is really expensive and it goes to something more than I think it's just like to run the collective.

SPEAKER_01I don't know. People just can kind of work out of there for free, which I guess is a thing because you do that at Starbucks or whatever else. But I remember they stood for something. I can't remember now what it actually was. But that's so much better than just going into like a Pete's coffee and you're like, thanks for your coffee. And I apologize to Pete's if you were watching. Like your coffee is still good. Yeah. You know, I'm sure there's like some there's charities. Everyone has a charity going at some point. Right. I'm like, we're getting called out. It's like.

SPEAKER_00Okay. Super homie with the super homie stone.

SPEAKER_01What up, dude? Yeah. Man, okay, so we're almost done. So build is passing, which is good. That's good. It's running itself in the browser, so please do. Yeah. Oh, yeah, just keep going. Keep going. All right. So let's see. Now it's got to go open up the browser. Let's see what it does. Bring something next. I don't know what the new rules.

SPEAKER_00Daniel's calling this out. Have you seen the new AI provider workflows in Airflow? Will Databricks do something similar with workflows? I heard about it. I've not gotten a chance to check it out. I also really want to check out, like, this GrokBot thing, too.

SPEAKER_01Oh. All right. I gotta go. Hold on. I gotta go add some security setting stuff. I don't know if that was for, like, opening up the stuff in the browser. Hold on one sec. All right, well, I got, that's fine. That's fine. I was asked to, like, to do, anyways, the app wanted to do something on its own. I'll have to go take a look at that afterwards. So screenshot app is running.

SPEAKER_01I think when it did, like, I was looking for, like, the local Chrome, it got, like, a security pop-up came up, and it's like, hey, it's like Omnigent wants to do some stuff on, like, your local browser, blah, blah, blah. Now I got to log in to the... Good. You're not seeing anything coming up right now, right?

SPEAKER_00No.

SPEAKER_01Perfect. Good. Keeping my security secure.

SPEAKER_00It's important to do that. While you're keeping your security secure, I'm going to plug my meetup one more time. Oh, do it. Yeah. So we're doing the coffee meetup. And then at the end of September, we're going to do a dinner meetup. And I'm actually having Dataflint is going to come and talk about cost optimization in Databricks. So it is actually, it will be the inaugural kickoff of the Databricks user group for East Tennessee

SPEAKER_00that will be kind of co-combined with the low-key data and AI kickoff, low-key data and AI meetup that I'm doing. So super excited about, I'm going to butcher his name, but the CEO of...

SPEAKER_01Oh, many?

SPEAKER_00Many, yeah. He's going to come and talk at that. So super excited for him to come and talk at that.

SPEAKER_01It's going to be super cool. He's awesome. So that's going to be fun. Ask him about drums. He loves playing the drums. So going back to the music theme of today as well. Very cool.

SPEAKER_00Nice. See, it all comes back around.

SPEAKER_01It sure does. And to that point, this thing, I believe, is going to come back around. It's interesting, right? When left to your own devices, what do you do to get around security? It's like, well, I couldn't get to Chrome because you waited too long, Scott. So instead, we're just going to go and we're going to run headless Chrome ourselves. We're going to take some screenshots. They're probably using Playwright for that stuff. And it's like, does this look how I want it? It's like, I hope it does. We'll find out, right? So models have gotten super smart. So again, going back to, I don't know, I was asked yesterday,

SPEAKER_01I was talking with a friend at dinner last night, and the question came up of where do we see things in 2027? What does your workflow look, what do you believe it will look like? I believe it's going to look similar, just faster, with less mistakes, things like that as well. But also, I'm finding recently that I mean, for something like this where it's like we don't really care about like the end result other than the fact that maybe it compiles and it does something fun and it's like that's neat. We did it in less than an hour.

SPEAKER_01Like I feel like they're like the mistakes. I'm no longer kind of seeing the same kind of like horrible mistakes like with the code. It's like it runs, it compiles, it's writing good tests as long as you're kind of directing it in that direction. But I feel like in a year from now, it's going to be a lot more of this. Well, actually, before I say exactly what I think, I don't know, Chris, if you have any thoughts on what will be in the year.

SPEAKER_00So I do. And I like where you're going with that, right? So years ago, one of my biggest, when I was younger, one of my biggest frustrations before I learned how to ask enough questions and maneuver around it was that people did People give you what they think that you need to know, right? Not that they're necessarily hoarding information, but especially for data groups,

SPEAKER_00understanding what the full scope of what the project is and what that end vision really looks like is a huge thing for data groups because we're basically the The groundwork of what that end vision

SPEAKER_01looks like for most things.

SPEAKER_00And it's going to come around full circle to that AI here in a second. And to that point, I think that a lot for me, a lot of the biggest issues that I saw with AI when I was first using it was I treated it the same way I was treated, right? I was giving it like bits and pieces of what I needed to do, right? What I wanted it to do instead of giving it the full picture, right? Giving it that whole context of what that looks like, what my end goal is.

SPEAKER_00And I think that I found, especially with Omniget, Giving it that very detailed full picture. I've even given people the guidance to actually go out and sketch out what you want it to look like at the end, even if you're literally using pen and paper. I'm old, so I literally have a notebook beside me with a pen. And so whether you're using Lucid or you're using PowerPoint or you're literally, thank you, Scott. I appreciate it. I got it too.

SPEAKER_00I'm also old. I have a clean stock as well full of pens.

SPEAKER_01That's where they end up.

SPEAKER_00That's right. But taking a snapshot, take a picture of that, upload it into the LLM along with everything, all the context that you could give it. It does an amazing job these days. The biggest things I see are double checking things like idipotency and making sure that that's in place. Double checking that When it's loading data that it's not doing overwrite

SPEAKER_00for everything instead of. Right. The same kind of things that you would double check like a junior on. Right. Or somebody doing an individual like little

SPEAKER_01portfolio project on.

SPEAKER_00But yeah, those are my my thoughts. I think that in. 2027 will be. We'll be sitting here looking at Omnigent, probably waiting for it to be done and maybe giving it some steering or some direction.

SPEAKER_01Hopefully it's not working on the same project still.

SPEAKER_00Hopefully it's not still working on the same project.

SPEAKER_01We'll see. It's going to be good.

SPEAKER_00So what was the rest of your thought there?

SPEAKER_01I think my thought was, the question kind of came up, if every engineer is a 10x engineer, if they have done the due diligence and learned how to use a good coding agent, then when do we start seeing the 100x type engineers? And does a 100x engineer mean that you need less people? And I think a lot of this kind of goes back to, you can do a lot of good work if you understand how things should be built. But the question was more on like, you know, kind of like,

SPEAKER_01it's like, you know, can AI design a really good distributed system or is AI very good at doing like front end like development? And like, I find that it's very good for the most part in creating like, you know, kind of like a closed loop system that has like a very specific, you know, starting point and ending point. But I believe by like next year, it could be like, you know, you know, I don't know, let's create like a company like a Databricks, right? Like, do you have enough tokens to go and kind of essentially like, I shouldn't say that, but say a different company, right?

SPEAKER_01It's fine.

SPEAKER_00What if they snowflake?

SPEAKER_01No, I can't say that either. So we can be vendor neutral, right? So if you think about the things that you use on a daily basis, like how many tokens would it take for you to recreate that? And it might not be a ton, but then it's like, what will it take you to scale that out to be able to support like enterprise, like, you know, customers? I think that's the part right now where I don't think AI is good enough yet to be able to do like all of like the, it's kind of the check and balance work of like really, I mean, this might be just my, like me being naive at this point,

SPEAKER_01maybe this is already a thing, but like with enough direction and with enough small projects, you could probably get there, but like, you can't one shot like an enterprise

SPEAKER_00You can't. So I ran into this with one of my coaching students the other day. And my article, not this past week, but the week before was about it, my Substack article, about salting, data skew and salting. That's one of those, when you're working on a project, keep the future in mind kind of thing. Keep scale in mind. Yes, you have this size of data now, and it looks pretty clean. But I promise you in a year, in two years, it's not going to be. You're going to run into data skew and you're going to need

SPEAKER_00to employ something like salting in order to get it to process efficiently so you don't have a node sitting out there that's been running for an hour and a half while the rest of the nodes have been sitting there. That's one of those things that right now AI is not necessarily going to pick up on and set up up front.

SPEAKER_01Unless you tell it, right? I mean, those are some of those, like, the upfront things to, like, if you think about, like, going into planning mode, right? Like, the same thing, like, to your point, if you're like, okay, like, you know, we're creating a pipeline that is, you know, producing, like, some specific type of data. It's throwing that into, like, a Kafka-like, you know, event bus. And now I want to actually have a good partition schema, like, in place, you know, partitioning, like, you know, I want it to be able to do like hash partitioning correctly, right? To get around the hotspots that you were talking about before.

SPEAKER_01And so it's like, therefore it's like, what's the smartest key I can use to be able to ensure that I'm not just doing like potentially like, you know, hash partitioning in a round robin way. But like in a lot of cases, if you're doing say like a clickstream, you want similar data to arrive in the same partition. It can hotspot, but it also depends on like what you know about like the data and the flow of data. So for a lot of like e-commerce type systems, things like that, Like, you know, even for us right now, like if we think about like any of like the, you know, the telemetry of what we're kind of doing in the background right now, like in like the Omnigen system, right? It's like,

SPEAKER_01you know, what's working, what's not working, blah, blah, blah, like all this information. A lot of the sessions. Are kind of bounded. There's an upward bounds of the total number of events that could be produced. But that's also with humans running systems. And so anyways, if we think about this just as how do humans do certain things, you can kind of understand what the behavior could be. Most people aren't going to sit there and have four to eight hour sessions on, say, I don't know, whatever, on Amazon. You're usually going to go in. You're like, I know exactly what I want or something. I'm going to browse for a little bit. I'm going to buy it.

SPEAKER_01I'm going to leave. So you might have 100 events. You're not going to have a million events, like unless you're fake, you're a bot. Bots can get millions of events, right?

SPEAKER_00Yeah.

SPEAKER_01Yeah. I don't know. But yeah. Tell the system how you want to partition, right? It's like, yeah. That's also good advice too, like how to think for the future. Because if a system, like this is always the thing, right? Where it's like you made something and that something is great. And now people are going to put it into production faster than you want it in production. And then you're going to have to support it in production. And that's always the case. You don't have time to kind of go back and be like, let's re-architect. It's like, well, it's running. So I hope you did the right thing.

SPEAKER_00We hope so. We hope so. All right. So we've got some stuff here. It keeps on calling out my faults. No, it's fine. Don't worry about it.

SPEAKER_01No one can get into this unless you're on our VPN. But it's like, cool. So I think this thing worked. Maybe. Cool. All right. So it's running. Let me go and open that up over here. Take a look at it. All right. Cool. That's pretty cool so far. Hold on. Let me just make the screen bigger so we can actually take a look at this. We're going to learn new words we didn't know before.

SPEAKER_01It's going to be great. All right.

SPEAKER_00Cool.

SPEAKER_01So this looks pretty cool to me.

SPEAKER_00It looks really cool.

SPEAKER_01Yeah.

SPEAKER_00For something that you built in 40 minutes.

SPEAKER_01Can you hear this?

SPEAKER_00No.

SPEAKER_01Fuck. OK. Excuse my language. Let's see. I wonder, does StreamYard, is there an option for my, because I'm presenting the screen, but I'm not syncing audio with it, right?

SPEAKER_00Right. That's an excellent question.

SPEAKER_01Yeah.

SPEAKER_00Let's see.

SPEAKER_01But honestly, I'm going to be like, so Opus 5, while I was like, hey, you're not doing a great job, you're not super speedy, this thing works just like GarageBand. Does it really? Okay. And it has sound, which I'm like, this is really cool, too. All right. And it wrote a song for us already.

SPEAKER_00It did.

SPEAKER_01Oh, fuck. Sorry. I like it when I'm excited about stuff. This is super cool. Okay. But no, it's looping. I wish I could. Okay. So this is going to definitely go on GitHub. I'm going to share it so other people can play with it. Add to it, do whatever else you want. But it'd be interesting, I think, given the rise of AI-generated music or synthetic music. It'd be interesting, we talked about lo-fi before. It'd be interesting as an extension or a follow-up to now

SPEAKER_01that we have this wonderful editing environment that we know has an API behind it, it'd be interesting in a follow-up session to have it just go build some background music for us to have these conversations with. What it created right now sounds like the intro to almost like Is it like ghouls and goblins, goblins and ghosts? I can't remember the name of the game, but it sounds like a retro game that's playing on the sequencer. So there's a couple of things that need to be done. I don't know what these things are back here.

SPEAKER_01It's not fitting the screen super well. But the piano roll is great. This is great. All right, so hats off to these machines. I feel like I did a good job, even though we can't hear it. And that's all right. So it says, go to your Chrome.

SPEAKER_00I wonder if you opened it in Chrome?

SPEAKER_01I can try that. Yeah, let me unshare this screen. Then we'll go back to. I'll just open up a new window for the sequencer. We'll share that separately. OK. So some screens let you share audio. OK. So we'll see if this one does it for this one. I mean, because honestly, because we're in StreamYard on Chrome, so we know it shares audio.

SPEAKER_01OK. So this is in. Starting at the beginning, can you hear it?

SPEAKER_00No.

SPEAKER_01You know, that's going to be OK.

SPEAKER_00It'll be OK. So let's see. Oh, also share tab audio.

SPEAKER_01Ooh, yes.

SPEAKER_00When you click on share screen. OK, I'm going to do this again, right? And do just the Chrome tab instead of the whole window. So share screen, Chrome tab, choose the Chrome tab. And at the bottom, it says also share tab audio. And make sure that that's clicked over.

SPEAKER_01Oh, OK. Yes, allow. All right, this is going to be great. Pro tip, be careful sharing it. OK, thank you. All right, well, here we go.

SPEAKER_00Can I hear it?

SPEAKER_01Oh, I can hear it now. I'm not going to get a Grammy or Emmy or whatever, right, for this. Like it worked. Again, 40 minutes. Let's go up to, I like 240 beat per minute. So we're going to go up there. It's going to now seem to be like

SPEAKER_00It's very 1990 video game. You can

SPEAKER_01actually mix it.

SPEAKER_00What happens when you change the square base to something else?

SPEAKER_01Oh, this one?

SPEAKER_00Does it actually give you options?

SPEAKER_01Where do you want to go? Do you want to go to kick? Yeah, let's see. All right. Sounds like more underwater.

SPEAKER_00OK, this is super cool. This is really neat.

SPEAKER_01All right, cool. Job well done. So this is going to get open sourced. Anybody who wants it can use it. I don't know what license to give it because I don't know where the music came from or if it was just literally generated on the fly. So I'm going to take a look at that a little bit more as well. But I'm actually going to use it. I've used some of the digital audio tools before, like the GarageBands and things like that. I feel like with this now I have the ability to be like, wait a second, I got a dumb idea. What if we just added something like this?

SPEAKER_01Because I really do like the multi-pad sequencer. And all it does is it just makes the lights just go through the pads on a given tempo. And as you record, you just go, boom. And it tells you, it's like, I'm coming back again. You go, doom, doom. And you can just do bass drums like this. And it's just simple. You don't have to be a musician to get it, right? Create some dead mouse-like stuff or whatever.

SPEAKER_00That was awesome, man. That worked out way better than I thought. That's fantastic.

SPEAKER_01I guided it. It's funny to see what a single prompt can do nowadays. The reasoning is, I guess, to all the complaints of only having the one model registered, that was great. Opus 5, thank you for being good at music.

SPEAKER_00Good job, Anthropic. You did fantastic today. Thank you.

SPEAKER_01Also, I feel like that only cost me $5. That's pretty good.

SPEAKER_00Yeah. Five bucks, and yeah. And if you already have a subscription, yeah.

SPEAKER_01Oh, yeah. I mean, it's crazy, right? I don't know. I always say it's crazy after everything that works out. When it doesn't work out, it's also fine. It's like, whatever, that's fine.

SPEAKER_00Different kind of crazy.

SPEAKER_01Yeah, it is a little bit crazy. It's like, why didn't it work better? It's like, how do we learn from this? What lessons can we teach ourselves now? But I feel like with some of this stuff, too, it's interesting because I don't think I would have thought of this type of interface. Like, in my head, like, here. I don't want to stand up in front of the camera, so I'm just going to do this for a sec. Hold on.

SPEAKER_00Okay. All right.

SPEAKER_01Anyways, I have, like, this. And, like, you just, like, press the buttons as it goes, right? So it's a sequencer. And so it's, like, that's kind of what I had in my head because that's the last, like, kind of MIDI tool that I was using. And that one does a really cool job because the longer you hold down a key, you get nice sustain and things like that as well. But yeah, the UI account was great and I feel good about that. I feel like it's a good hour spent, right? It was.

SPEAKER_00It was fantastic hours spent. Any time I get to come and hang out with you, it's a good hour spent.

SPEAKER_01We're learning new things, teaching people about touching grass.

SPEAKER_00Yeah, touching grass.

SPEAKER_01With a bunch of friends online.

SPEAKER_00And watch this. I'm going to bring it back around to the dreaming thing, going back to the lessons learned. You have a session. You have lessons learned. Whether it worked out or not, then you trigger the dreaming skill to save your dreaming thing, all the lessons learned.

SPEAKER_01Speaking of, I have it running right now. Here, I just have to share the other screen. Here, do that.

SPEAKER_00I'm going to share this one again.

SPEAKER_01Also, thanks for the sharing tip. I didn't know about the share screen and audio. So I feel like now that I know that, for the next time, if we go back to music or anything else, we can do that. So yeah, so we're using the dreaming skill right now. And now it's basically going through the transcripts. It's creating our agents in Claude, because right now Claude is the only agent that's running in the session. It's also creating the proxy for other agents to be able to get to this knowledge from as well. So Chris, do you want to talk a little bit about the dreaming skill as well?

SPEAKER_01Say what's going on. Yeah.

SPEAKER_00Really quick for those of you that maybe didn't show up last time. Didn't show up, that sounded me. tsk. But we're not here last time. So we built the dreaming skill. Scott had a huge hand in this as well, so I appreciate you, Scott, working on this.

SPEAKER_01Thanks, man.

SPEAKER_00But we created the dreaming skill. Basically, it was born for me a lot from getting into Omnigent when it first launched and wanting to end the fact that Claude Anthropic does have this dream skill that works behind the scenes. But it basically will distill all the learning and Context from your session across your different harnesses into a single dreaming file so

SPEAKER_00that it basically can share that learning and that knowledge across your different harnesses, especially as you're using the amount of harness like Omnijet. That's super high level, but that's my

SPEAKER_01final step. I can finish off by we can narrate the dream as well. I can also create a skill to read it out loud so I don't have to do it. I could do it in my voice. But it's like, so we now have this dream skill over here. And this is basically linked with the agent's MD. The browser MIDI sequencer, piano roll over the timeline with looping, built-in web audio synth, web MIDI, center MIDI file import support, client only, no server database, et cetera. It's all local. So it's like the WASM of editors, if you want to think about it like that.

SPEAKER_01Timing never depends on React for the engine. So these are like the invariants that are easy to break. Note timing never depends on React. So the engine itself writes notes under the audio context clock. That's ahead of time so that everything can be rendered correctly so that it actually sounds good. This is interesting. Playheads derived, not counted. I don't even know what half this shit means. So it's cool. It's good for this, right? It went through and did all the math to do everything else. It's creating that nice timeline to be able to do the loops.

SPEAKER_01Some of these things are some of the most important parts of actually having a good sequence or a good editor at home. Certain things, if you think about an R&B song, a lot of times you might have the same drum beat the entire time. Do it once really well and loop that. Other stuff might come in and out of play as well on a different track. But a lot of that, it's like you have to have a lot of this deep knowledge into what it means to compose music. This now has everything that it needs. It's pretty short.

SPEAKER_00Don't fix these things, which is such an important part when you're working

SPEAKER_01with AI. Just use it. Just do it. This is essentially all the information you need to know about to be able to do something similar again and or for future self to be able to rebuild the context into why it did what it did. With that, that's great.

SPEAKER_00That's awesome. Well, Scott, thanks for being here. We're gonna have to do this again. These are fantastic. I love doing these with you. So thank you all for joining us for Live Data Friday. And for those of you that also wanna hear me talk for another hour, I am gonna be on Shane's podcast here in a minute over on Substack. So Scott, man, thanks. I'll let you get back to it. And we'll see you all again soon.

SPEAKER_00All right. Bye, everyone. Cheers.

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. I Tallied 10 Finance Data Engineer Postings. dbt Showed Up Once.

    Nov 17, 20261 hr 6 minAudio

    On November 4th I ran a free live session with more than 200 people registered. Mostly analysts and folks in data-adjacent roles, with a good handful of career changers working through courses and certs. I talk to...

    Credit the people on this episode

  2. Live Data Friday With Scott Haines

    Nov 1, 202655 minAudio

    Apache Iceberg just shipped a new release, and Iceberg support is now native in open-source Unity Catalog. On this Live Data Friday, I was joined by Scott Haines, Staff Developer Advocate at Databricks, O’Reilly author,...

    Credit the people on this episode

  3. The $180K Maintenance Trap: Fixing the Modern Data Stack with AI

    Sep 24, 202627 minAudio

    In this live session, I'm joined by Srinath Reddy B to dig into what it really costs a business when data breaks. Failed pipelines, broken golden records, and poor data quality don't just slow teams down. They quietly...

    Credit the people on this episode

  4. AURA - Chris Gambill & Shane Gibson

    Aug 15, 202655 minAudio

    The future isn’t “AI instead of data engineering.” It’s AI plus well-structured context, governed logic, versioned rules, and strong feedback loops. That means the teams who win won’t be the ones blindly vibe coding...

    Credit the people on this episode

  5. Live Data Friday!

    Jul 24, 20261 hr 1 minAudio

    Thank you to everyone who tuned into my live video! Join me for my next live video in the app. Get full access to Chris Gambill at gambilldataengineering.substack.com/subscribe

    Credit the people on this episode