Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Code & Curiosity by Maximilian Schwarzmüller

Code & Curiosity by Maximilian Schwarzmüller artwork

APIs are locking up - thanks to Gen AI

Nov 16, 202512 minAudio
  • Apple Podcasts
ShareXLinkedInFacebookWhatsAppEmail

There's a disturbing "trend" (kind of): Some APIs are locking up or charging (quite a bit of) money for usage. X (Twitter) is one example. Reddit is another recent one. And they won't be the last ones to do that.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

There's kind of a trend, if you wanna call it like this, or definitely something we see become more common over the last one or two years, which is not really surprising, but nonetheless, something I wanna talk about. The trend of APIs of websites or web services becoming more and more locked down or private or more expensive to use, whatever you wanna call it. And the most recent example, which is the reason why I'm creating this video

here, is the Reddit API, because, uh, two days ago, there's been a post by the, the Reddit team in the Reddit development forum where they essentially announced that the usage of their API now needs approval. So there is an approval process for using the API for responsible use to support responsible builders, and I'll get back to that and what that means, but it's kind of in line, if you wanna call it like this, to what

Twitter, X, did two years ago already. They made their API really expensive to use, at least at scale. So if you wanna interact with Twitter, with X, programmatically, if you wanna build yet another social media scheduling tool and you wanna support X, well, that could get expensive depending on how you build it, uh, because the, the free usage is quite limited. You can, for example, read

100 posts per month and write 500 posts per month, which might be more than enough for your own little tool that you're building for yourself, but if you are building a SaaS product on top of the X API, that will not suffice, so you'll have to pay, but not even the basic tier might be case. It might be, it might not be. And the pro tier might still not be enough. Now chances are, it may be enough, but it

is also, uh, quite expensive. And now for Reddit, there, as I mentioned, i- it's not about paying or about, uh, the price they're asking, but it's about an approval process so that not every application can start using their API. And the question of course is, why are companies doing that? Well, there are a couple of reasons and one big important reason. Obviously, you could say, why would they

not do it? Why would they give you access to their free? And you could argue, well, because in the past, before AI, they may have benefited from doing so. Because if people can build products on top of, let's say, X, if I can build a social media scheduling application, that might be in Twitter's and X's interest because more posts on X could mean more engagement, uh, more people reading and

interacting with those posts, so that might not be too bad. And there is a reason why you can write more than you can read. You could think that you should be able to read more than write because writes are more expensive to their database, to their infrastructure it's the opposite. They allow you to write more than they read. And just as a side note, X also has a new program which they're testing, it's a pilot right now, uh, where they, um, want to give

you, uh, pay per use access to their API. But it stays the same. You have to pay to use it and it can get expensive. Now, why are companies doing that? Well, the big answer, of course, is AI, or specifically, of course, gen AI. Because with the rise of gen AI, it has become clear that all that data which these companies own, all these Reddit posts, all the posts on X, that is a valuable resource because

those gen AI models, of course, need data in their training or for their training process. They, data is the most important thing there because as we all know, ChatGPT or the GPT models were trained essentially on the entire data, publicly available data, you could find on the internet. Um, and still, these models will need vast amounts of data for their training. Nowadays, of course, there is the entire concept or

idea of using synthetic data as well as real data for the training process, and to my understanding, that seems to work quite well, though we'll see if that maybe still is a problem and there is like a, a ceiling due to the limited data that's available because the entire data in the internet has already been consumed, so now you're just generating more from that knowledge that was gathered from that internet data, so there might be a ceiling there. It's not entirely clear yet. Um, but anyways,

data is super important and of course there's still new data being Now more data than ever is generated by AI though, to be fair, so that is synthetic data in the end, lots of data is being generated, uh, including some data by humans on X and Reddit every day, and of course, those platforms don't want to give away that data for free anymore. They did in the past because we didn't see coming with, uh, large language models and, um,

now of course they want to protect their data because a site like X of course sits on lots of data, lots of valuable posts, at least to some degree, let's be honest.Most of the posts are total BS, but at least some decent posts there and definitely valuable in the sense of being valuable for training. And those sites don't wanna give that data away for free anymore, which is why they're locking it down. There also is a reason why we

see more and more web scraping businesses, uh, coming up almost every day because now with large language models, even if we ignore the training part, many of the applications that we wanna build with help of large language models or on top of large language models will need access to recent data. If you're building a smart chatbot and you're using OpenAI's models under the hood, you probably wanna pull in some recent chatbot more useful. You wanna add web search, you

wanna be able to have your chatbot answer questions related to the most recent posts on X. So you wanna pull that data into your application and then reach your chat history and the prompts you sent to the large language model with that data that then hopefully allows the model to generate a better answer, uh, for what the user asked. And that's why these sites are kind of locking down their APIs to make it harder to get

access to the data because in the past they gave it away for free. They don't wanna do that anymore. Obviously, there still are ways to get that data, as I mentioned. There's a plethora of web crawling companies and not all of these companies, uh, respect the fact that certain sites don't want to get crawled. Now I did actually a livestream on the topic of building our own web crawler, um, a while back, and I'll, uh, provide a link to that

episode. You can watch the full, uh, livestream episode, uh, below this episode, of course. So you can build your own crawler and I did that with Crawl4AI. And essentially what you're building there is, um, an application that spins up a browser and simulates being a user and visiting a website to then extract that website content, to extract the rendered HTML content and so on. Um, that is how you can build and use a crawler in the livestream.

I just, um, built it to crawl my own website, that clear. So I, uh, did not start crawling X there because web scraping and crawling is kind of a gray zone. And, uh, there are many sites that clearly state in their terms that they do not allow web crawling. So you are violating those terms if you do, which is why sites like Firecrawl, for example, won't crawl X links. If I

take an X link and I try to scrape that, uh, I'll get an error that this is not supported. Or actually here it doesn't even start as it seems in the past, I did get an error. So there are sites that don't allow that. There probably also are sites that do, and you can definitely build your own crawler that doesn't give anything about anything and extract any content of any site you wanna extract. Now I will say, of course, that many

sites also try to implement some technical hurdles that make it harder to crawl them, but in the end if you really want to, you can get around pretty much all of them. It might not be legal, it might be violating their terms, but it is possible. Because, and that takes us back to the beginning, to the main topic, because of course all that data is super valuable. However, this does have I believe a real downside or a, an implication that's not great for us as

developers. Because I totally get that these sites don't wanna give away access to their data, and just as a side note, it's kind of not their data. It's the data of the users using the site, but that's a whole different story. But I get that they don't wanna give away access to this data. The problem of course is that kind of as a, a, an additional casualty, we as developers are limited in what we can build, uh, with those APIs. Sure, you might get approval for the Reddit API

something they're happy with. But of course you also might not get that approval. And for X you have to pay quite a bit of money depending on what you're building, even if what you're building has nothing extracting that data and using it for model training or anything like that. It limits the amount of useful stuff we can build on top of other services and sites. And that of course in turn, um,

might also hurt those sites because some useful products from which they might benefit then maybe won't get built. But of course I guess that is a price they're happy to pay because either they'll get paid by these people that build products on top of them, or at least they prevent that data extraction. So yeah, uh, I expect that we will see more sites and services, uh, locking down their APIs. I think we'll see more sites becoming pretty

protective about their data, pretty aggressive against crawlers, which is absolutely their right, there. Um, but I think that might also hurt us as developer because it kind of limits the, the stuff we can build around other popular services and sites. These are my two cents. What do you think about this topic?

Open the transcript file

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. We will have to accept AI-generated content

    Oct 15, 202612 minAudio

    I think we'll have to get used to (and accept) AI generated content. Not to slop though.

    • Apple Podcasts

    Credit the people on this episode

  2. Is software dead?

    Oct 13, 202612 minAudio

    There's a lot of talk about software being dead. Unsurprisingly, it's complicated.

    • Apple Podcasts

    Credit the people on this episode

  3. GPT 6 Astra - Good Beyond 3D?

    Sep 9, 20268 minAudio

    GPT 6 Astra was released. It's a good model, I have to say. Especially for 3D work it seems to be a real game changer. But due to all the hype, you may be disappointed...

    • Apple Podcasts

    Credit the people on this episode

  4. You may ignore Omarchy but not its impact

    Aug 26, 20268 minAudio

    Omarchy got a lot of hype and visibility recently. But even if you don't care or don't use it, you shouldn't ignore it.

    • Apple Podcasts

    Credit the people on this episode

  5. SvelteKit 3, Solid 2 - Do they matter anymore?

    Aug 18, 202614 minAudio

    We're not going back to the JS framework wars. BUT new frameworks and framework versions are important. For a different reason now!

    • Apple Podcasts

    Credit the people on this episode