There's a disturbing "trend" (kind of): Some APIs are locking up or charging (quite a bit of) money for usage. X (Twitter) is one example. Reddit is another recent one. And they won't be the last ones to do that.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
There's kind of a trend, if you wanna call it like this, or definitely something we see become more common over the last one or two years, which is not really surprising, but nonetheless, something I wanna talk about. The trend of APIs of websites or web services becoming more and more locked down or private or more expensive to use, whatever you wanna call it. And the most recent example, which is the reason why I'm creating this video
here, is the Reddit API, because, uh, two days ago, there's been a post by the, the Reddit team in the Reddit development forum where they essentially announced that the usage of their API now needs approval. So there is an approval process for using the API for responsible use to support responsible builders, and I'll get back to that and what that means, but it's kind of in line, if you wanna call it like this, to what
Twitter, X, did two years ago already. They made their API really expensive to use, at least at scale. So if you wanna interact with Twitter, with X, programmatically, if you wanna build yet another social media scheduling tool and you wanna support X, well, that could get expensive depending on how you build it, uh, because the, the free usage is quite limited. You can, for example, read
100 posts per month and write 500 posts per month, which might be more than enough for your own little tool that you're building for yourself, but if you are building a SaaS product on top of the X API, that will not suffice, so you'll have to pay, but not even the basic tier might be case. It might be, it might not be. And the pro tier might still not be enough. Now chances are, it may be enough, but it
is also, uh, quite expensive. And now for Reddit, there, as I mentioned, i- it's not about paying or about, uh, the price they're asking, but it's about an approval process so that not every application can start using their API. And the question of course is, why are companies doing that? Well, there are a couple of reasons and one big important reason. Obviously, you could say, why would they
not do it? Why would they give you access to their free? And you could argue, well, because in the past, before AI, they may have benefited from doing so. Because if people can build products on top of, let's say, X, if I can build a social media scheduling application, that might be in Twitter's and X's interest because more posts on X could mean more engagement, uh, more people reading and
interacting with those posts, so that might not be too bad. And there is a reason why you can write more than you can read. You could think that you should be able to read more than write because writes are more expensive to their database, to their infrastructure it's the opposite. They allow you to write more than they read. And just as a side note, X also has a new program which they're testing, it's a pilot right now, uh, where they, um, want to give
you, uh, pay per use access to their API. But it stays the same. You have to pay to use it and it can get expensive. Now, why are companies doing that? Well, the big answer, of course, is AI, or specifically, of course, gen AI. Because with the rise of gen AI, it has become clear that all that data which these companies own, all these Reddit posts, all the posts on X, that is a valuable resource because
those gen AI models, of course, need data in their training or for their training process. They, data is the most important thing there because as we all know, ChatGPT or the GPT models were trained essentially on the entire data, publicly available data, you could find on the internet. Um, and still, these models will need vast amounts of data for their training. Nowadays, of course, there is the entire concept or
idea of using synthetic data as well as real data for the training process, and to my understanding, that seems to work quite well, though we'll see if that maybe still is a problem and there is like a, a ceiling due to the limited data that's available because the entire data in the internet has already been consumed, so now you're just generating more from that knowledge that was gathered from that internet data, so there might be a ceiling there. It's not entirely clear yet. Um, but anyways,
data is super important and of course there's still new data being Now more data than ever is generated by AI though, to be fair, so that is synthetic data in the end, lots of data is being generated, uh, including some data by humans on X and Reddit every day, and of course, those platforms don't want to give away that data for free anymore. They did in the past because we didn't see coming with, uh, large language models and, um,
now of course they want to protect their data because a site like X of course sits on lots of data, lots of valuable posts, at least to some degree, let's be honest.Most of the posts are total BS, but at least some decent posts there and definitely valuable in the sense of being valuable for training. And those sites don't wanna give that data away for free anymore, which is why they're locking it down. There also is a reason why we
see more and more web scraping businesses, uh, coming up almost every day because now with large language models, even if we ignore the training part, many of the applications that we wanna build with help of large language models or on top of large language models will need access to recent data. If you're building a smart chatbot and you're using OpenAI's models under the hood, you probably wanna pull in some recent chatbot more useful. You wanna add web search, you
wanna be able to have your chatbot answer questions related to the most recent posts on X. So you wanna pull that data into your application and then reach your chat history and the prompts you sent to the large language model with that data that then hopefully allows the model to generate a better answer, uh, for what the user asked. And that's why these sites are kind of locking down their APIs to make it harder to get
access to the data because in the past they gave it away for free. They don't wanna do that anymore. Obviously, there still are ways to get that data, as I mentioned. There's a plethora of web crawling companies and not all of these companies, uh, respect the fact that certain sites don't want to get crawled. Now I did actually a livestream on the topic of building our own web crawler, um, a while back, and I'll, uh, provide a link to that
episode. You can watch the full, uh, livestream episode, uh, below this episode, of course. So you can build your own crawler and I did that with Crawl4AI. And essentially what you're building there is, um, an application that spins up a browser and simulates being a user and visiting a website to then extract that website content, to extract the rendered HTML content and so on. Um, that is how you can build and use a crawler in the livestream.
I just, um, built it to crawl my own website, that clear. So I, uh, did not start crawling X there because web scraping and crawling is kind of a gray zone. And, uh, there are many sites that clearly state in their terms that they do not allow web crawling. So you are violating those terms if you do, which is why sites like Firecrawl, for example, won't crawl X links. If I
take an X link and I try to scrape that, uh, I'll get an error that this is not supported. Or actually here it doesn't even start as it seems in the past, I did get an error. So there are sites that don't allow that. There probably also are sites that do, and you can definitely build your own crawler that doesn't give anything about anything and extract any content of any site you wanna extract. Now I will say, of course, that many
sites also try to implement some technical hurdles that make it harder to crawl them, but in the end if you really want to, you can get around pretty much all of them. It might not be legal, it might be violating their terms, but it is possible. Because, and that takes us back to the beginning, to the main topic, because of course all that data is super valuable. However, this does have I believe a real downside or a, an implication that's not great for us as
developers. Because I totally get that these sites don't wanna give away access to their data, and just as a side note, it's kind of not their data. It's the data of the users using the site, but that's a whole different story. But I get that they don't wanna give away access to this data. The problem of course is that kind of as a, a, an additional casualty, we as developers are limited in what we can build, uh, with those APIs. Sure, you might get approval for the Reddit API
something they're happy with. But of course you also might not get that approval. And for X you have to pay quite a bit of money depending on what you're building, even if what you're building has nothing extracting that data and using it for model training or anything like that. It limits the amount of useful stuff we can build on top of other services and sites. And that of course in turn, um,
might also hurt those sites because some useful products from which they might benefit then maybe won't get built. But of course I guess that is a price they're happy to pay because either they'll get paid by these people that build products on top of them, or at least they prevent that data extraction. So yeah, uh, I expect that we will see more sites and services, uh, locking down their APIs. I think we'll see more sites becoming pretty
protective about their data, pretty aggressive against crawlers, which is absolutely their right, there. Um, but I think that might also hurt us as developer because it kind of limits the, the stuff we can build around other popular services and sites. These are my two cents. What do you think about this topic?
GPT 6 Astra was released. It's a good model, I have to say. Especially for 3D work it seems to be a real game changer. But due to all the hype, you may be disappointed...