I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
Are the AI companies worried about anything but their valuations?
I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need to fund any risk mitigations, security coordination, and other safety work. If these things are not imposed on the market, the most ruthless player will create the biggest mess they can get away with.
I don’t see a better way to do this, right now, than to allow Strips, or CF, to create some way to charge the crawlers. Imagine I have some really expensive content, I can put it behind the paywall, or subscription model, but that’s very specific to the human consumer. What if I do want to search engine or content aggregator to consume some of the content? It would be quite hard for me to sign an exclusive agreement with OpenAI, like what Reddit or NYT are doing. It might be quite beneficial to create some infrastructure to let bots automatically to pay for access. It might even be crypto based, I don’t think it really matters. It just has to be easy for me to integrate it, and the bots need to find it easy to use.
If we all start doing this, I think this could be of benefit to all content creators, such as musicians, movie makers, game developers, and even independent journalists.
We used to pay for software and things, but what has happened? Walled gardens and production monopolies are taking over. This has largely reduced the quality of all the content we consume.
I think that to prove I am not a crawler I probably just need to supply a Safari user agent string. The crawlers on my sites actually self identify for the most part. The malicious ones are banned when they try to post spam.
Because I also use AI, and I actually want the content from the site to show up for me, as well as other AI users.
More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.
If these were actual legitimate bots using a known user-agent and coming from the same subnet so you could throttle their requests it would be fine.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
All anti-bot measures are inherently flawed, and if you overdo it, your false positives cause real harm by excluding actual people from using your services.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
HoustonArchitecture.com, a site that posts about new development projects in the Houston area, had the exact same thing happen to them. (They also got hacked; don't remember the details.) They now pay the Cloudflare tax.
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
Yes. Any number of methods work. From requiring a cookie to requiring CSS to be loaded to a PoW to a JavaScript redirect. Most scraper bots are really dumb. If you do it cookie based, you can also put a poison link on every page that invalidates their cookie and starts returning poison on every page after that, so they're damned if they do and damned if they don't.
I have tried cookie based sessions and I have tried multiple things. I have been battling DDOS and script kiddies longer than CF has been a thing. You’re wrong in that you think I could continue to maintain the site without CF in front of it, I would just shut it down. It’s not a profitable forum, it’s just a hobby.
If I recall by browsing the docs, there's a lot of fine grained client control over the blocking params. Which makes sense since the main product of cf is bot blocking.
So you can definitely block by IP, by fingerprint, both or neither
Cloudflare is also great at playing both sides, and they're trying pretty hard to push for pay-to-crawl because they'll probably get a 30% cut along the way.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.