Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...
There is a section in the article answering your question if you read it.
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.
I’d love a service like spamcop.net where I could submit my access_log and they lookup the abuse addresses and file abuse reports in my name. Maybe if people’s Internet access gets suspended they’ll think about installing random apps that work as a proxy in the background.
That is a ridiculous way to try and deal with the problem of residential proxies.
You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".
If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.
You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.
Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."
Similar to how people running an open SMTP are complicit in promoting spam, I see people running a wild public proxy as complicit in this malicious scraping activity.
And similar to how most people running mail daemons are using blackhole lists nowadays and are keen to not end up on there, maybe ISPs and web hosters can use the AbuseIPDB to sort out their customers.
Just doing nothing doesn't appear to stop the scans hammering my poor Raspberry Pi serving my few Git repositories.
