I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..
They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.
Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
That's bad for static content.
Try adding search, with pagination and 16 filters that can toggled on off. And 1000 tags, give each page 5.
I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..
They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.
Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.