Item

hahnbee · 2026-08-30T23:23:45.000Z

what constitutes as a bad crawler and what information do you have that's attracting so many bots?

charles_f · 2026-08-30T23:38:11.000Z

Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers