Hacker News (curated)new | past | comments | ask | show | jobs| show hidden

Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact | github