Hacker News (curated)
new
|
past
|
comments
|
ask
|
show
|
jobs
| show hidden
charles_f
4 hours ago
|
parent
|
context
Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
|
github