so true. the article authors wishing crawlers will use git instead is so funny because the crawlers don't care at all. they are scrapping everything with brute force. they don't care about your content or effective alternatives, and one more site driving their real users crazy with Anubis is nothing more than a new blip in their dashboard. the crawler operators will not even look at the url.