It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.
turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set
