On Friday 15 April 2011 07:12:53 Mike Perry wrote:
What does using the Tor network have to do with the ethics of crawling the web/Internet? What makes it not OK to crawl the Internet anonymously, but makes it acceptable to seek that same information so long as you are not anonymous? Or are we being Kantian here, and saying that if everyone crawled the Internet, we'd be doomed. So therefore, only Google can crawl the Internet? That doesn't work either.
Google and various others crawl the Internet to make a searchable index of information. Mallory crawls the Internet to look for open formmail scripts, misconfigured FTP servers, and the like. What Mallory is looking for is likely to be found on a computer that doesn't have a lot of pointers to it, so he searches IP address space for open ports. Most of his inquiries result in "no such host" or "port closed". Google follows links, so it doesn't make many inquiries to closed ports.
cmeclax