[tor-talk] Hidden Services

[grarpamp]

> People use robots.txt to indicate that they don't want their site to
> be added to indexes.

They use it to indicate that they don't want their site to be crawled.
Tor2Web isn't crawling anything, thus they have no need or obligation
to fetch and consider anyone's robots in the first place.

Nobody in their right mind is going to crawl and index 5 sites and then
ask all 100 sites linked to from those pages for their robots.txt before
listing those 100 links. That's not how things are done on the net.
Depending on your vantage point, crawling the subject site isn't
necessarily required to index it.

And if a site is so concerned about someone else publishing a link,
however obtained, then they should name it something innocent and
password protect it or use better operational security to begin with.