[tor-bugs] #29565 [Obfuscation/Snowflake]: Fix broker robots.txt to disallow crawling
Tor Bug Tracker & Wiki
blackhole at torproject.org
Sat Feb 23 08:11:17 UTC 2019
#29565: Fix broker robots.txt to disallow crawling
---------------------------------------+--------------------
Reporter: dcf | Owner: (none)
Type: defect | Status: new
Priority: Medium | Milestone:
Component: Obfuscation/Snowflake | Version:
Severity: Normal | Keywords: easy
Actual Points: | Parent ID:
Points: | Reviewer:
Sponsor: |
---------------------------------------+--------------------
From comment:11:ticket:28848 and https://github.com/ahf/snowflake-
notes/blob/fb4304a7df08c6ddeeb103f38fc9103721a20cd9/Broker.markdown#the-
robotstxt-handler:
> - Was the question about crawling ever answered? I can't think of a very
good reason not to allow it. Even if censors were crawling the web for
Snowflake brokers, they could get this information much more easily just
from the source code.
I believe the intention behind the robots.txt handler is to prevent search
engines from indexing any pages on the site, because there's no permanent
information there, not for any security or anti-enumeration reason.
ahf points out that the current robots.txt achieves the opposite: it
allows crawling of all pages by anyone. Instead of
{{{
User-agent: *
Disallow:
}}}
it should be
{{{
User-agent: *
Disallow: /
}}}
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29565>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list