August 2019 - tor-commits - lists.torproject.org

[bridgedb/master] Merge branch 'feature/9316' into develop
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit 46dde4ce5842a5bf73f16c2c4de451bb5c21dcc4 Merge: 081ff36 5cde59d Author: Philipp Winter <phw(a)nymity.ch> Date: Thu Aug 15 14:05:05 2019 -0700 Merge branch 'feature/9316' into develop CHANGELOG | 11 +- bridgedb.conf | 16 +- bridgedb/distributors/email/autoresponder.py | 16 + bridgedb/distributors/https/server.py | 11 + bridgedb/distributors/moat/server.py | 15 + bridgedb/main.py | 55 ++-- bridgedb/metrics.py | 461 +++++++++++++++++++++++++++ bridgedb/proxy.py | 15 +- bridgedb/test/test_metrics.py | 204 ++++++++++++ scripts/get-tor-exits | 2 +- 10 files changed, 775 insertions(+), 31 deletions(-)

1 0

[bridgedb/master] Update comment.
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit 85a69d1be3f14a8ba0c5ba8e4840e40214217941 Author: Philipp Winter <phw(a)nymity.ch> Date: Mon Aug 12 13:57:43 2019 -0700 Update comment. Several variables that are referenced in the comment no longer exist. --- bridgedb/main.py | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-) diff --git a/bridgedb/main.py b/bridgedb/main.py index 1a617b2..4d1d38a 100644 --- a/bridgedb/main.py +++ b/bridgedb/main.py @@ -368,10 +368,9 @@ def run(options, reactor=reactor): State should be saved before calling this method, and will be saved again at the end of it. - The internal variables, ``cfg``, ``hashring``, ``proxyList``, - ``ipDistributor``, and ``emailDistributor`` are all taken from a - :class:`~bridgedb.persistent.State` instance, which has been saved to - a statefile with :meth:`bridgedb.persistent.State.save`. + The internal variables ``cfg`` and ``hashring`` are taken from a + :class:`~bridgedb.persistent.State` instance, which has been saved to a + statefile with :meth:`bridgedb.persistent.State.save`. :type cfg: :class:`Conf` :ivar cfg: The current configuration, including any in-memory @@ -380,18 +379,6 @@ def run(options, reactor=reactor): :type hashring: A :class:`~bridgedb.Bridges.BridgeSplitter` :ivar hashring: A class which takes an HMAC key and splits bridges into their hashring assignments. - :type proxyList: :class:`~bridgedb.proxy.ProxySet` - :ivar proxyList: The container for the IP addresses of any currently - known open proxies. - :ivar ipDistributor: A - :class:`~bridgedb.distributors.https.distributor.HTTPSDistributor`. - :ivar emailDistributor: A - :class:`~bridgedb.distributors.email.distributor.EmailDistributor`. - :ivar dict tasks: A dictionary of ``{name: task}``, where name is a - string to associate with the ``task``, and ``task`` is some - scheduled event, repetitive or otherwise, for the :class:`reactor - <twisted.internet.epollreactor.EPollReactor>`. See the classes - within the :api:`twisted.internet.tasks` module. """ logging.debug("Caught SIGHUP") logging.info("Reloading...")

1 0

[bridgedb/master] Support handing out decoy bridges to bots.
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit 7ceb25e306a5af456c4a4ba1f1f5b2a72d6eb77c Author: Philipp Winter <phw(a)nymity.ch> Date: Wed Aug 14 15:00:59 2019 -0700 Support handing out decoy bridges to bots. This patch makes it possible to identify bots by inspecting HTTP request headers. A CSV file, specified by BLACKLISTED_REQUEST_HEADERS_FILE, contains mappings from request header to a regular expression of the header's value, e.g.: Accept-Language,[Kk]lingon User-Agent,Spa+ce ... Once a regular expression matches a client's request, we probably caught a bot. This patch also makes it possible to respond to bot requests with a decoy bridge, e.g., to study what the owners of the bot intend to do with the bridge. Decoy bridges are configured in the CSV file DECOY_BRIDGES_FILE. The file maps a transport type and its IP address version to bridge lines, e.g.: vanillav4,1.2.3.4:1234 FINGERPRINT obfs4v4,obfs4 1.2.3.4:1234 FINGERPRINT ARGS ... This fixes <https://bugs.torproject.org/31252> --- CHANGELOG | 9 +++ bridgedb.conf | 19 ++++++ bridgedb/antibot.py | 123 ++++++++++++++++++++++++++++++++++ bridgedb/distributors/https/server.py | 10 +++ bridgedb/distributors/moat/server.py | 6 ++ bridgedb/main.py | 6 ++ bridgedb/test/test_antibot.py | 108 +++++++++++++++++++++++++++++ 7 files changed, 281 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 32e6fe5..03390d6 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -16,6 +16,15 @@ Changes in version 0.8.0 - YYYY-MM-DD Use stem instead of leekspin to create test descriptors. We now don't need to depend on leekspin anymore. + * FIXES #31252 https://bugs.torproject.org/31252 + Add an anti-bot mechanism that allows us to detect bots by matching + HTTP request headers for blacklisted patterns. For example, bots may + have their Accept-Language set to "Klingon". Blacklisted patterns are + configured in BLACKLISTED_REQUEST_HEADERS_FILE. When BridgeDB detects + a bot request, we can answer their request with a decoy bridge that's + only handed out to bots. Decoy bridges are configured in + DECOY_BRIDGES_FILE. + Changes in version 0.7.1 - 2019-06-07 * FIXES #28496 https://bugs.torproject.org/28496 diff --git a/bridgedb.conf b/bridgedb.conf index ba43bb6..a0e00a8 100644 --- a/bridgedb.conf +++ b/bridgedb.conf @@ -301,6 +301,25 @@ PROBING_RESISTANT_TRANSPORTS = ['scramblesuit', 'obfs4'] # menu). DEFAULT_TRANSPORT = 'obfs4' +# HTTP headers that suggest that a request was issued by a bot. The CSV +# file must have the following format: +# <HEADER>,<REGEXP> +# ... +# For example: +# Accept-Language,[Kk]lingon +BLACKLISTED_REQUEST_HEADERS_FILE="blacklisted-request-headers.csv" + +# Decoy bridges that we are handing out to bots that we detected using the +# regular expressions in BLACKLISTED_REQUEST_HEADERS_FILE. The CSV file must +# have the following format: +# <TRANSPORT>v<IP_VERSION>,<BRIDGE_LINE> +# ... +# For example: +# vanillav4,1.2.3.4:1234 0123456789ABCDEF0123456789ABCDEF01234567 +# vanillav6,[::1]:1234 0123456789ABCDEF0123456789ABCDEF01234567 +# obfs4v4,obfs4 1.2.3.4:1234 public-key=... node-id=... iat-mode=... +DECOY_BRIDGES_FILE="decoy-bridges.csv" + #------------------------------- # Moat Distribution Options \ #------------------------------------------------------------------------------ diff --git a/bridgedb/antibot.py b/bridgedb/antibot.py new file mode 100644 index 0000000..e724c68 --- /dev/null +++ b/bridgedb/antibot.py @@ -0,0 +1,123 @@ +# -*- coding: utf-8 ; test-case-name: bridgedb.test.test_metrics ; -*- +# _____________________________________________________________________________ +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: please see included AUTHORS file +# :copyright: (c) 2019, The Tor Project, Inc. +# (c) 2019, Philipp Winter +# :license: see LICENSE for licensing information +# _____________________________________________________________________________ + +"""Functions for dealing with bot requests.""" + +import re +import logging + +# Maps transport types and IP version (e.g., "obfs4v4", "vanillav4", or +# "vanillav6") to bridge lines (e.g., "1.2.3.4:1234 ...". +DECOY_BRIDGES = {} + +# Maps HTTP request headers (e.g., "Accept-Language") to regular expressions +# that suggest that the request was issued by a bot (e.g., "[Kk]lingon"). +BLACKLISTED_REQUEST_HEADERS = {} + + +def _loadCSV(filename): + """Load and return the content of the given CSV file. + + :param str filename: The filename to read. + :rtype: dict + :returns: A dictionary mapping keys (first column) to values (second + column). + """ + + csv = dict() + try: + with open(filename) as fh: + for line in fh.readlines(): + if line.count(",") != 1: + logging.warning("Line must have exactly one comma: %s" % + line) + continue + key, value = line.split(",") + csv[key.strip()] = value.strip() + except IOError as err: + logging.warning("I/O error while reading from file %s: %s" % + (filename, err)) + + return csv + + +def loadBlacklistedRequestHeaders(filename): + """Load and globally set a dictionary of blacklisted request headers. + + :param str filename: The filename to read. + """ + + content = _loadCSV(filename) + blacklisted = dict() + # Turn dictionary values into compiled regular expressions. + for header, regexp in content.items(): + try: + blacklisted[header] = re.compile(regexp) + except Exception as err: + logging.warning("Skipping regexp %s because we couldn't compile " + "it: %s" % (regexp, err)) + + global BLACKLISTED_REQUEST_HEADERS + BLACKLISTED_REQUEST_HEADERS = blacklisted + + +def loadDecoyBridges(filename): + """Load and globally set a dictionary of decoy bridges. + + :param str filename: The filename to read. + """ + + d = _loadCSV(filename) + # Turn our bridge lines (which are strings) into lists. + decoyBridges = {ttype: [line] for ttype, line in d.items()} + + global DECOY_BRIDGES + DECOY_BRIDGES = decoyBridges + + +def getDecoyBridge(transport, ipVersion): + """Return a decoy bridge or, if none is available, None. + + :param str transport: The desired transport, e.g., "vanilla" or "obfs4". + :param int ipVersion: The IP version, which must be either 4 or 6. + :rtype: list + :returns: Return a list of bridge lines or, if we don't have any, None. + """ + + if ipVersion not in [4, 6]: + return None + + logging.info("Returning IPv%d decoy bridge for transport %s." % + (ipVersion, transport)) + return DECOY_BRIDGES.get("%sv%d" % (transport, ipVersion), None) + + +def isRequestFromBot(request): + """Determine if the given request is coming from a bot. + + :type request: :api:`twisted.web.http.Request` + :param request: A ``Request`` object, including POST arguments which + should include two key/value pairs. + :rtype: bool + :returns: True if the request is coming from a bot and False otherwise. + """ + + for header, badRegexp in BLACKLISTED_REQUEST_HEADERS.items(): + value = request.getHeader(header) + if value is None: + continue + + if badRegexp.search(value) is not None: + logging.info("Found bot request. Headers: %s" % + request.requestHeaders) + return True + + return False diff --git a/bridgedb/distributors/https/server.py b/bridgedb/distributors/https/server.py index 732f8bf..e5df7da 100644 --- a/bridgedb/distributors/https/server.py +++ b/bridgedb/distributors/https/server.py @@ -53,6 +53,7 @@ from bridgedb import strings from bridgedb import translations from bridgedb import txrecaptcha from bridgedb import metrics +from bridgedb import antibot from bridgedb.distributors.common.http import setFQDN from bridgedb.distributors.common.http import getFQDN from bridgedb.distributors.common.http import getClientIP @@ -916,6 +917,15 @@ class BridgesResource(CustomErrorHandlingResource, CSPResource): bridgeLines = [replaceControlChars(bridge.getBridgeLine( bridgeRequest, self.includeFingerprints)) for bridge in bridges] + if antibot.isRequestFromBot(request): + transports = bridgeRequest.transports + # Return either a decoy bridge or no bridge. + if len(transports) > 2: + logging.warning("More than one transport requested") + return self.renderAnswer(request) + ttype = "vanilla" if len(transports) == 0 else transports[0] + return self.renderAnswer(request, antibot.getDecoyBridge(ttype, bridgeRequest.ipVersion)) + return self.renderAnswer(request, bridgeLines) def getResponseFormat(self, request): diff --git a/bridgedb/distributors/moat/server.py b/bridgedb/distributors/moat/server.py index 73d2423..10096e7 100644 --- a/bridgedb/distributors/moat/server.py +++ b/bridgedb/distributors/moat/server.py @@ -41,6 +41,7 @@ from twisted.web.server import Site from bridgedb import metrics from bridgedb import captcha from bridgedb import crypto +from bridgedb import antibot from bridgedb.distributors.common.http import setFQDN from bridgedb.distributors.common.http import getFQDN from bridgedb.distributors.common.http import getClientIP @@ -735,6 +736,11 @@ class CaptchaCheckResource(CaptchaResource): logging.warn(("Not enough bridges of the type specified to " "fulfill the following request: %s") % bridgeRequest) + if antibot.isRequestFromBot(request): + ttype = transport or "vanilla" + bridgeLines = antibot.getDecoyBridge(ttype, + bridgeRequest.ipVersion) + # If we have no bridges at all to give to the client, then # return a JSON API 404 error. if not bridgeLines: diff --git a/bridgedb/main.py b/bridgedb/main.py index 5d9b0c6..94f4921 100644 --- a/bridgedb/main.py +++ b/bridgedb/main.py @@ -26,6 +26,7 @@ from bridgedb import proxy from bridgedb import runner from bridgedb import util from bridgedb import metrics +from bridgedb import antibot from bridgedb.bridges import MalformedBridgeInfo from bridgedb.bridges import MissingServerDescriptorDigest from bridgedb.bridges import ServerDescriptorDigestMismatch @@ -417,6 +418,11 @@ def run(options, reactor=reactor): proxy.loadProxiesFromFile(proxyfile, proxies, removeStale=True) metrics.setProxies(proxies) + logging.info("Reloading blacklisted request headers...") + antibot.loadBlacklistedRequestHeaders(config.BLACKLISTED_REQUEST_HEADERS_FILE) + logging.info("Reloading decoy bridges...") + antibot.loadDecoyBridges(config.DECOY_BRIDGES_FILE) + logging.info("Reparsing bridge descriptors...") (hashring, emailDistributorTmp, diff --git a/bridgedb/test/test_antibot.py b/bridgedb/test/test_antibot.py new file mode 100644 index 0000000..1cda86a --- /dev/null +++ b/bridgedb/test/test_antibot.py @@ -0,0 +1,108 @@ +# -*- coding: utf-8 ; test-case-name: bridgedb.test.test_metrics ; -*- +# _____________________________________________________________________________ +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: please see included AUTHORS file +# :copyright: (c) 2019, The Tor Project, Inc. +# (c) 2019, Philipp Winter +# :license: see LICENSE for licensing information +# _____________________________________________________________________________ + +"""Tests for :mod:`bridgedb.antibot`.""" + +import os +import tempfile + +from twisted.trial import unittest +from twisted.web.test.requesthelper import DummyRequest + +from bridgedb import antibot + + +class AntiBot(unittest.TestCase): + """Unittests for :mod:`bridgedb.antibot`.""" + + def write_file(self, content): + """ + Write the given content to a temporary file. + + We're responsible for deleting the file once we're done. + """ + fd, filename = tempfile.mkstemp(prefix="bridgedb") + fh = os.fdopen(fd, "w") + fh.write(content) + fh.close() + return filename + + def test_load_csv(self): + """Load a valid CSV file.""" + content = "foo,bar\nbar,foo\n" + filename = self.write_file(content) + + csv = antibot._loadCSV(filename) + self.assertEqual(csv["foo"], "bar") + self.assertEqual(csv["bar"], "foo") + + os.unlink(filename) + + def test_load_invalid_csv(self): + """Load an invalid CSV file that has two commas in one line.""" + content = "foo,bar,bad\nbar,foo\n" + filename = self.write_file(content) + + csv = antibot._loadCSV(filename) + self.assertEqual(len(csv), 1) + + os.unlink(filename) + + def test_load_blacklisted_headers(self): + """Load valid blacklisted request headers.""" + content = "accept-language,[Kk]lingon" + filename = self.write_file(content) + + antibot.loadBlacklistedRequestHeaders(filename) + + request = DummyRequest(['']) + verdict = antibot.isRequestFromBot(request) + self.assertFalse(verdict) + + request.requestHeaders.setRawHeaders("accept-language", + ["i speak kllingon"]) + antibot.loadBlacklistedRequestHeaders(filename) + verdict = antibot.isRequestFromBot(request) + self.assertFalse(verdict) + + request.requestHeaders.setRawHeaders("accept-language", + ["i speak klingon"]) + antibot.loadBlacklistedRequestHeaders(filename) + verdict = antibot.isRequestFromBot(request) + self.assertTrue(verdict) + + os.unlink(filename) + + def test_load_invalid_blacklisted_headers(self): + """Load invalid blacklisted request headers with a broken regexp.""" + content = "accept-language,[Klingon\nuser-agent,foo*" + filename = self.write_file(content) + + antibot.loadBlacklistedRequestHeaders(filename) + self.assertEqual(len(antibot.BLACKLISTED_REQUEST_HEADERS), 1) + + os.unlink(filename) + + def test_load_decoy_bridges(self): + """Load decoy bridges.""" + obfs4_line = "obfs4 1.2.3.4:1234 FINGERPRINT FOO BAR" + vanilla_line = "1.2.3.4:1234 FINGERPRINT" + + content = "vanillav4,%s\nobfs4v4,%s" % (vanilla_line, obfs4_line) + filename = self.write_file(content) + + antibot.loadDecoyBridges(filename) + self.assertEqual(antibot.getDecoyBridge("obfs4", 4), [obfs4_line]) + self.assertEqual(antibot.getDecoyBridge("vanilla", 4), [vanilla_line]) + self.assertEqual(antibot.getDecoyBridge("vanilla", 6), None) + self.assertEqual(antibot.getDecoyBridge("vanilla", 7), None) + + os.unlink(filename)

1 0

[bridgedb/master] Bump version number to 0.8.0.
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit 7df8ccc7d07d503dd46df0dd8808f7ab555c715a Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Aug 20 09:48:17 2019 -0700 Bump version number to 0.8.0. --- CHANGELOG | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index 03390d6..aba6d19 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,4 +1,4 @@ -Changes in version 0.8.0 - YYYY-MM-DD +Changes in version 0.8.0 - 2019-08-20 * FIXES https://bugs.torproject.org/9316 Make BridgeDB export usage metrics every 24 hours. At the end of each

1 0

[bridgedb/master] Merge branch 'feature/31252' into develop
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit 8291822c593d8dcf26dbdbc14483c841f6d0ef2e Merge: 0a1de3f 7ceb25e Author: Philipp Winter <phw(a)nymity.ch> Date: Fri Aug 16 11:38:23 2019 -0700 Merge branch 'feature/31252' into develop CHANGELOG | 9 +++ bridgedb.conf | 19 ++++++ bridgedb/antibot.py | 123 ++++++++++++++++++++++++++++++++++ bridgedb/distributors/https/server.py | 10 +++ bridgedb/distributors/moat/server.py | 6 ++ bridgedb/main.py | 6 ++ bridgedb/test/test_antibot.py | 108 +++++++++++++++++++++++++++++ 7 files changed, 281 insertions(+)

1 0

[bridgedb/master] Use stem instead of leekspin to make descriptors.
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit ed37530c24fbf8ef80c7976ee4da1b667e9ba725 Author: Philipp Winter <phw(a)nymity.ch> Date: Tue Jun 11 16:02:15 2019 -0700 Use stem instead of leekspin to make descriptors. Stem now provides the ability to create and sign descriptors, so there's no reason to use leekspin anymore. Stem is faster at creating descriptors and we're already heavily depending on stem anyway. This patch improves the preliminary work that Isis Lovecruft and Damian Johnson already did. Some code in this patch was taken from leekspin. This fixes bug 22755: <https://bugs.torproject.org/22755> --- .test.requirements.txt | 1 - .travis.requirements.txt | 1 - CHANGELOG | 4 + README.rst | 6 +- bridgedb/main.py | 10 +- bridgedb/runner.py | 37 +----- doc/HACKING.md | 10 +- scripts/create_descriptors | 298 +++++++++++++++++++++++++++++++++++++++++++++ scripts/setup-tests | 2 +- setup.py | 2 +- 10 files changed, 316 insertions(+), 55 deletions(-) diff --git a/.test.requirements.txt b/.test.requirements.txt index c84fb58..ad5342f 100644 --- a/.test.requirements.txt +++ b/.test.requirements.txt @@ -6,7 +6,6 @@ # $ make coverage # coverage==4.2 -git+https://git.torproject.org/user/phw/leekspin.git@d34c804cd0f01af5206833e62c0dedec8565b235#egg=leekspin mechanize==0.2.5 pep8==1.5.7 # pylint must be pinned until pylint bug #203 is fixed. See diff --git a/.travis.requirements.txt b/.travis.requirements.txt index 2d56b79..e6eaf10 100644 --- a/.travis.requirements.txt +++ b/.travis.requirements.txt @@ -15,7 +15,6 @@ #------------------------------------------------------------------------------ coverage==4.2 coveralls==1.2.0 -git+https://git.torproject.org/user/phw/leekspin.git@d34c804cd0f01af5206833e62c0dedec8565b235#egg=leekspin mechanize==0.2.5 sure==1.2.2 Babel==0.9.6 diff --git a/CHANGELOG b/CHANGELOG index ae0e651..32e6fe5 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -12,6 +12,10 @@ Changes in version 0.8.0 - YYYY-MM-DD * FIXES #26542 https://bugs.torproject.org/26542 Make BridgeDB distribute vanilla IPv6 bridges again. + * FIXES #22755 https://bugs.torproject.org/22755 + Use stem instead of leekspin to create test descriptors. We now don't + need to depend on leekspin anymore. + Changes in version 0.7.1 - 2019-06-07 * FIXES #28496 https://bugs.torproject.org/28496 diff --git a/README.rst b/README.rst index ee1c979..4f05ea5 100644 --- a/README.rst +++ b/README.rst @@ -315,11 +315,7 @@ To create a bunch of fake bridge descriptors to test BridgeDB, do:: bridgedb mock [-n NUMBER_OF_DESCRIPTORS] -Note that you will need to install -`leekspin <https://pypi.python.org/pypi/leekspin>`__ in order to run the -``bridgedb mock`` command. See ``doc/HACKING.md`` for details. - -And finally, to run the test suites, do:: +To run the test suites, do:: make coverage diff --git a/bridgedb/main.py b/bridgedb/main.py index 6b99127..5d9b0c6 100644 --- a/bridgedb/main.py +++ b/bridgedb/main.py @@ -551,15 +551,9 @@ def runSubcommand(options, config): # mentioned above with the email.server and https.server. from bridgedb import runner - statuscode = 0 - if options.subCommand is not None: logging.debug("Running BridgeDB command: '%s'" % options.subCommand) if 'descriptors' in options.subOptions: - statuscode = runner.generateDescriptors( - options.subOptions['descriptors'], config.RUN_IN_DIR) - - logging.info("Subcommand '%s' finished with status %s." - % (options.subCommand, statuscode)) - sys.exit(statuscode) + runner.generateDescriptors(int(options.subOptions['descriptors']), config.RUN_IN_DIR) + sys.exit(0) diff --git a/bridgedb/runner.py b/bridgedb/runner.py index b1a21d2..b6117e1 100644 --- a/bridgedb/runner.py +++ b/bridgedb/runner.py @@ -81,17 +81,6 @@ def find(filename): def generateDescriptors(count=None, rundir=None): """Run a script which creates fake bridge descriptors for testing purposes. - This will run Leekspin_ to create bridge server descriptors, bridge - extra-info descriptors, and networkstatus document. - - .. warning: This function can take a very long time to run, especially in - headless environments where entropy sources are minimal, because it - creates the keys for each mocked OR, which are embedded in the server - descriptors, used to calculate the OR fingerprints, and sign the - descriptors, among other things. - - .. _Leekspin: https://gitweb.torproject.org/user/phw/leekspin.git - :param integer count: Number of mocked bridges to generate descriptor for. (default: 3) :type rundir: string or None @@ -100,25 +89,11 @@ def generateDescriptors(count=None, rundir=None): directory MUST already exist, and the descriptor files will be created in it. If None, use the whatever directory we are currently in. """ - import subprocess - import os.path + from stem.descriptor.server_descriptor import RelayDescriptor - proc = None - statuscode = 0 - script = 'leekspin' - rundir = rundir if os.path.isdir(rundir) else None count = count if count else 3 - try: - proc = subprocess.Popen([script, '-n', str(count)], - close_fds=True, cwd=rundir) - finally: - if proc is not None: - proc.wait() - if proc.returncode: - print("There was an error generating bridge descriptors.", - "(Returncode: %d)" % proc.returncode) - statuscode = proc.returncode - else: - print("Sucessfully generated %s descriptors." % str(count)) - del subprocess - return statuscode + rundir = rundir if rundir else os.getcwd() + + for i in range(count): + with open(os.path.join(rundir, 'descriptor_%i' % i), 'w') as descriptor_file: + descriptor_file.write(RelayDescriptor.content(sign = True)) diff --git a/doc/HACKING.md b/doc/HACKING.md index aa6c119..a8ec640 100644 --- a/doc/HACKING.md +++ b/doc/HACKING.md @@ -12,20 +12,16 @@ with password ```writecode```. ## Generating bridge descriptors Developers wishing to test BridgeDB will need to generate mock bridge -descriptors. This is accomplished through the [leekspin -script](https://gitweb.torproject.org/user/phw/leekspin.git). To generate 20 -bridge descriptors, change to the bridgedb running directory and do: +descriptors. This is accomplished through the file **create-descriptors**. To +generate 20 bridge descriptors, change to the bridgedb running directory and do: - $ leekspin -n 20 + $ ./scripts/create-descriptors 20 It is recommended that you generate at least 250 descriptors for testing. Ideally, even more descriptors should be generated, somewhere in the realm of 2000, as certain bugs do not emerge until BridgeDB is processing thousands of descriptors. -**Leekspin is for testing purposes only and should never be deployed on a -production server.** We do not want to distribute fake bridges. - ## Git Workflow See this article on git branching [workflow][workflow]. The only modifications diff --git a/scripts/create_descriptors b/scripts/create_descriptors new file mode 100755 index 0000000..839a6ba --- /dev/null +++ b/scripts/create_descriptors @@ -0,0 +1,298 @@ +#!/usr/bin/env python2.7 +# +# This file is part of BridgeDB, a Tor bridge distribution system. + +import os +import random +import sys +import time +import ipaddr +import math +import argparse +import hashlib + +# A bunch of Tor version numbers. +SERVER_VERSIONS = ["0.2.2.39", + "0.2.3.24-rc", + "0.2.3.25", + "0.2.4.5-alpha", + "0.2.4.6-alpha", + "0.2.4.7-alpha", + "0.2.4.8-alpha", + "0.2.4.9-alpha", + "0.2.4.10-alpha", + "0.2.4.11-alpha", + "0.2.4.12-alpha", + "0.2.4.14-alpha", + "0.2.4.15-rc", + "0.2.4.16-rc", + "0.2.4.17-rc", + "0.2.4.18-rc", + "0.2.4.19", + "0.2.4.20", + "0.2.5.1-alpha", + ] + +try: + import stem + import stem.descriptor + from stem.descriptor.server_descriptor import RelayDescriptor + from stem.descriptor.extrainfo_descriptor import RelayExtraInfoDescriptor + from stem.descriptor.networkstatus import NetworkStatusDocumentV3 +except ImportError: + print("Creating descriptors requires stem <https://stem.torproject.org>") + sys.exit(1) + +if not hasattr(stem.descriptor, "create_signing_key"): + print("This requires stem version 1.6 or later but you are running " + "version %s" % stem.__version__) + sys.exit(1) + + +def make_output_dir(): + if not os.path.exists(os.getcwd()): + os.mkdir(os.getcwd()) + + +def write_descriptors(descs, filename): + make_output_dir() + with open(os.path.join(os.getcwd(), filename), "w") as descriptor_file: + for descriptor in descs: + descriptor_file.write(str(descriptor)) + + +def write_descriptor(desc, filename): + make_output_dir() + with open(os.path.join(os.getcwd(), filename), "w") as descriptor_file: + descriptor_file.write(str(desc)) + + +def check_ip_validity(ip): + if (ip.is_link_local or + ip.is_loopback or + ip.is_multicast or + ip.is_private or + ip.is_unspecified or + ((ip.version == 6) and ip.is_site_local) or + ((ip.version == 4) and ip.is_reserved)): + return False + return True + + +def get_transport_line(probing_resistant, addr, port): + """ + If probing_resistant is True, add a transport protocol that's resistant to + active probing attacks. + """ + + transports = [] + if probing_resistant: + transports.append("obfs2 %s:%s" % (addr, port-10)) + iat_mode = random.randint(0, 1) + node_id = hashlib.sha1(bytes(random.getrandbits(8))).hexdigest() + public_key = hashlib.sha256(bytes(random.getrandbits(8))).hexdigest() + transports.append("obfs4 %s:%s iat-mode=%s,node-id=%s,public-key=%s" % + (addr, port-20, iat_mode, node_id, public_key)) + + # Always include obfs4 and occasionally include scramblesuit. + + if random.randint(0, 1) > 0: + transports.append("scramblesuit 216.117.3.62:63174 " + "password=ABCDEFGHIJKLMNOPQRSTUVWXYZ234567") + else: + transports.append("obfs2 %s:%s" % (addr, port-10)) + transports.append("obfs3 %s:%s" % (addr, port-20)) + + return "\ntransport ".join(transports) + + +def get_hex_string(size): + hexstr = "" + for _ in range(size): + hexstr += random.choice("ABCDEF0123456789") + return hexstr + + +def get_random_ipv6_addr(): + valid_addr = None + while not valid_addr: + maybe = ipaddr.IPv6Address(random.getrandbits(128)) + valid = check_ip_validity(maybe) + if valid: + valid_addr = maybe + break + return str(valid_addr) + + +def get_random_ipv4_addr(): + return "%i.%i.%i.%i" % (random.randint(0, 255), + random.randint(0, 255), + random.randint(0, 255), + random.randint(0, 255)) + + +def get_protocol(tor_version): + line = "" + if tor_version is not None: + line += "opt " + line += "protocols Link 1 2 Circuit 1" + return line + + +def make_timestamp(now=None, fmt=None, variation=False, period=None): + now = int(now) if now is not None else int(time.time()) + fmt = fmt if fmt else "%Y-%m-%d %H:%M:%S" + + if variation: + then = 1 + if period is not None: + secs = int(period) * 3600 + then = now - secs + # Get a random number between one epochseconds number and another + diff = random.randint(then, now) + # Then rewind the clock + now = diff + + return time.strftime(fmt, time.localtime(now)) + + +def make_bandwidth(variance=30): + observed = random.randint(20 * 2**10, 2 * 2**30) + percentage = float(variance) / 100. + burst = int(observed + math.ceil(observed * percentage)) + bandwidths = [burst, observed] + nitems = len(bandwidths) if (len(bandwidths) > 0) else float("nan") + avg = int(math.ceil(float(sum(bandwidths)) / nitems)) + return "%s %s %s" % (avg, burst, observed) + + +def make_bridge_distribution_request(): + + methods = [ + "any", + "none", + "https", + "email", + "moat", + ] + + return random.choice(methods) + + +def create_server_desc(signing_key): + """ + Create and return a server descriptor. + """ + + nickname = ("Unnamed%i" % random.randint(0, 100000000000000))[:19] + + # We start at port 10 because we subtract from this port to get port + # numbers for IPv6 and obfuscation protocols. + + port = random.randint(10, 65535) + tor_version = random.choice(SERVER_VERSIONS) + timestamp = make_timestamp(variation=True, period=36) + + server_desc = RelayDescriptor.create({ + "router": "%s %s %s 0 0" % (nickname, get_random_ipv4_addr(), port), + "or-address": "[%s]:%s" % (get_random_ipv6_addr(), port-1), + "platform": "Tor %s on Linux" % tor_version, + get_protocol(tor_version): "", + "published": timestamp, + "uptime": str(int(random.randint(1800, 63072000))), + "bandwidth": make_bandwidth(), + "contact": "Somebody <somebody(a)example.com>", + "bridge-distribution-request": make_bridge_distribution_request(), + "reject": "*:*", + }, signing_key=signing_key) + + return server_desc + + +def create_extrainfo_desc(server_desc, signing_key, probing_resistant): + """ + Create and return an extrainfo descriptor. + """ + + ts = server_desc.published + + extrainfo_desc = RelayExtraInfoDescriptor.create({ + "extra-info": "%s %s" % (server_desc.nickname, + server_desc.fingerprint), + "transport": get_transport_line(probing_resistant, + server_desc.address, + server_desc.or_port), + "write-history": "%s (900 s) 3188736,2226176,2866176" % ts, + "read-history": "%s (900 s) 3891200,2483200,2698240" % ts, + "dirreq-write-history": "%s (900 s) 1024,0,2048" % ts, + "dirreq-read-history": "%s (900 s) 0,0,0" % ts, + "geoip-db-digest": "%s" % get_hex_string(40), + "geoip6-db-digest": "%s" % get_hex_string(40), + "dirreq-stats-end": "%s (86400 s)" % ts, + "dirreq-v3-ips": "", + "dirreq-v3-reqs": "", + "dirreq-v3-resp": "ok=16,not-enough-sigs=0,unavailable=0," + "not-found=0,not-modified=0,busy=0", + "dirreq-v3-direct-dl": "complete=0,timeout=0,running=0", + "dirreq-v3-tunneled-dl": "complete=12,timeout=0,running=0", + "bridge-stats-end": "%s (86400 s)" % ts, + "bridge-ips": "ca=8", + "bridge-ip-versions": "v4=8,v6=0", + "bridge-ip-transports": "<OR>=8", + }, signing_key=signing_key) + + return extrainfo_desc + + +def make_descriptors(count, num_probing_resistant): + """ + Create fake descriptors and write them to the working directory. + """ + + consensus_entries = [] + server_descriptors = [] + extrainfos_old = [] + extrainfos_new = [] + + for i in range(count): + signing_key = stem.descriptor.create_signing_key() + + server_desc = create_server_desc(signing_key) + server_descriptors.append(server_desc) + consensus_entries.append(server_desc.make_router_status_entry()) + + extrainfo_desc = create_extrainfo_desc(server_desc, + signing_key, + num_probing_resistant > 0) + if random.random() > 0.75: + extrainfos_new.append(extrainfo_desc) + else: + extrainfos_old.append(extrainfo_desc) + + if num_probing_resistant > 0: + num_probing_resistant -= 1 + + consensus = NetworkStatusDocumentV3.create(routers=consensus_entries) + write_descriptor(consensus, "networkstatus-bridges") + write_descriptors(server_descriptors, "bridge-descriptors") + write_descriptors(extrainfos_old, "cached-extrainfo") + write_descriptors(extrainfos_new, "cached-extrainfo.new") + + +if __name__ == "__main__": + + parser = argparse.ArgumentParser(description="Create fake descriptors.") + parser.add_argument("num_descs", + type=int, + help="The number of descriptors to create.") + parser.add_argument("--num-resistant-descs", + dest="num_resistant_descs", + type=int, + default=-1, + help="The number of active probing-resistant " + "descriptors to create") + args = parser.parse_args() + if args.num_resistant_descs == -1: + args.num_resistant_descs = args.num_descs + + make_descriptors(args.num_descs, args.num_resistant_descs) diff --git a/scripts/setup-tests b/scripts/setup-tests index ccfd6cd..1de3c71 100755 --- a/scripts/setup-tests +++ b/scripts/setup-tests @@ -29,7 +29,7 @@ sed -r -i -e "s/(SERVER_PUBLIC_FQDN = )(.*)/\1'127.0.0.1:6788'/" run/bridgedb.co sed -r -i -e "s/(MOAT_HTTP_IP = )(None)/\1'127.0.0.1'/" run/bridgedb.conf sed -r -i -e "s/(MOAT_HTTP_PORT = )(None)/\16790/" run/bridgedb.conf # Create descriptors -leekspin -n 100 -xp 50 +./scripts/create_descriptors 200 --num-resistant-descs 100 cp -t run/from-authority networkstatus-bridges cached-extrainfo* bridge-descriptors cp -t run/from-bifroest networkstatus-bridges cached-extrainfo* bridge-descriptors # Create TLS certificates diff --git a/setup.py b/setup.py index a31391c..d416289 100644 --- a/setup.py +++ b/setup.py @@ -387,7 +387,7 @@ setuptools.setup( 'scripts/get-tor-exits'], extras_require={'test': ["sure==1.2.2", "coverage==4.2", - "leekspin==1.1.4"]}, + "cryptography==1.9"]}, zip_safe=False, cmdclass=get_cmdclass(), include_package_data=True,

1 0

[bridgedb/master] Make BridgeDB export usage metrics.
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit 5cde59d9ccafdb248ca8aa9c1c9abbfe2edb5dc6 Author: Philipp Winter <phw(a)nymity.ch> Date: Mon Aug 12 14:05:48 2019 -0700 Make BridgeDB export usage metrics. Until now, we had no insight into how BridgeDB is being used. We don't know the relative popularity of our distribution method; we don't know how many users BridgeDB sees; we don't know how many requests succeed or fail; and we don't know the relative popularity of transports that users request. This patch attempts to answer these questions by making BridgeDB export usage metrics. At the end of each 24-hour measurement interval, BridgeDB will append usage metrics to the file METRICS_FILE, which is configured in bridgedb.conf. Our metrics keep track of the number of (un)successful requests per transport type per country code (or email provider) per distribution method. This way, we get to learn that, say, over the last 24 hours there were 31-40 users in Iran who successfully requested an obfs4 bridge over Moat. The corresponding metrics line would look as follows: bridgedb-metric-count moat.obfs4.ir.success.none 40 To make the metrics preserve user privacy, we don't collect user-identifying information and we introduce noise by rounding up metrics to our bin size which defaults to 10. This patch also extends the looping calls that BridgeDB spawns. When BridgeDB first starts, it loads proxies from the files PROXY_LIST_FILES. It augments this list of proxies with Tor exit relays that we download every three hours. --- CHANGELOG | 11 +- bridgedb.conf | 16 +- bridgedb/distributors/email/autoresponder.py | 16 + bridgedb/distributors/https/server.py | 11 + bridgedb/distributors/moat/server.py | 15 + bridgedb/main.py | 29 +- bridgedb/metrics.py | 461 +++++++++++++++++++++++++++ bridgedb/test/test_metrics.py | 204 ++++++++++++ 8 files changed, 755 insertions(+), 8 deletions(-) diff --git a/CHANGELOG b/CHANGELOG index 1229578..ae0e651 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,4 +1,13 @@ -Changes in version 0.7.2 - YYYY-MM-DD +Changes in version 0.8.0 - YYYY-MM-DD + + * FIXES https://bugs.torproject.org/9316 + Make BridgeDB export usage metrics every 24 hours. At the end of each + 24-hour measurement interval, BridgeDB will append usage metrics to the + file METRICS_FILE, which is configured in bridgedb.conf. Our metrics + keep track of the number of (un)successful requests per transport type + per country code (or email provider) per distribution method. This way, + we get to learn that, say, over the last 24 hours there were 31-40 users + in Iran who successfully requested an obfs4 bridge over Moat. * FIXES #26542 https://bugs.torproject.org/26542 Make BridgeDB distribute vanilla IPv6 bridges again. diff --git a/bridgedb.conf b/bridgedb.conf index 66b1983..ba43bb6 100644 --- a/bridgedb.conf +++ b/bridgedb.conf @@ -177,6 +177,9 @@ MASTER_KEY_FILE = "secret_key" # File to which we dump bridge pool assignments for statistics. ASSIGNMENTS_FILE = "assignments.log" +# Name of the file that contains BridgeDB's metrics. +METRICS_FILE = "bridgedb-metrics.log" + #------------------ # Logging Options \ #------------------------------------------------------------------------------ @@ -260,16 +263,19 @@ FORCE_FLAGS = [("Stable", 1)] # Only consider routers whose purpose matches this string. BRIDGE_PURPOSE = "bridge" -# TASKS is a dictionary mapping the names of tasks to the frequency with which -# they should be run (in seconds). If a task's value is set to 0, it will not -# be scheduled to run. +# TASKS is a dictionary mapping the names of tasks to a tuple consisting of the +# frequency with which they should be run (in seconds) and a boolean value +# expressing if the task should be run immediately after start up. If a task's +# frequency is set to 0, it will not be scheduled to run. TASKS = { # Download a list of Tor exit relays once every three hours (by running # scripts/get-exit-list) and add those exit relays to the list of proxies # loaded from the PROXY_LIST_FILES: - 'GET_TOR_EXIT_LIST': 3 * 60 * 60, + 'GET_TOR_EXIT_LIST': (3 * 60 * 60, True), # Delete *.unparseable descriptor files which are more than 24 hours old: - 'DELETE_UNPARSEABLE_DESCRIPTORS': 24 * 60 * 60, + 'DELETE_UNPARSEABLE_DESCRIPTORS': (24 * 60 * 60, False), + # Export usage metrics every 24 hours: + 'EXPORT_METRICS': (24 * 60 * 60, False), } # SUPPORTED_TRANSPORTS is a dictionary mapping Pluggable Transport methodnames diff --git a/bridgedb/distributors/email/autoresponder.py b/bridgedb/distributors/email/autoresponder.py index ff65a73..e69f78a 100644 --- a/bridgedb/distributors/email/autoresponder.py +++ b/bridgedb/distributors/email/autoresponder.py @@ -48,6 +48,8 @@ from twisted.internet import reactor from twisted.mail import smtp from twisted.python import failure +from bridgedb import strings +from bridgedb import metrics from bridgedb import safelog from bridgedb.crypto import NEW_BUFFER_INTERFACE from bridgedb.distributors.email import dkim @@ -62,6 +64,10 @@ from bridgedb.parse.addr import canonicalizeEmailDomain from bridgedb.util import levenshteinDistance from bridgedb import translations +# We use our metrics singleton to keep track of BridgeDB metrics such as +# "number of failed HTTPS bridge requests." +metrix = metrics.EmailMetrics() + def createResponseBody(lines, context, client, lang='en'): """Parse the **lines** from an incoming email request and determine how to @@ -424,6 +430,16 @@ class SMTPAutoresponder(smtp.SMTPClient): body = createResponseBody(self.incoming.lines, self.incoming.context, client, lang) + + # The string EMAIL_MISC_TEXT[1] shows up in an email if BridgeDB + # responds with bridges. Everything else we count as an invalid + # request. + translator = translations.installTranslations(lang) + if body is not None and translator.gettext(strings.EMAIL_MISC_TEXT[1]) in body: + metrix.recordValidEmailRequest(self) + else: + metrix.recordInvalidEmailRequest(self) + if not body: return # The client was already warned. messageID = self.incoming.message.getheader("Message-ID", None) diff --git a/bridgedb/distributors/https/server.py b/bridgedb/distributors/https/server.py index 8c50bc1..732f8bf 100644 --- a/bridgedb/distributors/https/server.py +++ b/bridgedb/distributors/https/server.py @@ -52,6 +52,7 @@ from bridgedb import crypto from bridgedb import strings from bridgedb import translations from bridgedb import txrecaptcha +from bridgedb import metrics from bridgedb.distributors.common.http import setFQDN from bridgedb.distributors.common.http import getFQDN from bridgedb.distributors.common.http import getClientIP @@ -85,6 +86,10 @@ logging.debug("Set template root to %s" % TEMPLATE_DIR) #: Localisations which BridgeDB supports which should be rendered right-to-left. rtl_langs = ('ar', 'he', 'fa', 'gu_IN', 'ku') +# We use our metrics singleton to keep track of BridgeDB metrics such as +# "number of failed HTTPS bridge requests." +metrix = metrics.HTTPSMetrics() + def replaceErrorPage(request, error, template_name=None, html=True): """Create a general error page for displaying in place of tracebacks. @@ -495,6 +500,7 @@ class CaptchaProtectedResource(CustomErrorHandlingResource, CSPResource): try: if self.checkSolution(request) is True: + metrix.recordValidHTTPSRequest(request) return self.resource.render(request) except ValueError as err: logging.debug(err.message) @@ -504,11 +510,14 @@ class CaptchaProtectedResource(CustomErrorHandlingResource, CSPResource): # work of art" as pennance for their sins. d = task.deferLater(reactor, 1, lambda: request) d.addCallback(redirectMaliciousRequest) + metrix.recordInvalidHTTPSRequest(request) return NOT_DONE_YET except Exception as err: logging.debug(err.message) + metrix.recordInvalidHTTPSRequest(request) return replaceErrorPage(request, err) + metrix.recordInvalidHTTPSRequest(request) logging.debug("Client failed a CAPTCHA; returning redirect to %s" % request.uri) return redirectTo(request.uri, request) @@ -764,10 +773,12 @@ class ReCaptchaProtectedResource(CaptchaProtectedResource): # breaking). Hence, the 'no cover' pragma. if solution.is_valid: # pragma: no cover logging.info("Valid CAPTCHA solution from %r." % clientIP) + metrix.recordValidHTTPSRequest(request) return (True, request) else: logging.info("Invalid CAPTCHA solution from %r: %r" % (clientIP, solution.error_code)) + metrix.recordInvalidHTTPSRequest(request) return (False, request) d = txrecaptcha.submit(challenge, response, self.secretKey, diff --git a/bridgedb/distributors/moat/server.py b/bridgedb/distributors/moat/server.py index 509d471..73d2423 100644 --- a/bridgedb/distributors/moat/server.py +++ b/bridgedb/distributors/moat/server.py @@ -38,6 +38,7 @@ from twisted.internet.error import CannotListenError from twisted.web import resource from twisted.web.server import Site +from bridgedb import metrics from bridgedb import captcha from bridgedb import crypto from bridgedb.distributors.common.http import setFQDN @@ -49,6 +50,10 @@ from bridgedb.schedule import Unscheduled from bridgedb.schedule import ScheduledInterval from bridgedb.util import replaceControlChars +# We use our metrics singleton to keep track of BridgeDB metrics such as +# "number of failed HTTPS bridge requests." +metrix = metrics.MoatMetrics() + #: The current version of the moat JSON API that we speak MOAT_API_VERSION = '0.1.0' @@ -681,6 +686,8 @@ class CaptchaCheckResource(CaptchaResource): error = self.checkRequestHeaders(request) if error: # pragma: no cover + logging.debug("Error while checking moat request headers.") + metrix.recordInvalidMoatRequest(request) return error.render(request) data = { @@ -694,7 +701,11 @@ class CaptchaCheckResource(CaptchaResource): } try: + pos = request.content.tell() encoded_client_data = request.content.read() + # We rewind the stream to its previous position to allow the + # metrix module to read the request's content too. + request.content.seek(pos) client_data = json.loads(encoded_client_data)["data"][0] clientIP = self.getClientIP(request) @@ -704,16 +715,19 @@ class CaptchaCheckResource(CaptchaResource): valid = self.checkSolution(challenge, solution, clientIP) except captcha.CaptchaExpired: logging.debug("The challenge had timed out") + metrix.recordInvalidMoatRequest(request) return self.failureResponse(5, request) except Exception as impossible: logging.warn("Unhandled exception while processing a POST /fetch request!") logging.error(impossible) + metrix.recordInvalidMoatRequest(request) return self.failureResponse(4, request) if valid: qrcode = None bridgeRequest = self.createBridgeRequest(clientIP, client_data) bridgeLines = self.getBridgeLines(bridgeRequest) + metrix.recordValidMoatRequest(request) # If we can only return less than the configured # MOAT_BRIDGES_PER_ANSWER then log a warning. @@ -736,6 +750,7 @@ class CaptchaCheckResource(CaptchaResource): return self.formatDataForResponse(data, request) else: + metrix.recordInvalidMoatRequest(request) return self.failureResponse(4, request) diff --git a/bridgedb/main.py b/bridgedb/main.py index 4d1d38a..6b99127 100644 --- a/bridgedb/main.py +++ b/bridgedb/main.py @@ -25,6 +25,7 @@ from bridgedb import persistent from bridgedb import proxy from bridgedb import runner from bridgedb import util +from bridgedb import metrics from bridgedb.bridges import MalformedBridgeInfo from bridgedb.bridges import MissingServerDescriptorDigest from bridgedb.bridges import ServerDescriptorDigestMismatch @@ -72,6 +73,22 @@ def writeAssignments(hashring, filename): except IOError: logging.info("I/O error while writing assignments to: '%s'" % filename) +def writeMetrics(filename, measurementInterval): + """Dump usage metrics to disk. + + :param str filename: The filename to write the metrics to. + :param int measurementInterval: The number of seconds after which we rotate + and dump our metrics. + """ + + logging.debug("Dumping metrics to file: '%s'" % filename) + + try: + with open(filename, 'a') as fh: + metrics.export(fh, measurementInterval) + except IOError as err: + logging.error("Failed to write metrics to '%s': %s" % (filename, err)) + def load(state, hashring, clear=False): """Read and parse all descriptors, and load into a bridge hashring. @@ -398,6 +415,7 @@ def run(options, reactor=reactor): for proxyfile in cfg.PROXY_LIST_FILES: logging.info("Loading proxies from: %s" % proxyfile) proxy.loadProxiesFromFile(proxyfile, proxies, removeStale=True) + metrics.setProxies(proxies) logging.info("Reparsing bridge descriptors...") (hashring, @@ -463,6 +481,8 @@ def run(options, reactor=reactor): if config.EMAIL_DIST and config.EMAIL_SHARE: addSMTPServer(config, emailDistributor) + metrics.setSupportedTransports(config.SUPPORTED_TRANSPORTS) + tasks = {} # Setup all our repeating tasks: @@ -483,14 +503,19 @@ def run(options, reactor=reactor): runner.cleanupUnparseableDescriptors, os.path.dirname(config.STATUS_FILE), delUnparseableSecs) + measurementInterval, _ = config.TASKS['EXPORT_METRICS'] + tasks['EXPORT_METRICS'] = task.LoopingCall( + writeMetrics, state.METRICS_FILE, measurementInterval) + # Schedule all configured repeating tasks: - for name, seconds in config.TASKS.items(): + for name, value in config.TASKS.items(): + seconds, startNow = value if seconds: try: # Set now to False to get the servers up and running when # first started, rather than spend a bunch of time in # scheduled tasks. - tasks[name].start(abs(seconds), now=False) + tasks[name].start(abs(seconds), now=startNow) except KeyError: logging.info("Task %s is disabled and will not run." % name) else: diff --git a/bridgedb/metrics.py b/bridgedb/metrics.py new file mode 100644 index 0000000..4e1c880 --- /dev/null +++ b/bridgedb/metrics.py @@ -0,0 +1,461 @@ +# -*- coding: utf-8 ; test-case-name: bridgedb.test.test_metrics ; -*- +# _____________________________________________________________________________ +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: please see included AUTHORS file +# :copyright: (c) 2019, The Tor Project, Inc. +# (c) 2019, Philipp Winter +# :license: see LICENSE for licensing information +# _____________________________________________________________________________ + +"""API for keeping track of BridgeDB statistics, e.g., the demand for bridges +over time. +""" + +import logging +import ipaddr +import operator +import json +import datetime + +from bridgedb import geo +from bridgedb.distributors.common.http import getClientIP +from bridgedb.distributors.email import request +from bridgedb.distributors.email.distributor import EmailRequestedHelp + +from twisted.mail.smtp import Address + +# Our data structure to keep track of exit relays. The variable is of type +# bridgedb.proxy.ProxySet. We reserve a special country code (determined by +# PROXY_CC below) for exit relays and other proxies. +PROXIES = None + +# Our custom country code for IP addresses that we couldn't map to a country. +# This can happen for private IP addresses or if our geo-location provider has +# no mapping. +UNKNOWN_CC = "??" + +# Our custom country code for IP addresses that are proxies, e.g., Tor exit +# relays. The code "zz" is free for assignment for user needs as specified +# here: <https://en.wikipedia.org/w/index.php?title=ISO_3166-1_alpha-2&oldid=9066112…> +PROXY_CC = "ZZ" + +# We use BIN_SIZE to reduce the granularity of our counters. We round up +# numbers to the next multiple of BIN_SIZE, e.g., 28 is rounded up to: +# 10 * 3 = 30. +BIN_SIZE = 10 + +# The prefix length that we use to keep track of the number of unique subnets +# we have seen HTTPS requests from. +SUBNET_CTR_PREFIX_LEN = 20 + +# All of the pluggable transports BridgeDB currently supports. +SUPPORTED_TRANSPORTS = None + +# Major and minor version number for our statistics format. +METRICS_MAJOR_VERSION = 1 +METRICS_MINOR_VERSION = 0 + + +def setProxies(proxies): + """Set the given proxies. + + :type proxies: :class:`~bridgedb.proxy.ProxySet` + :param proxies: The container for the IP addresses of any currently + known open proxies. + """ + logging.debug("Setting %d proxies." % len(proxies)) + global PROXIES + PROXIES = proxies + + +def setSupportedTransports(supportedTransports): + """Set the given supported transports. + + :param dict supportedTransports: The transport types that BridgeDB + currently supports. + """ + + logging.debug("Setting %d supported transports." % + len(supportedTransports)) + global SUPPORTED_TRANSPORTS + SUPPORTED_TRANSPORTS = supportedTransports + + +def isTransportSupported(transport): + """Return `True' if the given transport is supported or `False' otherwise. + + :param str transport: The transport protocol. + """ + + if SUPPORTED_TRANSPORTS is None: + logging.error("Bug: Variable SUPPORTED_TRANSPORTS is None.") + return False + + return transport in SUPPORTED_TRANSPORTS + + +def export(fh, measurementInterval): + """Export metrics by writing them to the given file handle. + + :param file fh: The file handle to which we're writing our metrics. + :param int measurementInterval: The number of seconds after which we rotate + and dump our metrics. + """ + + httpsMetrix = HTTPSMetrics() + emailMetrix = EmailMetrics() + moatMetrix = MoatMetrics() + + # Rotate our metrics. + httpsMetrix.rotate() + emailMetrix.rotate() + moatMetrix.rotate() + + numProxies = len(PROXIES) if PROXIES is not None else 0 + if numProxies == 0: + logging.error("Metrics module doesn't have any proxies.") + else: + logging.debug("Metrics module knows about %d proxies." % numProxies) + + now = datetime.datetime.utcnow() + fh.write("bridgedb-stats-end %s (%d s)\n" % ( + now.strftime("%Y-%m-%d %H:%M:%S"), + measurementInterval)) + fh.write("bridgedb-stats-version %d.%d\n" % (METRICS_MAJOR_VERSION, + METRICS_MINOR_VERSION)) + + httpsLines = httpsMetrix.getMetrics() + for line in httpsLines: + fh.write("bridgedb-metric-count %s\n" % line) + + moatLines = moatMetrix.getMetrics() + for line in moatLines: + fh.write("bridgedb-metric-count %s\n" % line) + + emailLines = emailMetrix.getMetrics() + for line in emailLines: + fh.write("bridgedb-metric-count %s\n" % line) + + +def resolveCountryCode(ipAddr): + """Return the country code of the given IP address. + + :param str ipAddr: The IP address to resolve. + + :rtype: str + :returns: A two-letter country code. + """ + + if ipAddr is None: + logging.warning("Given IP address was None. Using %s as country " + "code." % UNKNOWN_CC) + return UNKNOWN_CC + + if PROXIES is None: + logging.warning("Proxies are not yet set.") + elif ipAddr in PROXIES: + return PROXY_CC + + countryCode = geo.getCountryCode(ipaddr.IPAddress(ipAddr)) + + # countryCode may be None if GeoIP is unable to map an IP address to a + # country. + return UNKNOWN_CC if countryCode is None else countryCode + + +class Singleton(type): + _instances = {} + + def __call__(cls, *args, **kwargs): + if cls not in cls._instances: + cls._instances[cls] = super(Singleton, cls).__call__(*args, + **kwargs) + return cls._instances[cls] + + def clear(cls): + """Drop the instance (necessary for unit tests).""" + try: + del cls._instances[cls] + except KeyError: + pass + + +class Metrics(object): + """Base class representing metrics. + + This class provides functionality that our three distribution mechanisms + share. + """ + + # We're using a meta class to implement a singleton for Metrics. + __metaclass__ = Singleton + + def __init__(self, binSize=BIN_SIZE): + logging.debug("Instantiating metrics class.") + self.binSize = binSize + + # Metrics cover a 24 hour period. To that end, we're maintaining two + # data structures: our "hot" metrics are currently being populated + # while our "cold" metrics are finished, and valid for 24 hours. After + # that, our hot metrics turn into cold metrics, and we start over. + self.hotMetrics = dict() + self.coldMetrics = dict() + + def rotate(self): + """Rotate our metrics.""" + + self.coldMetrics = self.hotMetrics + self.hotMetrics = dict() + + def findAnomaly(self, request): + anomaly = "none" + + # TODO: Inspect email for traces of bots, Sherlock Homes-style! + # See <https://bugs.torproject.org/9316#comment:19> for the rationale. + # All classes that inherit from Metrics() should implement this method. + + return anomaly + + def getMetrics(self): + """Get our sanitized current metrics, one per line. + + Metrics are of the form: + + [ + "moat.obfs4.us.success.none 10", + "https.vanilla.de.success.none 30", + ... + ] + + :rtype: list + :returns: A list of metric lines. + """ + lines = [] + for key, value in self.coldMetrics.iteritems(): + # Round up our value to the nearest multiple of self.binSize to + # reduce the accuracy of our real values. + if (value % self.binSize) > 0: + value += self.binSize - (value % self.binSize) + lines.append("%s %d" % (key, value)) + return lines + + def set(self, key, value): + """Set the given key to the given value. + + :param str key: The time series key. + :param int value: The time series value. + """ + self.hotMetrics[key] = value + + def inc(self, key): + """Increment the given key. + + :param str key: The time series key. + """ + if key in self.hotMetrics: + self.hotMetrics[key] += 1 + else: + self.set(key, 1) + + def createKey(self, distMechanism, bridgeType, countryOrProvider, + success, anomaly): + """Create and return a time series key. + + :param str distMechanism: A string representing our distribution + mechanism, e.g., "https". + :param str bridgeType: A string representing the requested bridge + type, e.g., "vanilla" or "obfs4". + :param str countryOrProvider: A string representing the client's + two-letter country code or email provider, e.g., "it" or + "yahoo.com". + :param bool success: ``True`` if the request was successful and + BridgeDB handed out a bridge; ``False`` otherwise. + :param str anomaly: ``None`` if the request was not anomalous and hence + believed to have come from a real user; otherwise a string + representing the type of anomaly. + :rtype: str + :returns: A key that uniquely identifies the given metrics + combinations. + """ + + countryOrProvider = countryOrProvider.lower() + bridgeType = bridgeType.lower() + success = "success" if success else "fail" + + key = "%s.%s.%s.%s.%s" % (distMechanism, bridgeType, + countryOrProvider, success, anomaly) + + return key + + +class HTTPSMetrics(Metrics): + + def __init__(self): + super(HTTPSMetrics, self).__init__() + + # Maps subnets (e.g., "1.2.0.0/16") to the number of times we've seen + # requests from the given subnet. + self.subnetCounter = dict() + self.keyPrefix = "https" + + def getTopNSubnets(self, n=10): + + sortedByNum = sorted(self.subnetCounter.items(), + key=operator.itemgetter(1), + reverse=True) + return sortedByNum[:n] + + def _recordHTTPSRequest(self, request, success): + + logging.debug("HTTPS request has user agent: %s" % + request.requestHeaders.getRawHeaders("User-Agent")) + + # Pull the client's IP address out of the request and convert it to a + # two-letter country code. + ipAddr = getClientIP(request, + useForwardedHeader=True, + skipLoopback=False) + self.updateSubnetCounter(ipAddr) + countryCode = resolveCountryCode(ipAddr) + + transports = request.args.get("transport", list()) + if len(transports) > 1: + logging.warning("Expected a maximum of one transport but %d are " + "given." % len(transports)) + + if len(transports) == 0: + bridgeType = "vanilla" + elif transports[0] == "" or transports[0] == "0": + bridgeType = "vanilla" + else: + bridgeType = transports[0] + + # BridgeDB's HTTPS interface exposes transport types as a drop down + # menu but users can still request anything by manipulating HTTP + # parameters. + if not isTransportSupported(bridgeType): + logging.warning("User requested unsupported transport type %s " + "over HTTPS." % bridgeType) + return + + logging.debug("Recording %svalid HTTPS request for %s from %s (%s)." % + ("" if success else "in", + bridgeType, ipAddr, countryCode)) + + # Now update our metrics. + key = self.createKey(self.keyPrefix, bridgeType, countryCode, + success, self.findAnomaly(request)) + self.inc(key) + + def recordValidHTTPSRequest(self, request): + self._recordHTTPSRequest(request, True) + + def recordInvalidHTTPSRequest(self, request): + self._recordHTTPSRequest(request, False) + + def updateSubnetCounter(self, ipAddr): + + if ipAddr is None: + return + + nw = ipaddr.IPNetwork(ipAddr + "/" + str(SUBNET_CTR_PREFIX_LEN), + strict=False) + subnet = nw.network.compressed + logging.debug("Updating subnet counter with %s" % subnet) + + num = self.subnetCounter.get(subnet, 0) + self.subnetCounter[subnet] = num + 1 + + +class EmailMetrics(Metrics): + + def __init__(self): + super(EmailMetrics, self).__init__() + self.keyPrefix = "email" + + def _recordEmailRequest(self, smtpAutoresp, success): + + emailAddrs = smtpAutoresp.getMailTo() + if len(emailAddrs) == 0: + # This is just for unit tests. + emailAddr = Address("foo(a)gmail.com") + else: + emailAddr = emailAddrs[0] + + # Get the requested transport protocol. + try: + br = request.determineBridgeRequestOptions( + smtpAutoresp.incoming.lines) + except EmailRequestedHelp: + return + bridgeType = "vanilla" if not len(br.transports) else br.transports[0] + + # Over email, transports are requested by typing them. Typos happen + # and users can request anything, really. + if not isTransportSupported(bridgeType): + logging.warning("User requested unsupported transport type %s " + "over email." % bridgeType) + return + + logging.debug("Recording %svalid email request for %s from %s." % + ("" if success else "in", bridgeType, emailAddr)) + sld = emailAddr.domain.split(".")[0] + + # Now update our metrics. + key = self.createKey(self.keyPrefix, bridgeType, sld, success, + self.findAnomaly(request)) + self.inc(key) + + def recordValidEmailRequest(self, smtpAutoresp): + self._recordEmailRequest(smtpAutoresp, True) + + def recordInvalidEmailRequest(self, smtpAutoresp): + self._recordEmailRequest(smtpAutoresp, False) + + +class MoatMetrics(Metrics): + + def __init__(self): + super(MoatMetrics, self).__init__() + self.keyPrefix = "moat" + + def _recordMoatRequest(self, request, success): + + logging.debug("Moat request has user agent: %s" % + request.requestHeaders.getRawHeaders("User-Agent")) + + ipAddr = getClientIP(request, + useForwardedHeader=True, + skipLoopback=False) + countryCode = resolveCountryCode(ipAddr) + + try: + encodedClientData = request.content.read() + clientData = json.loads(encodedClientData)["data"][0] + transport = clientData["transport"] + bridgeType = "vanilla" if not len(transport) else transport + except Exception as err: + logging.warning("Could not decode request: %s" % err) + return + + if not isTransportSupported(bridgeType): + logging.warning("User requested unsupported transport type %s " + "over moat." % bridgeType) + return + + logging.debug("Recording %svalid moat request for %s from %s (%s)." % + ("" if success else "in", + bridgeType, ipAddr, countryCode)) + + # Now update our metrics. + key = self.createKey(self.keyPrefix, bridgeType, + countryCode, success, self.findAnomaly(request)) + self.inc(key) + + def recordValidMoatRequest(self, request): + self._recordMoatRequest(request, True) + + def recordInvalidMoatRequest(self, request): + self._recordMoatRequest(request, False) diff --git a/bridgedb/test/test_metrics.py b/bridgedb/test/test_metrics.py new file mode 100644 index 0000000..a870fc2 --- /dev/null +++ b/bridgedb/test/test_metrics.py @@ -0,0 +1,204 @@ +# -*- coding: utf-8 ; test-case-name: bridgedb.test.test_metrics ; -*- +# _____________________________________________________________________________ +# +# This file is part of BridgeDB, a Tor bridge distribution system. +# +# :authors: please see included AUTHORS file +# :copyright: (c) 2019, The Tor Project, Inc. +# (c) 2019, Philipp Winter +# :license: see LICENSE for licensing information +# _____________________________________________________________________________ + +"""Unittests for the :mod:`bridgedb.metrics` module. + +These tests are meant to ensure that the :mod:`bridgedb.metrics` module is +functioning as expected. +""" + +import StringIO +import json +import os + +from bridgedb import metrics +from bridgedb.test.https_helpers import DummyRequest +from bridgedb.distributors.email.server import SMTPMessage +from bridgedb.test.email_helpers import _createMailServerContext +from bridgedb.test.email_helpers import _createConfig +from bridgedb.distributors.moat import server + +from twisted.trial import unittest +from twisted.test import proto_helpers + + +class StateTest(unittest.TestCase): + + def setUp(self): + self.topDir = os.getcwd().rstrip('_trial_temp') + self.captchaDir = os.path.join(self.topDir, 'captchas') + + # Clear all singletons before each test to prevent cross-test + # interference. + type(metrics.HTTPSMetrics()).clear() + type(metrics.EmailMetrics()).clear() + type(metrics.MoatMetrics()).clear() + + metrics.setSupportedTransports({ + 'obfs2': False, + 'obfs3': True, + 'obfs4': True, + 'scramblesuit': True, + 'fte': True, + }) + + self.metrix = metrics.HTTPSMetrics() + self.key = self.metrix.createKey("https", "obfs4", "de", True, None) + + def test_binning(self): + + key = self.metrix.createKey("https", "obfs4", "de", True, None) + self.metrix.coldMetrics = self.metrix.hotMetrics + + # A value of 1 should be rounded up to 10. + self.metrix.inc(key) + metrixLines = self.metrix.getMetrics() + key, value = metrixLines[0].split(" ") + self.assertTrue(int(value) == 10) + + # A value of 10 should remain 10. + self.metrix.set(key, 10) + metrixLines = self.metrix.getMetrics() + key, value = metrixLines[0].split(" ") + self.assertTrue(int(value) == 10) + + # A value of 11 should be rounded up to 20. + self.metrix.inc(key) + metrixLines = self.metrix.getMetrics() + key, value = metrixLines[0].split(" ") + self.assertTrue(int(value) == 20) + + def test_key_manipulation(self): + + self.metrix = metrics.HTTPSMetrics() + key = self.metrix.createKey("email", "obfs4", "de", True, "none") + self.assertTrue(key == "email.obfs4.de.success.none") + + self.metrix.inc(key) + self.assertEqual(self.metrix.hotMetrics[key], 1) + + self.metrix.set(key, 10) + self.assertEqual(self.metrix.hotMetrics[key], 10) + + def test_rotation(self): + + key = self.metrix.createKey("moat", "obfs4", "de", True, "none") + self.metrix.inc(key) + oldHotMetrics = self.metrix.hotMetrics + self.metrix.rotate() + + self.assertEqual(len(self.metrix.coldMetrics), 1) + self.assertEqual(len(self.metrix.hotMetrics), 0) + self.assertEqual(self.metrix.coldMetrics, oldHotMetrics) + + def test_export(self): + + self.metrix.inc(self.key) + + self.metrix.coldMetrics = self.metrix.hotMetrics + pseudo_fh = StringIO.StringIO() + metrics.export(pseudo_fh, 0) + + self.assertTrue(len(pseudo_fh.getvalue()) > 0) + + lines = pseudo_fh.getvalue().split("\n") + self.assertTrue(lines[0].startswith("bridgedb-stats-end")) + self.assertTrue(lines[1].startswith("bridgedb-stats-version")) + self.assertTrue(lines[2] == + "bridgedb-metric-count https.obfs4.de.success.None 10") + + def test_https_metrics(self): + + origFunc = metrics.resolveCountryCode + metrics.resolveCountryCode = lambda _: "US" + + key1 = "https.obfs4.us.success.none" + req1 = DummyRequest([b"bridges?transport=obfs4"]) + # We have to set the request args manually when using a DummyRequest. + req1.args.update({'transport': ['obfs4']}) + req1.getClientIP = lambda: "3.3.3.3" + + self.metrix.recordValidHTTPSRequest(req1) + self.assertTrue(self.metrix.hotMetrics[key1] == 1) + + key2 = "https.obfs4.us.fail.none" + req2 = DummyRequest([b"bridges?transport=obfs4"]) + # We have to set the request args manually when using a DummyRequest. + req2.args.update({'transport': ['obfs4']}) + req2.getClientIP = lambda: "3.3.3.3" + self.metrix.recordInvalidHTTPSRequest(req2) + self.assertTrue(self.metrix.hotMetrics[key2] == 1) + + metrics.resolveCountryCode = origFunc + + def test_email_metrics(self): + + config = _createConfig() + context = _createMailServerContext(config) + message = SMTPMessage(context) + message.lines = [ + "From: foo(a)gmail.com", + "To: bridges(a)torproject.org", + "Subject: testing", + "", + "get transport obfs4", + ] + + message.message = message.getIncomingMessage() + responder = message.responder + tr = proto_helpers.StringTransportWithDisconnection() + tr.protocol = responder + responder.makeConnection(tr) + + email_metrix = metrics.EmailMetrics() + + key1 = "email.obfs4.gmail.success.none" + email_metrix.recordValidEmailRequest(responder) + self.assertTrue(email_metrix.hotMetrics[key1] == 1) + + key2 = "email.obfs4.gmail.fail.none" + email_metrix.recordInvalidEmailRequest(responder) + self.assertTrue(email_metrix.hotMetrics[key2] == 1) + + def test_moat_metrics(self): + + def create_moat_request(): + encoded_data = json.dumps({ + 'data': [{ + 'id': '2', + 'type': 'moat-solution', + 'version': server.MOAT_API_VERSION, + 'transport': 'obfs4', + 'solution': 'Tvx74PMy', + 'qrcode': False, + }] + }) + + request = DummyRequest(["fetch"]) + request.requestHeaders.addRawHeader('Content-Type', + 'application/vnd.api+json') + request.requestHeaders.addRawHeader('Accept', + 'application/vnd.api+json') + request.requestHeaders.addRawHeader('X-Forwarded-For', '3.3.3.3') + request.headers['X-Forwarded-For'.lower()] = '3.3.3.3' + request.method = b'POST' + request.writeContent(encoded_data) + + return request + + metrix = metrics.MoatMetrics() + metrix.recordValidMoatRequest(create_moat_request()) + metrix.recordInvalidMoatRequest(create_moat_request()) + + key1 = "moat.obfs4.us.success.none" + key2 = "moat.obfs4.us.fail.none" + self.assertTrue(metrix.hotMetrics[key1] == 1) + self.assertTrue(metrix.hotMetrics[key2] == 1)

1 0

[bridgedb/master] Merge branch 'fix/22755' into develop
by phw＠torproject.org 20 Aug '19

20 Aug '19

commit 0a1de3f58abe20f73510a43198365fb1fb9f9580 Merge: 46dde4c ed37530 Author: Philipp Winter <phw(a)nymity.ch> Date: Thu Aug 15 14:57:20 2019 -0700 Merge branch 'fix/22755' into develop .test.requirements.txt | 1 - .travis.requirements.txt | 1 - CHANGELOG | 4 + README.rst | 6 +- bridgedb/main.py | 10 +- bridgedb/runner.py | 37 +----- doc/HACKING.md | 10 +- scripts/create_descriptors | 298 +++++++++++++++++++++++++++++++++++++++++++++ scripts/setup-tests | 2 +- setup.py | 2 +- 10 files changed, 316 insertions(+), 55 deletions(-)

1 0

[translation/support-portal] Update translations for support-portal
by translation＠torproject.org 20 Aug '19

20 Aug '19

commit 8a4b40cf6b870f41e00f3522d10507e3aa47b833 Author: Translation commit bot <translation(a)torproject.org> Date: Tue Aug 20 15:59:52 2019 +0000 Update translations for support-portal --- contents+zh-CN.po | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/contents+zh-CN.po b/contents+zh-CN.po index 333d83a88..d20b5161e 100644 --- a/contents+zh-CN.po +++ b/contents+zh-CN.po @@ -3707,6 +3707,8 @@ msgid "" "connecting to (website, chat service, email provider, etc..) will see the " "[IP address](#ip-address) of the exit." msgstr "" +"在[Tor链路](#circuit)中最后一个[中继](#relay)会发送[流量](#traffic) " +"到公共互联网。你连接的服务（网站、聊天服务器、电子邮件提供商等等…）会看到出口的[IP地址](#ip-address)。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description) @@ -3722,6 +3724,9 @@ msgid "" "running on a given IP address on a given date. This service is often useful" " when dealing with law enforcement." msgstr "" +"ExoneraTor 服务维护了一个已成为 Tor 网络一部分的[中继](#relay)的[IP 地址](#ip-address) " +"的数据库。它将回答[Tor](#tor-/-tor-network/-core-" +"tor)中继在给定的日期是否运行在给定的IP地址上。这个服务在应对执法部门时常常是很管用的。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description) @@ -3757,6 +3762,8 @@ msgid "" "](#operating-system-os), with its mobile version (fennec) available for " "Android." msgstr "" +"Firefox 可使用于Windows, macOS和Linux [操作系统](#operating-system-" +"os)，同时其手机版(fennec)可适用于Android。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description) @@ -3808,7 +3815,7 @@ msgstr "### fte" msgid "" "FTE (format-transforming encryption) is a pluggable transport that disguises" " [Tor traffic](#traffic) as ordinary web (HTTP) traffic." -msgstr "" +msgstr "FTE（变形加密）是一个将[Tor流量](#traffic)混淆为普通网络（HTTP）流量的可插拔传输。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description) @@ -3827,6 +3834,8 @@ msgid "" "Twitter) with links to the latest version of [Tor Browser](#tor-browser), " "hosted at a variety of locations, such as Dropbox, Google Drive and GitHub." msgstr "" +"这是一种自动接收消息（电子邮件、XMPP、推特）并响应存储在多地如Dropbox，Google Drive和Github的 [Tor 浏览器" +"](#tor-browser)最新版本链接的服务。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description) @@ -3838,7 +3847,7 @@ msgstr "### GSoC" msgid "" "The Tor Project participates in the Google Summer of Code, which is a summer" " program for university students." -msgstr "" +msgstr "Tor 项目参与了Google编程之夏，这是一个针对大学生的暑期编程活动。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description)

1 0

[translation/support-portal] Update translations for support-portal
by translation＠torproject.org 20 Aug '19

20 Aug '19

commit 41383f7f5d11a1986e429e820670909feb358247 Author: Translation commit bot <translation(a)torproject.org> Date: Tue Aug 20 15:29:41 2019 +0000 Update translations for support-portal --- contents+zh-CN.po | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/contents+zh-CN.po b/contents+zh-CN.po index ec3449c20..333d83a88 100644 --- a/contents+zh-CN.po +++ b/contents+zh-CN.po @@ -308,7 +308,7 @@ msgid "" "accompanied by a file with the same name as the package and the extension " "\".asc\". These .asc files are OpenPGP signatures." msgstr "" -"我们[下载页面](https://www.torproject.org/download/)上的每个文件都附带一个与包名称相同的文件和扩展名“.asc…" +"我们[下载页面](https://www.torproject.org/download/)上的每个文件都附带一个与包名称相同的文件和扩展名“.asc…" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) @@ -402,7 +402,7 @@ msgstr "如果您正在使用macOS，您可以[安装 GPGTools](https://www.gpgt msgid "" "In order to verify the signature you will need to type a few commands in the" " Terminal (under \"Applications\")." -msgstr "" +msgstr "为了验证签名，您需要在（“应用程序”下的）终端中输入一些命令" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) @@ -414,14 +414,14 @@ msgstr "#### 对于 GNU/Linux 的用户：" msgid "" "If you are using GNU/Linux, then you probably already have GnuPG in your " "system, as most GNU/Linux distributions come with it preinstalled." -msgstr "" +msgstr "如果你使用 GNU/Linux，那么可能在你的系统中已经安装了 GnuPG，因为大多数 Linux 发行版都预装了它。" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) msgid "" "In order to verify the signature you will need to type a few commands in a " "terminal window. How to do this will vary depending on your distribution." -msgstr "" +msgstr "为了验证签名，您需要在终端窗口中输入一些命令。如何进行此操作将取决于您的发行版。" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) @@ -450,7 +450,7 @@ msgstr "" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) msgid "This should show you something like:" -msgstr "" +msgstr "这会向您展示像这样的内容：" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) @@ -521,14 +521,14 @@ msgid "" "download the corresponding \".asc\" signature file as well as the installer " "file itself, and verify it with a command that asks GnuPG to verify the file" " that you downloaded." -msgstr "" +msgstr "为了验证你下载的包的签名，除了安装文件本身，你还需要下载相应的“.asc”签名文件，并用一个命令让GnuPG验证你下载的文件。" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) msgid "" "The examples below assume that you downloaded these two files to your " "\"Downloads\" folder." -msgstr "" +msgstr "下面的例子假设你已经下载了这样的两个文件到你的\"下载\"文件夹。" #: https//support.torproject.org/tbb/how-to-verify-signature/ #: (content/tbb/how-to-verify-signature/contents+en.lrquestion.description) @@ -590,7 +590,7 @@ msgstr "" msgid "" "You may also want to [learn more about " "GnuPG](https://www.gnupg.org/documentation/)." -msgstr "" +msgstr "你也许会想了解[更多关于GnuPG](https://www.gnupg.org/documentation/)。" #: https//support.torproject.org/tbb/tbb-1/ #: (content/tbb/tbb-1/contents+en.lrquestion.title) @@ -2749,6 +2749,7 @@ msgstr "#Address noname.example.com" msgid "" "## A handle for your relay, so people don't have to refer to it by key." msgstr "" +"## A handle for your relay, so people don't have to refer to it by key." #: https//support.torproject.org/operators/operators-1/ #: (content/operators/operators-1/contents+en.lrquestion.description) @@ -2931,6 +2932,9 @@ msgid "" "href=\"https://metrics.torproject.org/rs.html#search\">Metrics</a></mark> to" " see whether your relay has successfully registered in the network." msgstr "" +"在几个小时后（让它有足够的时间来传播），你可以查询<mark><a " +"href=\"https://metrics.torproject.org/rs.html#search\">Metrics</a></mark> " +"来确认你的中继是否在网络中注册。" #: https//support.torproject.org/operators/operators-1/ #: (content/operators/operators-1/contents+en.lrquestion.description) @@ -3039,6 +3043,8 @@ msgid "" "* Look for a log entry in /var/log/tor/notices.log such as \"Self-testing " "indicates your ORPort is reachable from the outside. Excellent.\"" msgstr "" +"* 查找 /var/log/tor/notices.log 中例如 \"Self-testing indicates your ORPort is " +"reachable from the outside. Excellent.\" 的日志记录。" #: https//support.torproject.org/operators/operators-4/ #: (content/operators/operators-4/contents+en.lrquestion.title) @@ -3068,7 +3074,7 @@ msgstr "$ lsb_release -c" msgid "" "* As root, add the following lines to /etc/apt/sources.list. Replace " "'version' with the version you found in the previous step:" -msgstr "" +msgstr "* 以 root 用户身份把下面的行添加到 /etc/apt/sources.list 中。用前一步你获得的版本号代替'version'。" #: https//support.torproject.org/operators/operators-4/ #: (content/operators/operators-4/contents+en.lrquestion.description) @@ -3243,7 +3249,7 @@ msgstr "" msgid "" "These services use the special-use top level domain (TLD) .onion (instead of" " .com, .net, .org, etc..) and are only accessible through the Tor network." -msgstr "" +msgstr "这些服务使用特殊的顶级域名（TLD） .onion （而不是.com .net .org等）而且这些服务只有在 Tor 网络里可以连接。" #: https//support.torproject.org/onionservices/onionservices-2/ #: (content/onionservices/onionservices-2/contents+en.lrquestion.description) @@ -3403,7 +3409,7 @@ msgid "" "Like ordinary Tor [relays](#relay), bridges are run by volunteers; unlike " "ordinary relays, however, they are not listed publicly, so an adversary " "cannot identify them easily." -msgstr "" +msgstr "如同普通的Tor [中继](#relay)，网桥由志愿者经营; 然而，与普通中继不同，它们并未公开列出，因此对手无法轻易识别它们。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description) @@ -3504,7 +3510,7 @@ msgid "" "Checksums are [hash](#hash)values of files. If you have downloaded the " "software without errors, the given checksum and the checksum of your " "downloaded file will be identical." -msgstr "" +msgstr "校验和是文件的[hash](#hash)值。如果您下载的软件没有错误，给定的校验和与您下载文件的校验和将是相同的。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description) @@ -3620,6 +3626,9 @@ msgid "" "signatures, please see [here](https://support.torproject.org/tbb/how-to-" "verify-signature/)." msgstr "" +"密码学签名证明消息或文件的真实性。它是由[公钥密码学](#public-key-" +"cryptography)中密钥对的私有部分的持有者创建的，可以由相应的公钥进行验证。如果你从torproject.org下载软件，你会发现它是一个签…" +"/how-to-verify-signature/)。" #: https//support.torproject.org/misc/glossary/ #: (content/misc/glossary/contents+en.lrquestion.description)

1 0