August 2018 - tor-commits - lists.torproject.org

[webwml/master] update press clips (fixes #26403)
by hiro＠torproject.org 10 Aug '18

10 Aug '18

commit ea87dd7c810e074e6f5c3bd659ff9f58d45da108 Author: Traumschule <traumschule(a)riseup.net> Date: Fri Aug 10 18:44:14 2018 +0200 update press clips (fixes #26403) --- press/en/press.wml | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/press/en/press.wml b/press/en/press.wml index bbec43cb..95c58897 100644 --- a/press/en/press.wml +++ b/press/en/press.wml @@ -51,6 +51,36 @@ some of the significant Tor-related stories that have popped up.</p> </thead> <tr style="background-color: #e5e5e5;"> +<td>2018 July 02</td> +<td>Mada Masr</td> +<td><a +href="https://www.madamasr.com/en/2018/07/02/news/u/report-internet-censorship-ha…">Report: +Internet censorship has become a "more pervasiveâ" in past year</a></td> +</tr> + +<tr style=""> +<td>2018 June 28</td> +<td>CNET</td> +<td><a href="https://www.cnet.com/news/brave-advances-browser-privacy-with-tor-powered-t…">Brave +advances browser privacy with Tor-powered tabs</a></td> +</tr> + +<tr style="background-color: #e5e5e5;"> +<td>2018 June 27</td> +<td>ZDNet</td> +<td><a href="https://www.zdnet.com/article/venezuela-ramps-up-censorship-with-tor-ban/">Venezuela ramps +up censorship</a></td> +</tr> + +<tr style=""> +<td>2018 June 26</td> +<td>VICE</td> +<td><a +href="https://news.vice.com/en_us/article/59qvwz/venezuela-maduro-tor-network-cen…">Venezuela just +took a huge step towards controlling all access to the Internet</a></td> +</tr> + +<tr style="background-color: #e5e5e5;"> <td>2018 May 30</td> <td>ZDNet</td> <td><a href="https://www.zdnet.com/article/technology-that-changed-us-the-2000s/">Technology that changed us: The 2000s, from iPhone to Twitter</a></td>

1 0

[webwml/master] Fix merge conflict
by hiro＠torproject.org 10 Aug '18

10 Aug '18

commit c26087d3ee220629f59f01581b36ad09f691be93 Merge: 56c95f5e ea87dd7c Author: hiro <hiro(a)torproject.org> Date: Fri Aug 10 19:09:14 2018 +0200 Fix merge conflict press/en/press.wml | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --cc press/en/press.wml index 29e187c7,95c58897..6a259c12 --- a/press/en/press.wml +++ b/press/en/press.wml @@@ -50,7 -50,37 +50,38 @@@ some of the significant Tor-related sto </tr> </thead> + <tr style="background-color: #e5e5e5;"> + <td>2018 July 02</td> + <td>Mada Masr</td> -<td><a -href="https://www.madamasr.com/en/2018/07/02/news/u/report-internet-censorship-ha…">Report: ++<td><a ++href="https://www.madamasr.com/en/2018/07/02/news/u/report-internet-censorship-ha…">Report: + Internet censorship has become a "more pervasiveâ" in past year</a></td> + </tr> + <tr style=""> + <td>2018 June 28</td> + <td>CNET</td> -<td><a href="https://www.cnet.com/news/brave-advances-browser-privacy-with-tor-powered-t…">Brave ++<td><a href="https://www.cnet.com/news/brave-advances-browser-privacy-with-tor-powered-t…">Brave + advances browser privacy with Tor-powered tabs</a></td> + </tr> + + <tr style="background-color: #e5e5e5;"> + <td>2018 June 27</td> + <td>ZDNet</td> -<td><a href="https://www.zdnet.com/article/venezuela-ramps-up-censorship-with-tor-ban/">Venezuela ramps ++<td><a href="https://www.zdnet.com/article/venezuela-ramps-up-censorship-with-tor-ban/">Venezuela ramps + up censorship</a></td> + </tr> + + <tr style=""> + <td>2018 June 26</td> + <td>VICE</td> -<td><a -href="https://news.vice.com/en_us/article/59qvwz/venezuela-maduro-tor-network-cen…">Venezuela just ++<td><a ++href="https://news.vice.com/en_us/article/59qvwz/venezuela-maduro-tor-network-cen…">Venezuela just + took a huge step towards controlling all access to the Internet</a></td> + </tr> + + <tr style="background-color: #e5e5e5;"> ++>>>>>>> ea87dd7c810e074e6f5c3bd659ff9f58d45da108 <td>2018 May 30</td> <td>ZDNet</td> <td><a href="https://www.zdnet.com/article/technology-that-changed-us-the-2000s/">Technology that changed us: The 2000s, from iPhone to Twitter</a></td>

1 0

[tor/master] Merge remote-tracking branch 'tor-github/pr/239'
by nickm＠torproject.org 10 Aug '18

10 Aug '18

commit b7ed61167fb3ed9d3b623d008bf3360a3ffa95c4 Merge: 5980cb8a1 a207511bb Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Aug 10 12:35:06 2018 -0400 Merge remote-tracking branch 'tor-github/pr/239' changes/bug18642 | 5 +++++ src/core/or/relay.c | 8 ++++++++ src/feature/relay/dns.c | 41 +++++++++++++++++++++++++++++++++++++++-- src/feature/relay/dns.h | 2 ++ 4 files changed, 54 insertions(+), 2 deletions(-)

1 0

[tor/master] In cell_queues_check_size(), add DNS cache size to total memory allocation
by nickm＠torproject.org 10 Aug '18

10 Aug '18

commit a207511bb0cc86b20a5ed682898a74660efdd5b2 Author: Neel Chauhan <neel(a)neelc.org> Date: Wed Aug 8 20:52:57 2018 -0400 In cell_queues_check_size(), add DNS cache size to total memory allocation --- src/core/or/relay.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/core/or/relay.c b/src/core/or/relay.c index 62a4bca84..2a6bedcb8 100644 --- a/src/core/or/relay.c +++ b/src/core/or/relay.c @@ -2539,6 +2539,7 @@ cell_queues_check_size(void) geoip_client_cache_total_allocation(); alloc += geoip_client_cache_total; const size_t dns_cache_total = dns_cache_total_allocation(); + alloc += dns_cache_total; if (alloc >= get_options()->MaxMemInQueues_low_threshold) { last_time_under_memory_pressure = approx_time(); if (alloc >= get_options()->MaxMemInQueues) {

1 0

[tor/master] Add changes file for Bug #18642
by nickm＠torproject.org 10 Aug '18

10 Aug '18

commit 7c052d4ac38605092534e95daf69f208fadcafdc Author: Neel Chauhan <neel(a)neelc.org> Date: Mon Jul 16 17:37:24 2018 -0400 Add changes file for Bug #18642 --- changes/bug18642 | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/changes/bug18642 b/changes/bug18642 new file mode 100644 index 000000000..6b71f3c6d --- /dev/null +++ b/changes/bug18642 @@ -0,0 +1,5 @@ + o Minor features (denial-of-service avoidance): + - Make our OOM handler aware of the DNS cache so that it doesn't fill up + the memory. This check is important for our DoS mitigation subsystem. + Closes ticket 18642. Patch by Neel Chauhan +

1 0

[tor/master] Teach the OOM handler about the DNS cache
by nickm＠torproject.org 10 Aug '18

10 Aug '18

commit 32db806dfa85b016a2d56d0d5248f17c34b690c4 Author: Neel Chauhan <neel(a)neelc.org> Date: Mon Jul 16 17:36:22 2018 -0400 Teach the OOM handler about the DNS cache --- src/core/or/relay.c | 7 +++++++ src/feature/relay/dns.c | 30 ++++++++++++++++++++++++++++++ src/feature/relay/dns.h | 1 + 3 files changed, 38 insertions(+) diff --git a/src/core/or/relay.c b/src/core/or/relay.c index 32bb69d25..62a4bca84 100644 --- a/src/core/or/relay.c +++ b/src/core/or/relay.c @@ -64,6 +64,7 @@ #include "lib/crypt_ops/crypto_rand.h" #include "lib/crypt_ops/crypto_util.h" #include "feature/dircache/directory.h" +#include "feature/relay/dns.h" #include "feature/stats/geoip.h" #include "feature/hs/hs_cache.h" #include "core/mainloop/main.h" @@ -2537,6 +2538,7 @@ cell_queues_check_size(void) const size_t geoip_client_cache_total = geoip_client_cache_total_allocation(); alloc += geoip_client_cache_total; + const size_t dns_cache_total = dns_cache_total_allocation(); if (alloc >= get_options()->MaxMemInQueues_low_threshold) { last_time_under_memory_pressure = approx_time(); if (alloc >= get_options()->MaxMemInQueues) { @@ -2554,6 +2556,11 @@ cell_queues_check_size(void) (size_t)(get_options()->MaxMemInQueues / 10); alloc -= geoip_client_cache_handle_oom(now, bytes_to_remove); } + if (dns_cache_total > get_options()->MaxMemInQueues / 5) { + const size_t bytes_to_remove = + dns_cache_total - (size_t)(get_options()->MaxMemInQueues / 10); + alloc -= dns_cache_handle_oom(now, bytes_to_remove); + } circuits_handle_oom(alloc); return 1; } diff --git a/src/feature/relay/dns.c b/src/feature/relay/dns.c index 8a49829d5..48319b47c 100644 --- a/src/feature/relay/dns.c +++ b/src/feature/relay/dns.c @@ -2100,6 +2100,36 @@ dump_dns_mem_usage(int severity) (unsigned)hash_mem); } +/* Do a round of OOM cleanup on all DNS entries. Return the amount of removed + * bytes. It is possible that the returned value is lower than min_remove_bytes + * if the caches get emptied out so the caller should be aware of this. */ +size_t +dns_cache_handle_oom(time_t now, size_t min_remove_bytes) +{ + time_t time_inc = 0; + size_t total_bytes_removed = 0; + size_t current_size = dns_cache_total_allocation(); + + do { + /* If no DNS entries left, break loop. */ + if (!dns_cache_entry_count()) + break; + + /* Get cutoff interval and remove entries. */ + time_t cutoff = now + time_inc; + purge_expired_resolves(cutoff); + + /* Update amount of bytes removed and array size. */ + size_t bytes_removed = current_size - dns_cache_total_allocation(); + current_size -= bytes_removed; + total_bytes_removed += bytes_removed; + + time_inc += 3600; /* Increase time_inc by 1 hour. */ + } while (total_bytes_removed < min_remove_bytes); + + return total_bytes_removed; +} + #ifdef DEBUG_DNS_CACHE /** Exit with an assertion if the DNS cache is corrupt. */ static void diff --git a/src/feature/relay/dns.h b/src/feature/relay/dns.h index 1d5266c73..1dd6f903d 100644 --- a/src/feature/relay/dns.h +++ b/src/feature/relay/dns.h @@ -40,6 +40,7 @@ int dns_seems_to_be_broken_for_ipv6(void); void dns_reset_correctness_checks(void); size_t dns_cache_total_allocation(void); void dump_dns_mem_usage(int severity); +size_t dns_cache_handle_oom(time_t now, size_t min_remove_bytes); #ifdef DNS_PRIVATE #include "feature/relay/dns_structs.h"

1 0

[tor/master] Add and use dns_cache_total_allocation()
by nickm＠torproject.org 10 Aug '18

10 Aug '18

commit d807ca1b014e27a2808959dba1a37a0c1d53240f Author: Neel Chauhan <neel(a)neelc.org> Date: Mon Jul 16 17:19:53 2018 -0400 Add and use dns_cache_total_allocation() --- src/feature/relay/dns.c | 11 +++++++++-- src/feature/relay/dns.h | 1 + 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/src/feature/relay/dns.c b/src/feature/relay/dns.c index f4e39dfd3..8a49829d5 100644 --- a/src/feature/relay/dns.c +++ b/src/feature/relay/dns.c @@ -2076,14 +2076,21 @@ dns_cache_entry_count(void) return HT_SIZE(&cache_root); } +/* Return the total size in bytes of the DNS cache. */ +size_t +dns_cache_total_allocation(void) +{ + return sizeof(struct cached_resolve_t) * dns_cache_entry_count() + + HT_MEM_USAGE(&cache_root); +} + /** Log memory information about our internal DNS cache at level 'severity'. */ void dump_dns_mem_usage(int severity) { /* This should never be larger than INT_MAX. */ int hash_count = dns_cache_entry_count(); - size_t hash_mem = sizeof(struct cached_resolve_t) * hash_count; - hash_mem += HT_MEM_USAGE(&cache_root); + size_t hash_mem = dns_cache_total_allocation(); /* Print out the count and estimated size of our &cache_root. It undercounts hostnames in cached reverse resolves. diff --git a/src/feature/relay/dns.h b/src/feature/relay/dns.h index 69c1764b7..1d5266c73 100644 --- a/src/feature/relay/dns.h +++ b/src/feature/relay/dns.h @@ -38,6 +38,7 @@ void dns_launch_correctness_checks(void); int dns_seems_to_be_broken(void); int dns_seems_to_be_broken_for_ipv6(void); void dns_reset_correctness_checks(void); +size_t dns_cache_total_allocation(void); void dump_dns_mem_usage(int severity); #ifdef DNS_PRIVATE

1 0

[stem/master] Test for broken event listeners
by atagar＠torproject.org 09 Aug '18

09 Aug '18

commit b557d09c2721858785b6310f331e29c06b76887f Author: Damian Johnson <atagar(a)torproject.org> Date: Thu Aug 9 12:43:03 2018 -0700 Test for broken event listeners Mike made a great point that our event handling needs more error handling. In particular errors from event listeners and malformed events from tor can cause our internal event thread to break, preventing further event processing... https://trac.torproject.org/projects/tor/ticket/27053 Not yet fixed. Starting with a couple unit tests that repro these oversights. --- test/unit/control/controller.py | 85 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 84 insertions(+), 1 deletion(-) diff --git a/test/unit/control/controller.py b/test/unit/control/controller.py index 6f162899..fa934f34 100644 --- a/test/unit/control/controller.py +++ b/test/unit/control/controller.py @@ -5,6 +5,7 @@ integ tests, but a few bits lend themselves to unit testing. import datetime import io +import time import unittest import stem.descriptor.router_status_entry @@ -25,6 +26,11 @@ except ImportError: from mock import Mock, patch NS_DESC = 'r %s %s u5lTXJKGsLKufRLnSyVqT7TdGYw 2012-12-30 22:02:49 77.223.43.54 9001 0\ns Fast Named Running Stable Valid\nw Bandwidth=75' +TEST_TIMESTAMP = 12345 + +BW_EVENT = ControlMessage.from_str('650 BW 15 25', 'EVENT', normalize = True, arrived_at = TEST_TIMESTAMP) +CIRC_EVENT = ControlMessage.from_str('650 CIRC 4 LAUNCHED', 'EVENT', normalize = True, arrived_at = TEST_TIMESTAMP) +BAD_EVENT = ControlMessage.from_str('650 BW &15* 25', normalize = True, arrived_at = TEST_TIMESTAMP) class TestControl(unittest.TestCase): @@ -37,6 +43,12 @@ class TestControl(unittest.TestCase): with patch('stem.control.BaseController.msg', Mock()): self.controller = Controller(socket) + self.circ_listener = Mock() + self.controller.add_event_listener(self.circ_listener, EventType.CIRC) + + self.bw_listener = Mock() + self.controller.add_event_listener(self.bw_listener, EventType.BW) + def test_event_description(self): self.assertEqual("Logging at the debug runlevel. This is low level, high volume information about tor's internals that generally isn't useful to users.", stem.control.event_description('DEBUG')) self.assertEqual('Event emitted every second with the bytes sent and received by tor.', stem.control.event_description('BW')) @@ -586,7 +598,7 @@ class TestControl(unittest.TestCase): @patch('stem.control.Controller.is_authenticated', Mock(return_value = True)) @patch('stem.control.Controller._attach_listeners', Mock(return_value = ([], []))) @patch('stem.control.Controller.get_version') - def test_event_listening(self, get_version_mock): + def test_add_event_listener(self, get_version_mock): """ Exercises the add_event_listener and remove_event_listener methods. """ @@ -608,6 +620,64 @@ class TestControl(unittest.TestCase): self.assertRaises(InvalidRequest, self.controller.add_event_listener, Mock(), EventType.SIGNAL) + @patch('time.time', Mock(return_value = TEST_TIMESTAMP)) + @patch('stem.control.Controller.is_alive', Mock(return_value = True)) + def test_events_get_received(self): + """ + Trigger an event, checking that our listeners get notified. + """ + + self.controller._launch_threads() + + self._emit_event(CIRC_EVENT) + self.circ_listener.assert_called_once_with(CIRC_EVENT) + self.bw_listener.assert_not_called() + + self._emit_event(BW_EVENT) + self.bw_listener.assert_called_once_with(BW_EVENT) + + @patch('time.time', Mock(return_value = TEST_TIMESTAMP)) + @patch('stem.control.Controller.is_alive', Mock(return_value = True)) + def test_event_listing_with_error(self): + """ + Raise an exception in an event listener to confirm it doesn't break our + event thread. + """ + + self.skipTest('BUG: stem not working in this regard, about to fix') + return + + self.circ_listener.side_effect = ValueError('boom') + + self.controller._launch_threads() + + self._emit_event(CIRC_EVENT) + self.circ_listener.assert_called_once_with(CIRC_EVENT) + self.bw_listener.assert_not_called() + + self._emit_event(BW_EVENT) + self.bw_listener.assert_called_once_with(BW_EVENT) + + @patch('time.time', Mock(return_value = TEST_TIMESTAMP)) + @patch('stem.control.Controller.is_alive', Mock(return_value = True)) + def test_event_listing_with_malformed_event(self): + """ + Attempt to parse a malformed event emitted from Tor. It's important this + doesn't break our event thread. + """ + + self.skipTest('BUG: stem not working in this regard, about to fix') + return + + self.controller._launch_threads() + + self._emit_event(BAD_EVENT) + self.circ_listener.assert_called_once_with(CIRC_EVENT) + self.bw_listener.assert_not_called() + + self._emit_event(BW_EVENT) + self.bw_listener.assert_called_once_with(BW_EVENT) + @patch('stem.control.Controller.get_version', Mock(return_value = stem.version.Version('0.5.0.14'))) @patch('stem.control.Controller.msg', Mock(return_value = ControlMessage.from_str('250 OK\r\n'))) @patch('stem.control.Controller.add_event_listener', Mock()) @@ -732,3 +802,16 @@ class TestControl(unittest.TestCase): with patch('stem.control.Controller.msg', Mock(return_value = None)): get_version_mock.return_value = stem.version.Version('0.2.5.2') self.controller.drop_guards() + + def _emit_event(self, event): + # Converting an event back into an uncast ControlMessage, then feeding it + # into our controller's event queue. + + uncast_event = ControlMessage.from_str(event.raw_content()) + self.controller._event_queue.put(uncast_event) + self.controller._event_notice.set() + + # doesn't seem necessary in practice, but since event processing is + # asynchronous giving it a tiny bit of time to get handled + + time.sleep(0.005)

1 0

[sbws/master] Remove is_innited dependency
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 4c6eb9e83572e01049893909b3f935e1092186ac Author: juga0 <juga(a)riseup.net> Date: Wed Jul 18 19:51:28 2018 +0000 Remove is_innited dependency --- sbws/core/scanner.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index 5e98e99..f7a08d7 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -9,7 +9,7 @@ from ..lib.relayprioritizer import RelayPrioritizer from ..lib.destination import DestinationList from ..util.timestamp import now_isodt_str from ..util.state import State -from sbws.globals import (fail_hard, is_initted) +from sbws.globals import fail_hard import sbws.util.stem as stem_utils import sbws.util.requests as requests_utils from argparse import ArgumentDefaultsHelpFormatter @@ -367,9 +367,6 @@ def gen_parser(sub): def main(args, conf): - if not is_initted(args.directory): - fail_hard('Sbws isn\'t initialized. Try sbws init') - if conf.getint('scanner', 'measurement_threads') < 1: fail_hard('Number of measurement threads must be larger than 1')

1 0

[sbws/master] Remove is_initted and add config paths
by pastly＠torproject.org 09 Aug '18

09 Aug '18

commit 4e73693a132a28e53f9f646c0841fd344bace785 Author: juga0 <juga(a)riseup.net> Date: Wed Jul 18 19:56:08 2018 +0000 Remove is_initted and add config paths --- sbws/globals.py | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/sbws/globals.py b/sbws/globals.py index a00cfb1..56597b1 100644 --- a/sbws/globals.py +++ b/sbws/globals.py @@ -8,10 +8,6 @@ RESULT_VERSION = 4 WIRE_VERSION = 1 SPEC_VERSION = '1.1.0' -PKG_DIR = os.path.abspath(os.path.dirname(__file__)) - -SOCKET_TIMEOUT = 60 # seconds - # This is a dictionary of torrc options we always want to set when launching # Tor and that do not depend on any runtime configuration TORRC_STARTING_POINT = { @@ -27,23 +23,17 @@ TORRC_STARTING_POINT = { 'UseMicrodescriptors': '0', } +PKG_DIR = os.path.abspath(os.path.dirname(__file__)) +DEFAULT_CONFIG_PATH = os.path.join(PKG_DIR, 'config.default.ini') +DEFAULT_LOG_CONFIG_PATH = os.path.join(PKG_DIR, 'config.log.default.ini') +MINIMUM_USER_CONFIG_PATH = os.path.join(PKG_DIR, 'config.user.ini') +USER_CONFIG_PATH = os.path.join(os.path.expanduser('~'), '.sbws.ini') + +SOCKET_TIMEOUT = 60 # seconds SCALE_CONSTANT = 7500 BW_LINE_SIZE = 510 -def is_initted(d): - if not os.path.isdir(d): - log.debug('%s not initialized: %s doesn\'t exist', d, d) - return False - conf_fnames = [os.path.join(d, 'config.ini')] - for fname in conf_fnames: - if not os.path.isfile(fname): - log.debug('%s not initialized: missing %s', d, fname) - return False - log.debug('%s seems initialized.', d) - return True - - def fail_hard(*a, **kw): ''' Log something ... and then exit as fast as possible ''' log.critical(*a, **kw)

1 0