March 2015 - tor-commits - lists.torproject.org

[arm/master] Close controller connection last
by atagar＠torproject.org 15 Mar '15

15 Mar '15

commit a1cceb2d230fa5c4bd8a6ff1f6f38227e0648471 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 15 15:16:51 2015 -0700 Close controller connection last Closing wthe controller while panels are still trying to do work naturally causes a lot of noise (since their controller connection is swiped out from under them), and masks other issues. Might as well do this last. --- seth/starter.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/seth/starter.py b/seth/starter.py index 2b98706..abb4eec 100644 --- a/seth/starter.py +++ b/seth/starter.py @@ -279,19 +279,16 @@ def _shutdown_daemons(controller): Stops and joins on worker threads. """ - close_controller = threading.Thread(target = controller.close) - close_controller.setDaemon(True) - close_controller.start() - halt_threads = [ seth.controller.stop_controller(), seth.util.tracker.stop_trackers(), - close_controller, ] for thread in halt_threads: thread.join() + controller.close() + if __name__ == '__main__': main()

1 0

[arm/master] The start_time is a Stat attribute, not Category
by atagar＠torproject.org 15 Mar '15

15 Mar '15

commit 1e55b1be10342ee22225fcf843f2d3b34154ed14 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 15 15:37:44 2015 -0700 The start_time is a Stat attribute, not Category On small screens we balked with... Traceback (most recent call last): File "./run_seth", line 47, in <module> main() File "./run_seth", line 16, in main seth.starter.main() File "/usr/lib/python2.7/stem/util/conf.py", line 288, in wrapped return func(*args, config = config, **kwargs) File "/home/atagar/Desktop/seth/seth/starter.py", line 92, in main curses.wrapper(seth.controller.start_seth) File "/usr/lib/python2.7/curses/wrapper.py", line 43, in wrapper return func(stdscr, *args, **kwds) File "/home/atagar/Desktop/seth/seth/controller.py", line 570, in start_seth control.redraw(False) File "/home/atagar/Desktop/seth/seth/controller.py", line 401, in redraw panel_impl.redraw(force) File "/home/atagar/Desktop/seth/seth/util/panel.py", line 442, in redraw self.draw(self.max_x, self.max_y) File "/home/atagar/Desktop/seth/seth/graph_panel.py", line 641, in draw runtime = time.time() - param.start_time AttributeError: 'BandwidthStats' object has no attribute 'start_time' --- seth/graph_panel.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/seth/graph_panel.py b/seth/graph_panel.py index 1e248a3..d3097e6 100644 --- a/seth/graph_panel.py +++ b/seth/graph_panel.py @@ -638,7 +638,7 @@ class GraphPanel(panel.Panel): self.addstr(labeling_line, 0, ' ' * width) graph_column = min((width - 10) / 2, CONFIG['features.graph.max_width']) - runtime = time.time() - param.start_time + runtime = time.time() - param.primary.start_time primary_footer = 'total: %s, avg: %s/sec' % (param._size_label(param.primary.total), param._size_label(param.primary.total / runtime)) secondary_footer = 'total: %s, avg: %s/sec' % (param._size_label(param.secondary.total), param._size_label(param.secondary.total / runtime))

1 0

[arm/master] Namedtuple may not have __dict__
by atagar＠torproject.org 15 Mar '15

15 Mar '15

commit 76a04df1aca2722320058874407624f51141985f Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 15 15:35:04 2015 -0700 Namedtuple may not have __dict__ Sounds like there's a python bug where namedtuples might not have __dict__... http://ronrothman.com/public/leftbraned/python-2-7-3-bug-broke-my-namedtupl… This manifests itself for me when starting up with... Traceback (most recent call last): File "./run_seth", line 47, in <module> main() File "./run_seth", line 16, in main seth.starter.main() File "/usr/lib/python2.7/stem/util/conf.py", line 288, in wrapped return func(*args, config = config, **kwargs) File "/home/atagar/Desktop/seth/seth/starter.py", line 92, in main curses.wrapper(seth.controller.start_seth) File "/usr/lib/python2.7/curses/wrapper.py", line 43, in wrapper return func(stdscr, *args, **kwds) File "/home/atagar/Desktop/seth/seth/controller.py", line 570, in start_seth control.redraw(False) File "/home/atagar/Desktop/seth/seth/controller.py", line 401, in redraw panel_impl.redraw(force) File "/home/atagar/Desktop/seth/seth/util/panel.py", line 442, in redraw self.draw(self.max_x, self.max_y) File "/home/atagar/Desktop/seth/seth/header_panel.py", line 134, in draw self._draw_platform_section(0, 0, left_width, vals) File "/home/atagar/Desktop/seth/seth/header_panel.py", line 167, in _draw_platform_section x = self.addstr(y, x, vals.format('seth - {hostname}', space_left)) File "/home/atagar/Desktop/seth/seth/header_panel.py", line 473, in format formatted_msg = message.format(**super(Sampling, self).__dict__) AttributeError: 'super' object has no attribute '__dict__' --- seth/header_panel.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/seth/header_panel.py b/seth/header_panel.py index b0ece7d..dff3d08 100644 --- a/seth/header_panel.py +++ b/seth/header_panel.py @@ -469,8 +469,12 @@ def get_sampling(last_sampling = None): } class Sampling(collections.namedtuple('Sampling', attr.keys())): + def __init__(self, **attr): + super(Sampling, self).__init__(**attr) + self._attr = attr + def format(self, message, crop_width = None): - formatted_msg = message.format(**super(Sampling, self).__dict__) + formatted_msg = message.format(**self._attr) if crop_width: formatted_msg = str_tools.crop(formatted_msg, crop_width)

1 0

[tor/master] Make log.o depend on micro-revision.i
by nickm＠torproject.org 15 Mar '15

15 Mar '15

commit e75e0c72781bd6a0cd8ad30cc4814692645a9f04 Author: Nick Mathewson <nickm(a)torproject.org> Date: Sun Mar 15 17:09:58 2015 -0400 Make log.o depend on micro-revision.i Otherwise micro-revision.i might not get built on time. --- src/common/include.am | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/common/include.am b/src/common/include.am index 5b63392..5417cca 100644 --- a/src/common/include.am +++ b/src/common/include.am @@ -72,6 +72,8 @@ LIBOR_A_SOURCES = \ $(libor_extra_source) \ $(threads_impl_source) +src/common/log.o: micro-revision.i + LIBOR_CRYPTO_A_SOURCES = \ src/common/aes.c \ src/common/crypto.c \

1 0

[stem/master] Support for extrainfo descriptor hidden service stats
by atagar＠torproject.org 15 Mar '15

15 Mar '15

commit 4ad8f60e2622e1ae5a7a46d131350c8042f02d66 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 15 13:17:07 2015 -0700 Support for extrainfo descriptor hidden service stats Adding support for the new extrainfo descriptor stats for hidden services... https://gitweb.torproject.org/torspec.git/commit/?id=ddb630d --- docs/change_log.rst | 1 + stem/descriptor/extrainfo_descriptor.py | 50 +++++++++++++++++++++ test/unit/descriptor/extrainfo_descriptor.py | 61 ++++++++++++++++++++++++++ 3 files changed, 112 insertions(+) diff --git a/docs/change_log.rst b/docs/change_log.rst index 2154341..3764c9b 100644 --- a/docs/change_log.rst +++ b/docs/change_log.rst @@ -62,6 +62,7 @@ conversion (:trac:`14075`). * Added `support for hidden service descriptors <api/descriptor/hidden_service_descriptor.html>`_ (:trac:`15004`) * The :class:`~stem.descriptor.networkstatus.DirectoryAuthority` 'fingerprint' attribute was actually its 'v3ident' * Added consensus' new package attribute (:spec:`ab64534`) + * Added extra info' new hs_stats_end, hs_rend_cells, hs_rend_cells_attr, hs_dir_onions_seen, and hs_dir_onions_seen_attr attributes (:spec:`ddb630d`) * Updating Faravahar's address (:trac:`14487`) * **Utilities** diff --git a/stem/descriptor/extrainfo_descriptor.py b/stem/descriptor/extrainfo_descriptor.py index 1a7bff2..228532f 100644 --- a/stem/descriptor/extrainfo_descriptor.py +++ b/stem/descriptor/extrainfo_descriptor.py @@ -502,6 +502,32 @@ def _parse_bridge_ip_transports_line(descriptor, entries): descriptor.ip_transports = ip_transports +def _parse_hs_stats(keyword, stat_attribute, extra_attribute, descriptor, entries): + # "<keyword>" num key=val key=val... + + value, stat, extra = _value(keyword, entries), None, {} + + if value is not None: + value_comp = value.split() + + if not value_comp: + raise ValueError("'%s' line was blank" % keyword) + elif not value_comp[0].isdigit(): + raise ValueError("'%s' stat was non-numeric (%s): %s %s" % (keyword, value_comp[0], keyword, value)) + + stat = int(value_comp[0]) + + for entry in value_comp[1:]: + if '=' not in entry: + raise ValueError('Entries after the stat in %s lines should only be key=val entries: %s %s' % (keyword, keyword, value)) + + key, val = entry.split('=', 1) + extra[key] = val + + setattr(descriptor, stat_attribute, stat) + setattr(descriptor, extra_attribute, extra) + + _parse_geoip_db_digest_line = _parse_forty_character_hex('geoip-db-digest', 'geoip_db_digest') _parse_geoip6_db_digest_line = _parse_forty_character_hex('geoip6-db-digest', 'geoip6_db_digest') _parse_dirreq_v2_resp_line = functools.partial(_parse_dirreq_line, 'dirreq-v2-resp', 'dir_v2_responses', 'dir_v2_responses_unknown') @@ -529,6 +555,9 @@ _parse_dirreq_write_history_line = functools.partial(_parse_history_line, 'dirre _parse_exit_kibibytes_written_line = functools.partial(_parse_port_count_line, 'exit-kibibytes-written', 'exit_kibibytes_written') _parse_exit_kibibytes_read_line = functools.partial(_parse_port_count_line, 'exit-kibibytes-read', 'exit_kibibytes_read') _parse_exit_streams_opened_line = functools.partial(_parse_port_count_line, 'exit-streams-opened', 'exit_streams_opened') +_parse_hidden_service_stats_end_line = _parse_timestamp_line('hidserv-stats-end', 'hs_stats_end') +_parse_hidden_service_rend_relayed_cells_line = functools.partial(_parse_hs_stats, 'hidserv-rend-relayed-cells', 'hs_rend_cells', 'hs_rend_cells_attr') +_parse_hidden_service_dir_onions_seen_line = functools.partial(_parse_hs_stats, 'hidserv-dir-onions-seen', 'hs_dir_onions_seen', 'hs_dir_onions_seen_attr') _parse_dirreq_v2_ips_line = functools.partial(_parse_geoip_to_count_line, 'dirreq-v2-ips', 'dir_v2_ips') _parse_dirreq_v3_ips_line = functools.partial(_parse_geoip_to_count_line, 'dirreq-v3-ips', 'dir_v3_ips') _parse_dirreq_v2_reqs_line = functools.partial(_parse_geoip_to_count_line, 'dirreq-v2-reqs', 'dir_v2_requests') @@ -631,6 +660,14 @@ class ExtraInfoDescriptor(Descriptor): :var dict exit_kibibytes_read: traffic per port (keys are ints or 'other') :var dict exit_streams_opened: streams per port (keys are ints or 'other') + **Hidden Service Attributes:** + + :var datetime hs_stats_end: end of the sampling interval + :var int hs_rend_cells: rounded count of the RENDEZVOUS1 cells seen + :var int hs_rend_cells_attr: **\*** attributes provided for the hs_rend_cells + :var int hs_dir_onions_seen: rounded count of the identities seen + :var int hs_dir_onions_seen_attr: **\*** attributes provided for the hs_dir_onions_seen + **Bridge Attributes:** :var datetime bridge_stats_end: end of the period when stats were gathered @@ -643,6 +680,10 @@ class ExtraInfoDescriptor(Descriptor): **\*** attribute is either required when we're parsed with validation or has a default value, others are left as **None** if undefined + + .. versionchanged:: 1.4.0 + Added the hs_stats_end, hs_rend_cells, hs_rend_cells_attr, + hs_dir_onions_seen, and hs_dir_onions_seen_attr attributes. """ ATTRIBUTES = { @@ -714,6 +755,12 @@ class ExtraInfoDescriptor(Descriptor): 'exit_kibibytes_read': (None, _parse_exit_kibibytes_read_line), 'exit_streams_opened': (None, _parse_exit_streams_opened_line), + 'hs_stats_end': (None, _parse_hidden_service_stats_end_line), + 'hs_rend_cells': (None, _parse_hidden_service_rend_relayed_cells_line), + 'hs_rend_cells_attr': ({}, _parse_hidden_service_rend_relayed_cells_line), + 'hs_dir_onions_seen': (None, _parse_hidden_service_dir_onions_seen_line), + 'hs_dir_onions_seen_attr': ({}, _parse_hidden_service_dir_onions_seen_line), + 'bridge_stats_end': (None, _parse_bridge_stats_end_line), 'bridge_stats_interval': (None, _parse_bridge_stats_end_line), 'bridge_ips': (None, _parse_bridge_ips_line), @@ -756,6 +803,9 @@ class ExtraInfoDescriptor(Descriptor): 'exit-kibibytes-written': _parse_exit_kibibytes_written_line, 'exit-kibibytes-read': _parse_exit_kibibytes_read_line, 'exit-streams-opened': _parse_exit_streams_opened_line, + 'hidserv-stats-end': _parse_hidden_service_stats_end_line, + 'hidserv-rend-relayed-cells': _parse_hidden_service_rend_relayed_cells_line, + 'hidserv-dir-onions-seen': _parse_hidden_service_dir_onions_seen_line, 'dirreq-v2-ips': _parse_dirreq_v2_ips_line, 'dirreq-v3-ips': _parse_dirreq_v3_ips_line, 'dirreq-v2-reqs': _parse_dirreq_v2_reqs_line, diff --git a/test/unit/descriptor/extrainfo_descriptor.py b/test/unit/descriptor/extrainfo_descriptor.py index a30d47a..6049f50 100644 --- a/test/unit/descriptor/extrainfo_descriptor.py +++ b/test/unit/descriptor/extrainfo_descriptor.py @@ -527,6 +527,67 @@ k0d2aofcVbHr4fPQOSST0LXDrhFl5Fqo5um296zpJGvRUeO6S44U/EfJAGShtqWw desc_text = get_relay_extrainfo_descriptor({keyword: entry}, content = True) self._expect_invalid_attr(desc_text, attr) + def test_hidden_service_stats_end(self): + """ + Exercise the hidserv-stats-end, which should be a simple date. + """ + + desc = get_relay_extrainfo_descriptor({'hidserv-stats-end': '2012-05-03 12:07:50'}) + self.assertEqual(datetime.datetime(2012, 5, 3, 12, 7, 50), desc.hs_stats_end) + + test_entries = ( + '', + '2012', + '2012-05', + '2012-05-03', + '2012-05-03 12', + '2012-05-03 12:07', + '2012-05-03 12:07:-50', + ) + + for entry in test_entries: + desc_text = get_relay_extrainfo_descriptor({'hidserv-stats-end': entry}, content = True) + self._expect_invalid_attr(desc_text, 'hs_stats_end') + + def test_hidden_service_stats(self): + """ + Check the 'hidserv-rend-relayed-cells' and 'hidserv-dir-onions-seen', which + share the same format. + """ + + attributes = ( + ('hidserv-rend-relayed-cells', 'hs_rend_cells', 'hs_rend_cells_attr'), + ('hidserv-dir-onions-seen', 'hs_dir_onions_seen', 'hs_dir_onions_seen_attr'), + ) + + test_entries = ( + '', + '-50', + 'hello', + ' key=value', + '40 key', + '40 key value', + '40 key key=value', + ) + + for keyword, stat_attr, extra_attr in attributes: + # just the numeric stat (no extra attributes) + + desc = get_relay_extrainfo_descriptor({keyword: '345'}) + self.assertEqual(345, getattr(desc, stat_attr)) + self.assertEqual({}, getattr(desc, extra_attr)) + + # with extra attributes + + desc = get_relay_extrainfo_descriptor({keyword: '345 spiffy=true snowmen=neat'}) + self.assertEqual(345, getattr(desc, stat_attr)) + self.assertEqual({'spiffy': 'true', 'snowmen': 'neat'}, getattr(desc, extra_attr)) + + for entry in test_entries: + desc_text = get_relay_extrainfo_descriptor({keyword: entry}, content = True) + self._expect_invalid_attr(desc_text, stat_attr) + self._expect_invalid_attr(desc_text, extra_attr, {}) + def test_locale_mapping_lines(self): """ Uses valid and invalid data to tests lines of the form...

1 0

[stem/master] Adding HS_DESC_CONTENT to EventType
by atagar＠torproject.org 15 Mar '15

15 Mar '15

commit 858667a1452c7a2e87151bf83286107a3b5a002a Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 15 11:36:51 2015 -0700 Adding HS_DESC_CONTENT to EventType Issue caught by DonnchaC. --- stem/control.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/stem/control.py b/stem/control.py index d0e09cc..68883d9 100644 --- a/stem/control.py +++ b/stem/control.py @@ -187,6 +187,7 @@ If you're fine with allowing your script to raise exceptions then this can be mo **ERR** :class:`stem.response.events.LogEvent` **GUARD** :class:`stem.response.events.GuardEvent` **HS_DESC** :class:`stem.response.events.HSDescEvent` + **HS_DESC_CONTENT** :class:`stem.response.events.HSDescContentEvent` **INFO** :class:`stem.response.events.LogEvent` **NEWCONSENSUS** :class:`stem.response.events.NewConsensusEvent` **NEWDESC** :class:`stem.response.events.NewDescEvent` @@ -277,6 +278,7 @@ EventType = stem.util.enum.UppercaseEnum( 'ERR', 'GUARD', 'HS_DESC', + 'HS_DESC_CONTENT', 'INFO', 'NEWCONSENSUS', 'NEWDESC',

1 0

[translation/torbutton-torbuttonproperties] Update translations for torbutton-torbuttonproperties
by translation＠torproject.org 15 Mar '15

15 Mar '15

commit 078fc1854aaa0c1e846c8a24a71572e61f02bc98 Author: Translation commit bot <translation(a)torproject.org> Date: Sun Mar 15 16:45:48 2015 +0000 Update translations for torbutton-torbuttonproperties --- fa/torbutton.properties | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fa/torbutton.properties b/fa/torbutton.properties index 75bde0f..7e55c5e 100644 --- a/fa/torbutton.properties +++ b/fa/torbutton.properties @@ -2,9 +2,9 @@ torbutton.button.tooltip.disabled = فعال کردن تُر torbutton.button.tooltip.enabled = غيرفعال کردن تُر torbutton.circuit_display.internet = اینترنت torbutton.circuit_display.ip_unknown = IP ناشناخته -torbutton.circuit_display.onion_site = Onion site +torbutton.circuit_display.onion_site = سایت پیاز torbutton.circuit_display.this_browser = این مرورگر -torbutton.circuit_display.relay = relay +torbutton.circuit_display.relay = رله torbutton.circuit_display.tor_bridge = پل torbutton.circuit_display.unknown_country = کشور ناشناخته torbutton.panel.tooltip.disabled = برای فعال کردن تُرکليک کنيد

1 0

[metrics-web/master] Fix useless link.
by karsten＠torproject.org 15 Mar '15

15 Mar '15

commit 38fc8300182f40550ae67fab25b59d5e1cd35d68 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Sun Mar 15 17:16:29 2015 +0100 Fix useless link. Pointed out by Anonymous Kyoto on tor-talk@. --- website/web/WEB-INF/about.jsp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/web/WEB-INF/about.jsp b/website/web/WEB-INF/about.jsp index 9026133..84e5261 100644 --- a/website/web/WEB-INF/about.jsp +++ b/website/web/WEB-INF/about.jsp @@ -210,9 +210,9 @@ It simply takes time to report, collect, and process all the data. <b>Q: Are the raw numbers behind graphs available for download?</b> <br> A: Yes, all raw numbers are available in <a href="stats/">comma-separated -value files (.csv)</a>, which have their own -<a href="https://gitweb.torproject.org/metrics-web.git/tree/doc/stats-spec.txt">specification -document</a>. +value files (.csv)</a>, which are further explained on the "Data:" pages, +e.g., <a href="https://metrics.torproject.org/servers-data.html">"Data: +Number of relays and bridges"</a>. In addition to that, the raw data behind those .csv files are available via the <a href="https://collector.torproject.org/">CollecTor service</a>. If you do something cool with either the .csv files or the raw data,

1 0

[torspec/master] Add hidserv-stats from proposal 238.
by nickm＠torproject.org 15 Mar '15

15 Mar '15

commit ddb630dbad6ab103122ae95a164cdf2605ad6ff1 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Feb 24 12:05:32 2015 +0100 Add hidserv-stats from proposal 238. --- dir-spec.txt | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/dir-spec.txt b/dir-spec.txt index f74b16d..70a67dc 100644 --- a/dir-spec.txt +++ b/dir-spec.txt @@ -954,6 +954,45 @@ limit the number of listed ports and subsume any remaining opened streams under port "other". + "hidserv-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] + + YYYY-MM-DD HH:MM:SS defines the end of the included measurement + interval of length NSEC seconds (86400 seconds by default). + + A "hidserv-stats-end" line, as well as any other "hidserv-*" line, + is first added after the relay has been running for at least 24 + hours. + + "hidserv-rend-relayed-cells" SP num SP key=val SP key=val ... NL + [At most once.] + + Approximate number of RELAY cells seen in either direction on a + circuit after receiving and successfully processing a RENDEZVOUS1 + cell. + + The original measurement value is obfuscated in several steps: + first, it is rounded up to the nearest multiple of 'bin_size' + which is reported in the key=val part of this line; second, a + (possibly negative) noise value is added to the result of the + first step by randomly sampling from a Laplace distribution with + mu = 0 and b = (delta_f / epsilon) with 'delta_f' and 'epsilon' + being reported in the key=val part, too; third, the result of the + previous obfuscation steps is truncated to the next smaller + integer and included as 'num'. + + "hidserv-dir-onions-seen" SP num SP key=val SP key=val ... NL + [At most once.] + + Approximate number of unique hidden-service identities seen in + descriptors published to and accepted by this hidden-service + directory. + + The original measurement value is obfuscated in the same way as + the 'num' value reported in "hidserv-rend-relayed-cells", but + possibly with different parameters as reported in the key=val part + of this line. + "transport" transportname address:port [arglist] NL [Any number.]

1 0

[torspec/master] Merge remote-tracking branch 'karsten/hidserv-stats'
by nickm＠torproject.org 15 Mar '15

15 Mar '15

commit bc348154c63f46fc6cc56c49d339fcd0aab78f3d Merge: b4d3d46 ddb630d Author: Nick Mathewson <nickm(a)torproject.org> Date: Sun Mar 15 08:29:05 2015 -0400 Merge remote-tracking branch 'karsten/hidserv-stats' dir-spec.txt | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)

1 0