August 2014 - tor-commits - lists.torproject.org

[doctor/master] Messages about unmesured relays weren't being effectively suppressed
by atagar＠torproject.org 31 Aug '14

31 Aug '14

commit 67ae3828256ae5ceb2184cbe53bdf9f6504be999 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Aug 31 16:30:48 2014 -0700 Messages about unmesured relays weren't being effectively suppressed We suppress messages based on their content but the notice about unmeasured relays have stats that change each hour. As a result they effectively weren't being suppressed causing a notice each hour. Adding a hack to clobber the dynamic data so our suppression string will work... As_a_bandwidth_authority_gabelmoo_lacked_a_measurement_for_0_of_0_relays_(0%) 1409527674 I'm not thrilled with this solution, but we can add generic handling when we have more than one message that needs it. --- consensus_health_checker.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/consensus_health_checker.py b/consensus_health_checker.py index 25a1143..e3cc7bf 100755 --- a/consensus_health_checker.py +++ b/consensus_health_checker.py @@ -178,7 +178,18 @@ def main(): is_all_suppressed = True # either no issues or they're all already suppressed for issue in issues: - key = issue.get_message().replace(' ', '_') + if issue._template == 'TOO_MANY_UNMEASURED_RELAYS': + # Hack because this message has too much dynamic data to be effectively + # suppressed. Hate doing this here, so better would be to make this a + # config property. + + attr = dict(issue._attr) + attr.update({'unmeasured': 0, 'total': 0, 'percentage': 0}) + + key = CONFIG['msg'][issue._template].format(**attr).replace(' ', '_') + else: + key = issue.get_message().replace(' ', '_') + duration = issue.get_suppression_duration() if rate_limit_notice(key, duration):

1 0

[doctor/master] Enabling check for unmeasured relays
by atagar＠torproject.org 31 Aug '14

31 Aug '14

commit 0972f330efb6e947d60e7b1db631abe37fbc2094 Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Aug 31 12:31:42 2014 -0700 Enabling check for unmeasured relays Sebastian says he'd find these numbers useful so enabling the check. We're already alerting for other NOTICE runlevel issues so this shouldn't appreciably add to the noise. Also lowering the threshold of the alert to 5%. --- consensus_health_checker.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/consensus_health_checker.py b/consensus_health_checker.py index fde03aa..25a1143 100755 --- a/consensus_health_checker.py +++ b/consensus_health_checker.py @@ -233,8 +233,7 @@ def run_checks(consensuses, votes): consensuses_have_same_votes, has_all_signatures, voting_bandwidth_scanners, - # TODO: enable when the bandwidth scanners are fixed (#12989) - #unmeasured_relays, + unmeasured_relays, has_authority_flag, is_recommended_versions, bad_exits_in_sync, @@ -491,7 +490,7 @@ def unmeasured_relays(latest_consensus, consensuses, votes): total = len(vote.routers) percentage = 100 * unmeasured / total - if percentage >= 10: + if percentage >= 5: issues.append(Issue(Runlevel.NOTICE, 'TOO_MANY_UNMEASURED_RELAYS', authority = authority, unmeasured = unmeasured, total = total, percentage = percentage)) return issues

1 0

[obfs4/master] Add elideError(error) that knows how to sanitize net.Errors.
by yawning＠torproject.org 31 Aug '14

31 Aug '14

commit ce39988b11a56e763c8eedbbdab75a35b48366d4 Author: Yawning Angel <yawning(a)torproject.org> Date: Sun Aug 31 14:30:14 2014 +0000 Add elideError(error) that knows how to sanitize net.Errors. Instead of omitting errors entirely when running with the log scrubber, filter common network errors through elideError() that can scrub the common net.Error types and remove sensitive information. --- obfs4proxy/obfs4proxy.go | 80 +++++++++++++++++++++++++--------------------- 1 file changed, 44 insertions(+), 36 deletions(-) diff --git a/obfs4proxy/obfs4proxy.go b/obfs4proxy/obfs4proxy.go index c67bb21..8cb0cbf 100644 --- a/obfs4proxy/obfs4proxy.go +++ b/obfs4proxy/obfs4proxy.go @@ -80,6 +80,40 @@ func elideAddr(addrStr string) string { return elidedAddr } +func elideError(err error) string { + // Go's net package is somewhat rude and includes IP address and port + // information in the string representation of net.Errors. Figure out if + // this is the case here, and sanitize the error messages as needed. + if unsafeLogging { + return err.Error() + } + + // If err is not a net.Error, just return the string representation, + // presumably transport authors know what they are doing. + netErr, ok := err.(net.Error) + if !ok { + return err.Error() + } + + switch t := netErr.(type) { + case *net.AddrError: + return t.Err + " " + elidedAddr + case *net.DNSError: + return "lookup " + elidedAddr + " on " + elidedAddr + ": " + t.Err + case *net.InvalidAddrError: + return "invalid address error" + case *net.UnknownNetworkError: + return "unknown network " + elidedAddr + case *net.OpError: + return t.Op + ": " + t.Err.Error() + default: + // For unknown error types, do the conservative thing and only log the + // type of the error instead of assuming that the string representation + // does not contain sensitive information. + return fmt.Sprintf("network error: <%T>", t) + } +} + func clientSetup() (launched bool, listeners []net.Listener) { ptClientInfo, err := pt.ClientSetup(transports.Transports()) if err != nil { @@ -168,11 +202,7 @@ func clientHandler(f base.ClientFactory, conn *pt.SocksConn, proxyURI *url.URL) // the configuration phase. dialer, err := proxy.FromURL(proxyURI, proxy.Direct) if err != nil { - if unsafeLogging { - log.Printf("[ERROR]: %s(%s) - failed to obtain proxy dialer: %s", name, addrStr, err) - } else { - log.Printf("[ERROR]: %s(%s) - failed to obtain proxy dialer", name, addrStr) - } + log.Printf("[ERROR]: %s(%s) - failed to obtain proxy dialer: %s", name, addrStr, elideError(err)) conn.Reject() return } @@ -180,13 +210,7 @@ func clientHandler(f base.ClientFactory, conn *pt.SocksConn, proxyURI *url.URL) } remoteConn, err := dialFn("tcp", conn.Req.Target) // XXX: Allow UDP? if err != nil { - // Note: The error message returned from the dialer can include the IP - // address/port of the remote peer. - if unsafeLogging { - log.Printf("[ERROR]: %s(%s) - outgoing connection failed: %s", name, addrStr, err) - } else { - log.Printf("[ERROR]: %s(%s) - outgoing connection failed", name, addrStr) - } + log.Printf("[ERROR]: %s(%s) - outgoing connection failed: %s", name, addrStr, elideError(err)) conn.Reject() return } @@ -196,27 +220,19 @@ func clientHandler(f base.ClientFactory, conn *pt.SocksConn, proxyURI *url.URL) // bytes back and forth. remote, err := f.WrapConn(remoteConn, args) if err != nil { - if unsafeLogging { - log.Printf("[ERROR]: %s(%s) - handshake failed: %s", name, addrStr, err) - } else { - log.Printf("[ERROR]: %s(%s) - handshake failed", name, addrStr) - } + log.Printf("[ERROR]: %s(%s) - handshake failed: %s", name, addrStr, elideError(err)) conn.Reject() return } err = conn.Grant(remoteConn.RemoteAddr().(*net.TCPAddr)) if err != nil { - if unsafeLogging { - log.Printf("[ERROR]: %s(%s) - SOCKS grant failed: %s", name, addrStr, err) - } else { - log.Printf("[ERROR]: %s(%s) - SOCKS grant failed", name, addrStr) - } + log.Printf("[ERROR]: %s(%s) - SOCKS grant failed: %s", name, addrStr, elideError(err)) return } err = copyLoop(conn, remote) - if err != nil && unsafeLogging { - log.Printf("[INFO]: %s(%s) - closed connection: %s", name, addrStr, err) + if err != nil { + log.Printf("[INFO]: %s(%s) - closed connection: %s", name, addrStr, elideError(err)) } else { log.Printf("[INFO]: %s(%s) - closed connection", name, addrStr) } @@ -295,29 +311,21 @@ func serverHandler(f base.ServerFactory, conn net.Conn, info *pt.ServerInfo) { // Instantiate the server transport method and handshake. remote, err := f.WrapConn(conn) if err != nil { - if unsafeLogging { - log.Printf("[ERROR]: %s(%s) - handshake failed: %s", name, addrStr, err) - } else { - log.Printf("[ERROR]: %s(%s) - handshake failed", name, addrStr) - } + log.Printf("[ERROR]: %s(%s) - handshake failed: %s", name, addrStr, elideError(err)) return } // Connect to the orport. orConn, err := pt.DialOr(info, conn.RemoteAddr().String(), name) if err != nil { - if unsafeLogging { - log.Printf("[ERROR]: %s(%s) - failed to connect to ORPort: %s", name, addrStr, err) - } else { - log.Printf("[ERROR]: %s(%s) - failed to connect to ORPort", name, addrStr) - } + log.Printf("[ERROR]: %s(%s) - failed to connect to ORPort: %s", name, addrStr, elideError(err)) return } defer orConn.Close() err = copyLoop(orConn, remote) - if err != nil && unsafeLogging { - log.Printf("[INFO]: %s(%s) - closed connection: %s", name, addrStr, err) + if err != nil { + log.Printf("[INFO]: %s(%s) - closed connection: %s", name, addrStr, elideError(err)) } else { log.Printf("[INFO]: %s(%s) - closed connection", name, addrStr) }

1 0

[onionoo/master] Add version field.
by karsten＠torproject.org 31 Aug '14

31 Aug '14

commit b19c5d57a2a638e4280490e6c3888bcc65c767b3 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Sun Aug 31 08:45:18 2014 +0200 Add version field. Suggested by iwakeh in #12905. Implements that ticket. --- .../torproject/onionoo/server/ResponseBuilder.java | 5 +- web/protocol.html | 83 +++++++++++++++++++- 2 files changed, 84 insertions(+), 4 deletions(-) diff --git a/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java b/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java index 676a039..c951fd3 100644 --- a/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java +++ b/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java @@ -69,8 +69,11 @@ public class ResponseBuilder { return this.charsWritten; } + private static final String PROTOCOL_VERSION = "1.0"; + private void writeRelays(List<SummaryDocument> relays, PrintWriter pw) { - String header = "{\"relays_published\":\"" + relaysPublishedString + String header = "{\"version\":\"" + PROTOCOL_VERSION + + "\",\n\"relays_published\":\"" + relaysPublishedString + "\",\n\"relays\":["; this.charsWritten += header.length(); pw.write(header); diff --git a/web/protocol.html b/web/protocol.html index 0a123ba..9194c63 100644 --- a/web/protocol.html +++ b/web/protocol.html @@ -120,7 +120,7 @@ Only file a bug report if this problem persists. </ul> -<h4>Protocol changes</h4> +<h4>Protocol versions</h4> There are plenty of reasons why we may have to change the protocol described here. @@ -135,8 +135,25 @@ If you want to be informed of upcoming protocol changes, subscribe to the mailing list. -The Onionoo API is described by resource types and available methods -below. +All responses contain a "version" string that indicates whether clients +know whether they can parse the document or not. +Version strings consist of a major and a minor version number. +The major version number is raised when previously required fields are +dropped or turned into optional fields, when request parameters or +response documents are removed, or when there are structural changes. +The minor version number is raised when new fields, request parameters, or +response documents are added or optional fields are dropped. +If clients support the same major version given in a response but only a +lower minor version, they should be able to parse the document but may not +understand all fields. +If clients support a lower major version, they should not attempt to parse +a document, because there may be backward-incompatible changes. +The following versions have been used in the past six months: + +<ul> +<li>1.0: First assigned version number on August 31, +2014.</li> +</ul> </div>  @@ -476,6 +493,16 @@ Summary documents contain the following fields: <ul class="properties"> <li> +version +<code class="typeof">string</code> +required + +Onionoo protocol version string. +Added on August 31, 2014. + +</li> + +<li> relays_published <code class="typeof">string</code> required @@ -629,6 +656,16 @@ Details documents contain the following fields: <ul class="properties"> <li> +version +<code class="typeof">string</code> +required + +Onionoo protocol version string. +Added on August 31, 2014. + +</li> + +<li> relays_published <code class="typeof">string</code> required @@ -1404,6 +1441,16 @@ Bandwidth documents contain the following fields: <ul class="properties"> <li> +version +<code class="typeof">string</code> +required + +Onionoo protocol version string. +Added on August 31, 2014. + +</li> + +<li> relays_published <code class="typeof">string</code> required @@ -1567,6 +1614,16 @@ Weights documents contain the following fields: <ul class="properties"> <li> +version +<code class="typeof">string</code> +required + +Onionoo protocol version string. +Added on August 31, 2014. + +</li> + +<li> relays_published <code class="typeof">string</code> required @@ -1762,6 +1819,16 @@ Clients documents contain the following fields: <ul class="properties"> <li> +version +<code class="typeof">string</code> +required + +Onionoo protocol version string. +Added on August 31, 2014. + +</li> + +<li> relays_published <code class="typeof">string</code> required @@ -1930,6 +1997,16 @@ Uptime documents contain the following fields: <ul class="properties"> <li> +version +<code class="typeof">string</code> +required + +Onionoo protocol version string. +Added on August 31, 2014. + +</li> + +<li> relays_published <code class="typeof">string</code> required

1 0

[onionoo/master] Take out old news.
by karsten＠torproject.org 31 Aug '14

31 Aug '14

commit 072a76b9ce1bedf7b3311f6be8de43a3b029880d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Sun Aug 31 08:49:29 2014 +0200 Take out old news. --- web/protocol.html | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/web/protocol.html b/web/protocol.html index 9194c63..9a46c9e 100644 --- a/web/protocol.html +++ b/web/protocol.html @@ -376,7 +376,7 @@ Comparisons are case-insensitive. </li> <li> -family +family Return only the relay whose fingerprint matches the parameter value and all relays that this relay has listed in its family by fingerprint and @@ -388,7 +388,6 @@ The provided relay fingerprint must consist of 40 hex characters where case does not matter, and it must not be hashed using SHA-1. Bridges are not contained in the result, regardless of whether they define a family. -Added on July 4, 2014. </li>

1 0

[translation/tor-launcher-properties] Update translations for tor-launcher-properties
by translation＠torproject.org 31 Aug '14

31 Aug '14

commit 2d87bb0beb9d96d2c2339d3721430883ad2a98d5 Author: Translation commit bot <translation(a)torproject.org> Date: Sun Aug 31 04:15:29 2014 +0000 Update translations for tor-launcher-properties --- id/torlauncher.properties | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/id/torlauncher.properties b/id/torlauncher.properties index 9151a1a..43db58a 100644 --- a/id/torlauncher.properties +++ b/id/torlauncher.properties @@ -5,16 +5,16 @@ torlauncher.error_title=Tor Launcher torlauncher.tor_exited=Tor unexpectedly exited. This might be due to a bug in Tor itself, another program on your system, or faulty hardware. Until you restart Tor, the Tor Browser will not able to reach any websites. If the problem persists, please send a copy of your Tor Log to the support team. torlauncher.tor_exited2=Restarting Tor will not close your browser tabs. -torlauncher.tor_controlconn_failed=Could not connect to Tor control port. +torlauncher.tor_controlconn_failed=Tidak dapat tersambung pada port Tor torlauncher.tor_failed_to_start=Tor gagal untuk memulai. torlauncher.tor_control_failed=Failed to take control of Tor. torlauncher.tor_bootstrap_failed=Tor failed to establish a Tor network connection. torlauncher.tor_bootstrap_failed_details=%1$S failed (%2$S). -torlauncher.unable_to_start_tor=Unable to start Tor.\n\n%S -torlauncher.tor_missing=The Tor executable is missing. -torlauncher.torrc_missing=The torrc file is missing. -torlauncher.datadir_missing=The Tor data directory does not exist. +torlauncher.unable_to_start_tor=Tidak dapat memulai Tor\n\n%S +torlauncher.tor_missing=Tor executable hilang +torlauncher.torrc_missing=File torrc hilang +torlauncher.datadir_missing=Tor data direktori tidak ada. torlauncher.password_hash_missing=Failed to get hashed password. torlauncher.failed_to_get_settings=Unable to retrieve Tor settings.\n\n%S

1 0

r26933: {website} new news for the website (website/trunk/en)
by Roger Dingledine 31 Aug '14

31 Aug '14

Author: arma Date: 2014-08-31 03:41:05 +0000 (Sun, 31 Aug 2014) New Revision: 26933 Modified: website/trunk/en/index.wml Log: new news for the website Modified: website/trunk/en/index.wml =================================================================== --- website/trunk/en/index.wml 2014-08-29 20:27:36 UTC (rev 26932) +++ website/trunk/en/index.wml 2014-08-31 03:41:05 UTC (rev 26933) @@ -158,6 +158,14 @@ <table> <tr> <td> + <div class="calendar">Aug 30</div> + A call to arms: <a + href="https://blog.torproject.org/blog/call-arms-helping-internet-services-accept…">Helping + Internet services accept anonymous users</a>. + </td> + </tr> + <tr> + <td> <div class="calendar">Aug 18</div> iSEC Partners Conducts <a href="https://blog.torproject.org/blog/isec-partners-conducts-tor-browser-hardeni…"> @@ -178,15 +186,6 @@ Tor Browser releases</a>: 3.6.4 and 4.0-alpha-1. </td> </tr> - <tr> - <td> - <div class="calendar">Jul 30</div> - <a - href="https://blog.torproject.org/blog/tor-security-advisory-relay-early-traffic-…"> - Tor security advisory</a>: "relay early" traffic - confirmation attack. - </td> - </tr> </table> </div> <div id="home-users">

1 0

[doctor/master] Notice if too many relays are unmeasured by bandwidth authorities
by atagar＠torproject.org 30 Aug '14

30 Aug '14

commit 30540279d0105b056b7f6415298d35887c4ca215 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Aug 30 13:47:15 2014 -0700 Notice if too many relays are unmeasured by bandwidth authorities Adding a notice when bandwidth authorities lack an opinion on over 10% of the relays... NOTICE: As a bandwidth authority maatuska lacked a measurement for 1038 of 7418 relays (13%) NOTICE: As a bandwidth authority tor26 lacked a measurement for 1764 of 7418 relays (23%) NOTICE: As a bandwidth authority gabelmoo lacked a measurement for 1361 of 7266 relays (18%) This is disabled for now since the bandwidth authorities are presently having issues. This is for... https://trac.torproject.org/projects/tor/ticket/12989 --- consensus_health_checker.py | 24 ++++++++++++++++++++++++ data/consensus_health.cfg | 1 + 2 files changed, 25 insertions(+) diff --git a/consensus_health_checker.py b/consensus_health_checker.py index df827fb..fde03aa 100755 --- a/consensus_health_checker.py +++ b/consensus_health_checker.py @@ -233,6 +233,8 @@ def run_checks(consensuses, votes): consensuses_have_same_votes, has_all_signatures, voting_bandwidth_scanners, + # TODO: enable when the bandwidth scanners are fixed (#12989) + #unmeasured_relays, has_authority_flag, is_recommended_versions, bad_exits_in_sync, @@ -473,6 +475,28 @@ def voting_bandwidth_scanners(latest_consensus, consensuses, votes): return issues +def unmeasured_relays(latest_consensus, consensuses, votes): + "Checks that the bandwidth authorities have all formed an opinion about at least 90% of the relays." + + issues = [] + + for authority, vote in votes.items(): + if authority in CONFIG['bandwidth_authorities']: + unmeasured = 0 + + for desc in vote.routers.values(): + if not desc.measured: + unmeasured += 1 + + total = len(vote.routers) + percentage = 100 * unmeasured / total + + if percentage >= 10: + issues.append(Issue(Runlevel.NOTICE, 'TOO_MANY_UNMEASURED_RELAYS', authority = authority, unmeasured = unmeasured, total = total, percentage = percentage)) + + return issues + + def has_authority_flag(latest_consensus, consensuses, votes): "Checks that the authorities have the 'authority' flag in the present consensus." diff --git a/data/consensus_health.cfg b/data/consensus_health.cfg index af7c200..8f25fb4 100644 --- a/data/consensus_health.cfg +++ b/data/consensus_health.cfg @@ -9,6 +9,7 @@ msg CERTIFICATE_ABOUT_TO_EXPIRE => The certificate of the following directory au msg MISSING_SIGNATURE => Consensus fetched from {consensus_of} was missing the following authority signatures: {authorities} msg MISSING_BANDWIDTH_SCANNERS => The following directory authorities are not reporting bandwidth scanner results: {authorities} msg EXTRA_BANDWIDTH_SCANNERS => The following directory authorities were not expected to report bandwidth scanner results: {authorities} +msg TOO_MANY_UNMEASURED_RELAYS => As a bandwidth authority {authority} lacked a measurement for {unmeasured} of {total} relays ({percentage}%) msg MISSING_VOTES => The consensuses downloaded from the following authorities are missing votes that are contained in consensuses downloaded from other authorities: {authorities} msg MISSING_AUTHORITIES => The following authorities are missing from the consensus: {authorities} msg EXTRA_AUTHORITIES => The following authorities were not expected in the consensus: {authorities}

1 0

[translation/torcheck_completed] Update translations for torcheck_completed
by translation＠torproject.org 30 Aug '14

30 Aug '14

commit 0537d2981efacda920749005d534baa74a01ca36 Author: Translation commit bot <translation(a)torproject.org> Date: Sat Aug 30 18:15:13 2014 +0000 Update translations for torcheck_completed --- uk/torcheck.po | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/uk/torcheck.po b/uk/torcheck.po index de83c77..595f995 100644 --- a/uk/torcheck.po +++ b/uk/torcheck.po @@ -11,7 +11,7 @@ msgid "" msgstr "" "Project-Id-Version: The Tor Project\n" "POT-Creation-Date: 2012-02-16 20:28+PDT\n" -"PO-Revision-Date: 2014-08-30 17:13+0000\n" +"PO-Revision-Date: 2014-08-30 17:55+0000\n" "Last-Translator: Ігор Желіба <traumig(a)yandex.ua>\n" "Language-Team: Ukrainian (http://www.transifex.com/projects/p/torproject/language/uk/)\n" "MIME-Version: 1.0\n"

1 0

[translation/torcheck] Update translations for torcheck
by translation＠torproject.org 30 Aug '14

30 Aug '14

commit c3f28f7dffbb678a884ffb3e480bbdea09a37804 Author: Translation commit bot <translation(a)torproject.org> Date: Sat Aug 30 18:15:09 2014 +0000 Update translations for torcheck --- uk/torcheck.po | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/uk/torcheck.po b/uk/torcheck.po index de83c77..595f995 100644 --- a/uk/torcheck.po +++ b/uk/torcheck.po @@ -11,7 +11,7 @@ msgid "" msgstr "" "Project-Id-Version: The Tor Project\n" "POT-Creation-Date: 2012-02-16 20:28+PDT\n" -"PO-Revision-Date: 2014-08-30 17:13+0000\n" +"PO-Revision-Date: 2014-08-30 17:55+0000\n" "Last-Translator: Ігор Желіба <traumig(a)yandex.ua>\n" "Language-Team: Ukrainian (http://www.transifex.com/projects/p/torproject/language/uk/)\n" "MIME-Version: 1.0\n"

1 0