- tor-commits - lists.torproject.org

[newsletter/staging] Add newsletter for January
by hiro＠torproject.org 27 Mar '19

27 Mar '19

commit c6c795da7b4d66facbc5be75eb6bc2d61187730c Author: hiro <hiro(a)torproject.org> Date: Thu Jan 31 15:52:15 2019 +0100 Add newsletter for January --- content/archive/expect-more-from-tor/contents.lr | 187 +++++++++++++++++++++ .../archive/expect-more-from-tor/text/contents.lr | 97 +++++++++++ newsletter.lektorproject | 4 +- 3 files changed, 285 insertions(+), 3 deletions(-) diff --git a/content/archive/expect-more-from-tor/contents.lr b/content/archive/expect-more-from-tor/contents.lr new file mode 100644 index 0000000..2d81ad3 --- /dev/null +++ b/content/archive/expect-more-from-tor/contents.lr @@ -0,0 +1,187 @@ +_model: post +--- +_template: newsletter.html +--- +author: steph(a)torproject.org +--- +pub_date: 2019-01-31 +--- +title: Expect More from Tor in 2019 +--- +html_body: + +<table align="center" border="0" cellpadding="0" cellspacing="0" class="devicewidth" width="650"> + <tbody> + <tr> + <td width="100%"> + <table align="center" bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" class="devicewidth" width="650"> + <tbody> + <tr> + <td height="20" width="100%"><a href="https://newsletter.torproject.org"><img alt="tor-news-logo" src="https://blog.torproject.org/sites/default/files/inline-images/tor-news-logo…" style="width: 250px; height: 75px;" /></a></td> + </tr> + <tr> + <td> + <table align="center" border="0" cellpadding="0" cellspacing="0" class="devicewidth" width="650"> + <tbody> + <tr> + <td width="100%"> + <table align="center" bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" class="devicewidth" width="650"> + <tbody> + <tr> + <td style="padding:0 15px 15px 15px;"> + <hr /> + <h1>Expect More From Tor in 2019</h1> + + <p><a href="https://blog.torproject.org/expect-more-tor-2019"><img alt="tor onion " src="https://blog.torproject.org/sites/default/files/styles/full_width/public/im…" style="width: 650px; height: 325px;" /></a></p> + + <p>The Tor Project achieved a lot in the last year. We spent 2018 fighting for the fundamental human rights to privacy and freedom online and made our software more accessible than ever before.</p> + + <p>This is an important moment in time. More people are looking for solutions to effectively protect their privacy. <a href="https://blog.torproject.org/strength-numbers-usable-tools-dont-need-be-inva…">93% of the people we met</a> doing 1:1 usability studies said they knew they needed some protection online. More people need robust censorship circumvention tools as <a href="https://blog.torproject.org/strength-numbers-internet-freedom-line">internet freedom declines</a> around the world. More people understand the risks that come from <a href="https://www.theguardian.com/technology/2019/jan/20/shoshana-zuboff-age-of-s…">surveillance as the business model</a> of the internet. And we have been working hard to make Tor more accessible than ever before with the goal that anyone online can enjoy the protections our software provides.</p> + + <p>In 2018, we:</p> + + <ul> + <li> + <p>Gave Tor Browser a UX overhaul with the launch of <a href="https://blog.torproject.org/new-release-tor-browser-80">Tor Browser 8.0</a>, making it easier and friendlier to use than ever.</p> + </li> + <li> + <p>Made it easier for people in countries that censor the internet and censor Tor to circumvent censorship with the ability to fetch bridges from inside Tor Browser.</p> + </li> + <li> + <p><a href="https://blog.torproject.org/strength-numbers-internet-freedom-movement-must…">Localized</a> Tor Browser into 9 previously unsupported languages, bringing the number of available languages to 25.</p> + </li> + <li> + <p>Launched the alpha version of Tor Browser for Android.</p> + </li> + <li> + <p>Improved our Core Tor code for mobile devices, optimizing its performances and making it easier for third party mobile apps to embed Tor.</p> + </li> + <li> + <p>Traveled to meet our users <a href="https://blog.torproject.org/strength-numbers-usable-tools-dont-need-be-inva…">face-to-face</a> and get feedback without using popular and invasive data-collection practices.</p> + </li> + <li> + <p>Improved the security of v3 onion services with the <a href="https://blog.torproject.org/announcing-vanguards-add-onion-services">vanguards add-on</a>.</p> + </li> + <li> + <p>Published <a href="https://ooni.torproject.org/post/ooni-in-2018/">10 research reports through OONI</a> on censorship and network disruptions happening around the world.</p> + </li> + <li> + <p>Pulled in a <a href="https://blog.torproject.org/strength-numbers-final-count">record number of donations</a> as we <a href="https://techcrunch.com/2019/01/11/tor-lessens-reliance-us-grants/">reduced our reliance</a> on government funding. We received donations from 115 countries around the world.</p> + </li> + <li> + <p>Said <a href="https://blog.torproject.org/strength-numbers-onion-blooms">goodbye to Shari Steele</a>, who helped usher the Tor Project into a new stage of organizational maturity, and welcomed our new Executive Director, <a href="https://blog.torproject.org/strength-numbers-community-key">Isabela Bagueros</a>.</p> + </li> + </ul> + + <p>These developments, plus the reality of threats everyone faces online, make <a href="https://www.wired.com/story/tor-anonymity-easier-than-ever/">2019 the year to try Tor</a>.</p> + + <p><a href="https://blog.torproject.org/expect-more-tor-2019">Find out </a>what you can expect from us this year.</p> + + <hr /> + <h1><span class="quickedit-field" data-quickedit-field-id="node/1684/title/en/full" property="schema:name">Tor Browser at TPL: Defending Intellectual Freedom, and Winning Awards Doing So</span><span class="quickedit-field" data-quickedit-field-id="node/1578/title/en/full" property="schema:name"> </span></h1> + + <p><span class="quickedit-field" data-quickedit-field-id="node/1684/title/en/full" property="schema:name"><a href="https://blog.torproject.org/tor-browser-tpl-defending-intellectual-freedom-…" target="_blank"><img alt="toronto-library" src="https://blog.torproject.org/sites/default/files/styles/full_width/public/im…" style="width: 650px; height: 378px;" /></a></span></p> + + <p><em>Guest post by Jonathon Hodge, Digital Literacy Service Lead, Toronto Public Library</em></p> + + <p>Every public library worker will know that person: the one who is worried about being spied on.</p> + + <p>For a long time in public libraries, that person was treated with the kindness and respect we treat every person, regardless of whether we felt that their concerns may have been overblown. The difference between that bygone past and today, is that today, that person is right! The internet is <a href="https://www.newyorker.com/news/amy-davidson/the-n-s-a-verizon-scandal">spying on them</a>; it’s <a href="https://www.theatlantic.com/technology/archive/2014/08/google-knows-you-bet…">spying on all of us</a>. Even if we <a href="https://www.recode.net/2018/4/20/17254312/facebook-shadow-profiles-data-col…">don’t use it very much</a>. Public libraries have long offered effective guidance to the wealth of information society produces. So the question today is, ‘Are we doing enough for ‘that person’, and by extension, for all of us?’</p> + + <p>In Toronto, we felt that the answer was NO. Our communities let us know that they do not know enough about the actual threats they contend with on the internet, they do not know what tools to use or actions to take to protect themselves.</p> + + <p>Public librarians can do a great deal to arm our users with the knowledge, the tools, and the confidence to navigate the surveillance society online. We in Toronto felt that Tor Browser should be the centerpiece of a multi-vector <a href="https://torontopubliclibrary.typepad.com/news_releases/2016/11/toronto-publ…">Digital Privacy Initiative</a>, that combines privacy education, and technology training and providing privacy-enabling tools at the point of service.</p> + + <p><a href="https://blog.torproject.org/tor-browser-tpl-defending-intellectual-freedom-…">Find out more about TPL's initiative</a>.</p> + + <hr /> + <h1>New Releases</h1> + + <h2>Tor Browser 8.0.5</h2> + + <p>This new release updates Firefox to 60.5.0esr and Tor to the first stable release in the 0.3.5 series, 0.3.5.7. <a href="https://blog.torproject.org/new-release-tor-browser-805">Full changelog</a>.</p> + + <h2>Tor 0.4.01-alpha</h2> + + <p>Tor 0.4.0.1-alpha is the first release in the new 0.4.0.x series. It introduces improved features for power and bandwidth conservation, more accurate reporting of bootstrap progress for user interfaces, and an experimental backend for an exciting new adaptive padding feature. <a href="https://blog.torproject.org/new-release-tor-0401-alpha">Full changelog</a>.</p> + + <h2 class="title"><span class="quickedit-field" data-quickedit-field-id="node/1669/title/en/teaser" property="schema:name">Tor 0.3.5.7</span></h2> + + <p>The Tor 0.3.5 series includes several new features and performance improvements, including client authorization for v3 onion services, cleanups to bootstrap reporting, support for improved bandwidth measurement tools, experimental support for NSS in place of OpenSSL, and much more. <a href="https://blog.torproject.org/new-releases-tor-0357-03410-and-03311">Full changelog</a>.</p> + + <hr /> + <h1>Upcoming Events with Tor</h1> + + <p><a href="https://blog.torproject.org/events/fosdem-brussels">FOSDEM</a>. Brussels, Belgium. 2-3 February, 2019.</p> + + <p><a href="https://blog.torproject.org/events/tor-meetup-berlin">Tor Meetup</a>. Berlin, Germany. 2 February, 2019.</p> + + <p><a href="https://blog.torproject.org/events/it-defense-stuttgart">IT Defense</a>. Stuttgart, Germany. 6-8 February, 2019.</p> + + <p><a href="https://blog.torproject.org/events/libreplanet-boston">LibrePlanet</a>. Boston, USA. 23-24 March, 2019.</p> + + <p><a href="https://blog.torproject.org/events/know-2019-vegas-0">KNOW Conference</a>. Las Vegas, USA. 24-27 March, 2019.</p> + + <hr /> + <h1>Join Our Community</h1> + + <p>Getting involved with Tor is easy. <a href="https://trac.torproject.org/projects/tor/wiki/TorRelayGuide">Run a relay</a> to make the network faster and more decentralized.</p> + + <p>Learn about each of our <a href="https://trac.torproject.org/projects/tor/wiki/WikiStart#Teams">teams </a>and start collaborating.</p> + + <p><a href="https://torproject.org/donate/donate-sin-tn3">Donate</a> to help keep Tor fast, strong, and secure.</p> + + <div style="background-color: #68b030; padding: 6px 8px 6px 8px; +-webkit-border-radius:3px; border-radius:3px; margin: 0 auto; width:200px; text-align: center;"><a href="https://donate.torproject.org" style="font-size: 24px; font-family: Source sans pro, Helvetica, Arial, sans-serif; font-weight: bold; color: #ffffff; text-decoration: none; display:inline-block;" target="_blank">DONATE</a></div> + </td> + </tr> + <tr> + <td style="padding:0 15px;"> + <p>The Tor Project is a US 501(c)(3) non-profit organization advancing human rights and freedoms by creating and deploying free and open-source anonymity and privacy technologies, supporting their unrestricted availability and use, and furthering their scientific and popular understanding.</p> + + <hr /> + <p><a href="https://facebook.com/torproject"><img alt="" src="https://blog.torproject.org/sites/default/files/inline-images/tor-facebook.…" style="width: 25px; height: 25px; margin: 3px;" /></a><a href="https://twitter.com/torproject"> <img alt="tor-twitter" src="https://blog.torproject.org/sites/default/files/inline-images/tor-twitter.p…" style="width: 25px; height: 25px; margin: 3px;" /></a><a href="https://instagram.com/torproject"> <img alt="tor-insta" src="https://blog.torproject.org/sites/default/files/inline-images/instagram-tor…" style="width: 25px; height: 25px; margin: 3px;" /></a></p> + + <br /> + <a href="https://torproject.org">torproject.org</a></small></p> + </td> + </tr> + </tbody> + </table> + </td> + </tr> + </tbody> + </table> + </td> + </tr> + </tbody> + </table> + </td> + </tr> + </tbody> +</table> + +<table bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" id="backgroundTable" st-sortable="left-image" width="100%"> + <tbody> + <tr> + <td> + <table align="center" border="0" cellpadding="0" cellspacing="0" class="devicewidth" width="650"> + <tbody> + <tr> + <td width="100%"> + <table align="center" bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" class="devicewidth" width="650"> + <tbody> + <tr> + <td> </td> + </tr> + </tbody> + </table> + </td> + </tr> + </tbody> + </table> + </td> + </tr> + </tbody> +</table> + diff --git a/content/archive/expect-more-from-tor/text/contents.lr b/content/archive/expect-more-from-tor/text/contents.lr new file mode 100644 index 0000000..0f2938a --- /dev/null +++ b/content/archive/expect-more-from-tor/text/contents.lr @@ -0,0 +1,97 @@ +_model: post +--- +_template: post.html +--- +author: steph(a)torproject.org +--- +pub_date: 2019-01-31 +--- +title: Expect More from Tor in 2019 +--- +body: + +## Expect More from Tor in 2019 ## + +The Tor Project achieved a lot in the last year. We spent 2018 fighting for the fundamental human rights to privacy and freedom online and made our software more accessible than ever before. + +In the last year, a significant shift took place in the public understanding of how big tech handles our sensitive, personal information, and how these companies build tools that further censorship in repressive places. For instance, Google’s leaked plans to launch a censored search engine in China were met with global protest. The world is watching. + +At the Tor Project, this is an important moment in time. More people are looking for solutions to effectively protect their privacy. 93% of the people we met doing 1:1 usability studies said they knew they needed some protection online. More people need robust censorship circumvention tools as internet freedom declines around the world. More people understand the risks that come from surveillance as the business model of the internet. And we have been working hard to make Tor more accessible than ever before with the goal that anyone online can enjoy the protections our software provides. + +In 2018, we: + +- Gave Tor Browser a UX overhaul with the launch of Tor Browser 8.0, making it easier and friendlier to use than ever. + +- Made it easier for people in countries that censor the internet and censor Tor to circumvent censorship with the ability to fetch bridges from inside Tor Browser. + +- Localized Tor Browser into 9 previously unsupported languages, bringing the number of available languages to 25. + +- Launched the alpha version of Tor Browser for Android. + +- Improved our Core Tor code for mobile devices, optimizing its performances and making it easier for third party mobile apps to embed Tor. + +- Traveled to meet our users face-to-face and get feedback without using popular and invasive data-collection practices. + +- Improved the security of v3 onion services with the vanguards add-on. + +- Published 10 research reports through OONI on censorship and network disruptions happening around the world. + +- Pulled in a record number of donations as we reduced our reliance on government funding. We received donations from 115 countries around the world. + +- Said goodbye to Shari Steele, who helped usher the Tor Project into a new stage of organizational maturity, and welcomed our new Executive Director, Isabela Bagueros. + +These developments, plus the reality of threats everyone faces online, make 2019 the year to try Tor. + +Find out what you can expect from us this year: https://blog.torproject.org/expect-more-tor-2019 + +## Tor Browser at TPL: Defending Intellectual Freedom, and Winning Awards Doing So // + +Guest post by Jonathon Hodge, Digital Literacy Service Lead, Toronto Public Library + +Every public library worker will know that person: the one who is worried about being spied on. + +For a long time in public libraries, that person was treated with the kindness and respect we treat every person, regardless of whether we felt that their concerns may have been overblown. The difference between that bygone past and today, is that today, that person is right! The internet is spying on them; it’s spying on all of us. Even if we don’t use it very much. Public libraries have long offered effective guidance to the wealth of information society produces. So the question today is, ‘Are we doing enough for ‘that person’, and by extension, for all of us?’ + +In Toronto, we felt that the answer was NO. Our communities let us know that they do not know enough about the actual threats they contend with on the internet, they do not know what tools to use or actions to take to protect themselves. + +Public librarians can do a great deal to arm our users with the knowledge, the tools, and the confidence to navigate the surveillance society online. We in Toronto felt that Tor Browser should be the centerpiece of a multi-vector Digital Privacy Initiative, that combines privacy education, and technology training and providing privacy-enabling tools at the point of service. + +With the software being easy to install and maintain, and with a new public appetite for secure technology, I would encourage other public libraries to install Tor Browser. Our professional defense of intellectual freedom can no longer exist only at the realm of policy. In this age when our tech spies on us for the sake of massive internet companies and the State, our defense must be a technological one as well. That tech is Tor. + +Find out more about TPL's initiative: https://blog.torproject.org/tor-browser-tpl-defending-intellectual-freedom-… + +## New Releases ## + +Tor Browser 8.0.5 +This new release updates Firefox to 60.5.0esr and Tor to the first stable release in the 0.3.5 series, 0.3.5.7. Full changelog: https://blog.torproject.org/new-release-tor-browser-805 + +Tor 0.4.01-alpha +This is the first release in the new 0.4.0.x series. It introduces improved features for power and bandwidth conservation, more accurate reporting of bootstrap progress for user interfaces, and an experimental backend for an exciting new adaptive padding feature. Full changelog: https://blog.torproject.org/new-release-tor-0401-alpha + +Tor 0.3.5.7 +The Tor 0.3.5 series includes several new features and performance improvements, including client authorization for v3 onion services, cleanups to bootstrap reporting, support for improved bandwidth- measurement tools, experimental support for NSS in place of OpenSSL, and much more. Full changelog: https://blog.torproject.org/new-releases-tor-0357-03410-and-03311 + +## Upcoming Events with Tor ## + +FOSDEM. Brussels, Belgium. 2-3 February, 2019. https://blog.torproject.org/events/fosdem-brussels +Tor Meetup. Berlin, Germany. 2 February, 2019. https://blog.torproject.org/events/tor-meetup-berlin +IT Defense. Stuttgart, Germany. 6-8 February, 2019. https://blog.torproject.org/events/it-defense-stuttgart +LibrePlanet. Boston, USA. 23-24 March, 2019. https://blog.torproject.org/events/libreplanet-boston +KNOW Conference. Las Vegas, USA. 24-27 March, 2019. https://blog.torproject.org/events/know-2019-vegas-0 + + +## Join Our Community ## + +Getting involved with Tor is easy. Run a relay to make the network faster and more decentralized: https://trac.torproject.org/projects/tor/wiki/TorRelayGuide + +Learn about each of our teams and start collaborating: https://trac.torproject.org/projects/tor/wiki/WikiStart#Teams + +Donate to help keep Tor fast, strong, and secure. https://donate.torproject.org + +-- + +The Tor Project is a US 501(c)(3) non-profit organization advancing human rights and freedoms by creating and deploying free and open-source anonymity and privacy technologies, supporting their unrestricted availability and use, and furthering their scientific and popular understanding. + +Twitter: https://twitter.com/torproject +Facebook: https://facebook.com/torproject +Instagram: https://instagram.com/torproject diff --git a/newsletter.lektorproject b/newsletter.lektorproject index 3ecfebd..7e924e8 100644 --- a/newsletter.lektorproject +++ b/newsletter.lektorproject @@ -1,10 +1,8 @@ [project] -name = Tor Newsletter Archive +name = Tor Newsletter Archive [alternatives.en] name = English primary = yes url_prefix = / locale = en_US - -

1 0

[newsletter/staging] Update menu
by hiro＠torproject.org 27 Mar '19

27 Mar '19

commit d5fe8df3cebb78d3eaf6e67d6f281c24e2fdb434 Author: hiro <hiro(a)torproject.org> Date: Mon Feb 11 17:42:20 2019 +0100 Update menu --- databags/menu+en.json | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/databags/menu+en.json b/databags/menu+en.json index d167ebd..94551ae 100644 --- a/databags/menu+en.json +++ b/databags/menu+en.json @@ -1,18 +1,30 @@ { - "archive": { - "path": "/archive", - "label": "Archive" - }, - "about": { - "path": "https://www.torproject.org/", - "label": "About Tor" - }, - "donate":{ - "path": "https://donate.torproject.org/", - "label": "Donate" - }, - "RSS":{ - "path": "/rss", - "label": "RSS" - } + "archive": { + "path": "/archive", + "label": "Archive" + }, + "about": { + "path": "https://www.torproject.org/", + "label": "About Tor" + }, + "documentation":{ + "path": "https://www.torproject.org/docs/documentation.html.en", + "label": "Documentation" + }, + "support":{ + "path": "https://support.torproject.org/", + "label": "Support" + }, + "blog":{ + "path": "https://blog.torproject.org/", + "label": "Blog" + }, + "donate":{ + "path": "https://donate.torproject.org/", + "label": "Donate" + }, + "RSS":{ + "path": "/rss", + "label": "RSS" + } }

1 0

[sbws/master] fix: relayprioritizer: number of prioritized relays
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit 4fa29b025ea353a8cb55a2e81298c092297a3287 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 22 09:17:42 2019 +0000 fix: relayprioritizer: number of prioritized relays The number of prioritized relays was set to the minimum, but in a small testing network this could be lesser than the number of relays in the network. --- sbws/lib/relayprioritizer.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbws/lib/relayprioritizer.py b/sbws/lib/relayprioritizer.py index bb64b06..4932ea8 100644 --- a/sbws/lib/relayprioritizer.py +++ b/sbws/lib/relayprioritizer.py @@ -143,7 +143,9 @@ class RelayPrioritizer: # Increment the number of times ``best_priority`` has been run. self.increment_priority_lists() # Increment the number of relays that have been "prioritized". - self.increment_priority_relays(upper_limit) + # Because in a small testing network the upper limit could be smaller + # than the number of relays in the network, use the length of the list. + self.increment_priority_relays(len(relays[0:upper_limit])) for relay in relays[0:upper_limit]: log.debug('Returning next relay %s with priority %f', relay.nickname, relay.priority)

1 0

[sbws/master] fix: destination: Multiply errors by the threads
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit 612807e3155d4a85d004995af0dc06b45b22905d Author: juga0 <juga(a)riseup.net> Date: Mon Mar 25 14:28:12 2019 +0000 fix: destination: Multiply errors by the threads Since when a destination fails, all the threads using it will also fail that moment. For now not checking which threads are actually using it. Also lower the time to retry. Closes: #29891. --- sbws/core/scanner.py | 19 +++++++++++++++++-- sbws/globals.py | 4 +++- sbws/lib/destination.py | 35 ++++++++++++++++++++++++----------- tests/unit/lib/test_destination.py | 12 ++++++------ 4 files changed, 50 insertions(+), 20 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index 32752ca..fdb44eb 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -29,7 +29,6 @@ from multiprocessing.dummy import Pool import time import os import logging -import requests import random from .. import settings @@ -367,6 +366,19 @@ def measure_relay(args, conf, destinations, cb, rl, relay): def dispatch_worker_thread(*a, **kw): + # If at the point where the relay is actually going to be measured there + # are not any functional destinations or the `end_event` is set, do not + # try to start measuring the relay, since it will fail anyway. + try: + # a[2] is the argument `destinations` + functional_destinations = a[2].functional_destinations + # In case the arguments or the method change, catch the possible exceptions + # but ignore here that there are not destinations. + except (IndexError, TypeError): + log.debug("Wrong argument or attribute.") + functional_destinations = True + if not functional_destinations or settings.end_event.is_set(): + return None return measure_relay(*a, **kw) @@ -530,7 +542,10 @@ def main_loop(args, conf, controller, relay_list, circuit_builder, result_dump, loop_tstop = time.time() loop_tdelta = (loop_tstop - loop_tstart) / 60 - log.debug("Measured %s relays in %s minutes", num_relays, loop_tdelta) + # At this point, we know the relays that were queued to be measured. + # That does not mean they were actually measured. + log.debug("Attempted to measure %s relays in %s minutes", + num_relays, loop_tdelta) # In a testing network, exit after first loop if controller.get_conf('TestingTorNetwork') == '1': log.info("In a testing network, exiting after the first loop.") diff --git a/sbws/globals.py b/sbws/globals.py index 5003973..723625d 100644 --- a/sbws/globals.py +++ b/sbws/globals.py @@ -127,7 +127,9 @@ DESTINATION_VERIFY_CERTIFICATE = True # whether the destination is functional or not. NUM_DESTINATION_ATTEMPTS_STORED = 10 # Time to wait before trying again a destination that wasn't functional. -DELTA_SECONDS_RETRY_DESTINATION = 60 * 60 * 3 +# Because intermitent failures with CDN destinations, start trying again +# after 5 min. +DELTA_SECONDS_RETRY_DESTINATION = 60 * 5 # Number of consecutive times a destination can fail before considering it # not functional. MAX_NUM_DESTINATION_FAILURES = 3 diff --git a/sbws/lib/destination.py b/sbws/lib/destination.py index 21a907c..3ccd94f 100644 --- a/sbws/lib/destination.py +++ b/sbws/lib/destination.py @@ -229,12 +229,19 @@ class Destination: the time to try again is incremented and resetted as soon as the destination does not fail. """ + # NOTE: does a destination fail because several threads are using + # it at the same time? + # If a destination fails for 1 minute and there're 3 threads, the + # 3 threads will fail. + # Failed the last X consecutive times if self._are_last_attempts_failures(): + # The log here will appear in all the the queued + # relays and threads. log.warning("The last %s times the destination %s failed." - "It will not be used again in %s hours.\n", + "Disabled for %s minutes.", self._max_num_failures, self.url, - self._delta_seconds_retry / 60 / 60) + self._delta_seconds_retry / 60) log.warning("Please, add more destinations or increment the " "number of maximum number of consecutive failures " "in the configuration.") @@ -285,19 +292,22 @@ class Destination: return p @staticmethod - def from_config(conf_section, max_dl): + def from_config(conf_section, max_dl, number_threads): assert 'url' in conf_section url = conf_section['url'] verify = _parse_verify_option(conf_section) try: - max_num_failures = conf_section.getint('max_num_failures') + # Because one a destination fails, all the threads that are using + # it at that moment will fail too, multiply by the number of + # threads. + max_num_failures = (conf_section.getint('max_num_failures') + or MAX_NUM_DESTINATION_FAILURES) except ValueError: - log.warning("Configuration max_num_failures is wrong, ignoring.") - max_num_failures = None - if max_num_failures: - return Destination(url, max_dl, verify, max_num_failures) - else: - return Destination(url, max_dl, verify) + # If the operator did not setup the number, set to the default. + max_num_failures = MAX_NUM_DESTINATION_FAILURES + + max_num_failures *= number_threads + return Destination(url, max_dl, verify, max_num_failures) class DestinationList: @@ -331,7 +341,10 @@ class DestinationList: log.debug('Loading info for destination %s', key) dests.append(Destination.from_config( conf[dest_sec], - conf.getint('scanner', 'max_download_size'))) + # Multiply by the number of threads since all the threads will + # fail at the same time. + conf.getint('scanner', 'max_download_size'), + conf.getint('scanner', 'measurement_threads'))) if len(dests) < 1: msg = 'No enabled destinations in config. Please see '\ 'docs/source/man_sbws.ini.rst" or "man 5 sbws.ini" ' \ diff --git a/tests/unit/lib/test_destination.py b/tests/unit/lib/test_destination.py index fc8b489..ca2ff21 100644 --- a/tests/unit/lib/test_destination.py +++ b/tests/unit/lib/test_destination.py @@ -5,9 +5,9 @@ from sbws.lib import destination def test_destination_is_functional(): - eight_hours_ago = datetime.utcnow() - timedelta(hours=8) - four_hours_ago = datetime.utcnow() - timedelta(hours=4) - two_hours_ago = datetime.utcnow() - timedelta(hours=2) + eleven_mins_ago = datetime.utcnow() - timedelta(minutes=11) + six_mins_ago = datetime.utcnow() - timedelta(minutes=6) + four_mins_ago = datetime.utcnow() - timedelta(minutes=4) d = destination.Destination('unexistenturl', 0, False) assert d.is_functional() @@ -29,19 +29,19 @@ def test_destination_is_functional(): d.add_failure() d.add_failure() # And last failure was 2h ago - d.add_failure(two_hours_ago) + d.add_failure(four_mins_ago) assert d._are_last_attempts_failures() assert not d._is_last_try_old_enough() assert not d.is_functional() # But if the last failure was 4h ago, try to use it again # And last failure was 4h ago - d.add_failure(four_hours_ago) + d.add_failure(six_mins_ago) assert d._is_last_try_old_enough() assert d.is_functional() # If last failure was 8h ago, try to use it again again - d.add_failure(eight_hours_ago) + d.add_failure(eleven_mins_ago) assert d._is_last_try_old_enough() assert d.is_functional()

1 0

[sbws/master] fix: Store the error when there is not destination
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit e835710e4a59c7255211e5a67a4e2a3b5c831371 Author: juga0 <juga(a)riseup.net> Date: Fri Mar 22 16:15:56 2019 +0000 fix: Store the error when there is not destination --- sbws/core/scanner.py | 27 ++++++++++++++++----------- sbws/lib/resultdump.py | 9 +++++++++ 2 files changed, 25 insertions(+), 11 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index d82aaca..5b28637 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -13,7 +13,7 @@ from ..lib.circuitbuilder import GapsCircuitBuilder as CB from ..lib.resultdump import ResultDump from ..lib.resultdump import ( ResultSuccess, ResultErrorCircuit, ResultErrorStream, - ResultErrorSecondRelay, ResultError, # ResultErrorDestination + ResultErrorSecondRelay, ResultError, ResultErrorDestination ) from ..lib.relaylist import RelayList from ..lib.relayprioritizer import RelayPrioritizer @@ -261,20 +261,25 @@ def measure_relay(args, conf, destinations, cb, rl, relay): ] # Pick a destionation dest = destinations.next() - # If there is no any destination at this point, it can not continue. + # When there're no any functional destinations. if not dest: - # XXX: this should return a ResultError - # instead of stopping the scanner once a destination can be recovered. + # NOTE: When there're still functional destinations but only one of + # them fail, the error will be included in `ResultErrorStream`. + # Since this is being executed in a thread, the scanner can not + # be stop here, but the `end_event` signal can be set so that the + # main thread stop the scanner. + # It might be useful to store the fact that the destinations fail, + # so store here the error, and set the signal once the error is stored + # (in `resultump`). log.critical("There are not any functional destinations.\n" "It is recommended to set several destinations so that " "the scanner can continue if one fails.") - # NOTE: Because this is executed in a thread, stop_threads can not - # be call from here, it has to be call from the main thread. - # Instead set the singleton end event, that will call stop_threads - # from the main process. - # Errors with only one destination are set in ResultErrorStream. - settings.end_event.set() - return None + reason = "No functional destinations" + # Resultdump will set end_event after storing the error + return [ + ResultErrorDestination(relay, [], '', our_nick, msg=reason), + ] + # Pick a relay to help us measure the given relay. If the given relay is an # exit, then pick a non-exit. Otherwise pick an exit. helper = None diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py index 2ebfa20..f7ec55b 100644 --- a/sbws/lib/resultdump.py +++ b/sbws/lib/resultdump.py @@ -776,6 +776,15 @@ class ResultDump: "destination {}: {}".format( result.fingerprint, result.nickname, result.circ, result.dest_url, result.msg) + # When the error is that there are not more functional destinations. + if result.type == "error-destination": + log.info("Shutting down because there are not functional " + "destinations.") + # NOTE: Because this is executed in a thread, stop_threads can not + # be call from here, it has to be call from the main thread. + # Instead set the singleton end event, that will call stop_threads + # from the main process. + settings.end_event.set() log.info(msg) def enter(self):

1 0

[sbws/master] fix: docs: Remove contributing from readme
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit a9dc014e3fc04d13d8595127a1a5dcbba66adb54 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 26 08:39:30 2019 +0000 fix: docs: Remove contributing from readme and move contributing from the root directory to the docs/ one, since it links to testing but testing is not in the root directory. This way when a user only reads the readme locally, does not try to find testing in the root directory and instead will search for more documentation in docs/. --- README.md | 6 ------ docs/source/CONTRIBUTING.rst | 1 - CONTRIBUTING.rst => docs/source/contributing.rst | 0 docs/source/index.rst | 2 +- 4 files changed, 1 insertion(+), 8 deletions(-) diff --git a/README.md b/README.md index 8c57464..e796e46 100644 --- a/README.md +++ b/README.md @@ -42,12 +42,6 @@ Deploying and running See [./DEPLOY.rst](DEPLOY.rst) (in local directory or GitHub) or [DEPLOY.html](DEPLOY.html) (local build or Read the Docs). -Contributing --------------- - -See [./CONTRIBUTING.rst](CONTRIBUTING.rst) (in local directory or GitHub) or -[CONTRIBUTING.html](CONTRIBUTING.html) (local build or Read the Docs). - Changelog -------------- diff --git a/docs/source/CONTRIBUTING.rst b/docs/source/CONTRIBUTING.rst deleted file mode 120000 index e9a8ba6..0000000 --- a/docs/source/CONTRIBUTING.rst +++ /dev/null @@ -1 +0,0 @@ -../../CONTRIBUTING.rst \ No newline at end of file diff --git a/CONTRIBUTING.rst b/docs/source/contributing.rst similarity index 100% rename from CONTRIBUTING.rst rename to docs/source/contributing.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index e6c5d6e..9806ef3 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -19,7 +19,6 @@ and in ``sbws`` Debian package: README INSTALL DEPLOY - CONTRIBUTING CHANGELOG AUTHORS man_sbws @@ -37,6 +36,7 @@ Included in the .. toctree:: :maxdepth: 1 + contributing testing documenting how_works

1 0

[sbws/master] fix: scanner: catch all possible exceptions
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit eb749c09cc28ff5297b4ac770543a6bfbcb996ba Author: juga0 <juga(a)riseup.net> Date: Sat Mar 23 13:28:31 2019 +0000 fix: scanner: catch all possible exceptions calling requests.get, since RequestException does not catch all. --- sbws/core/scanner.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index 5b28637..32752ca 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -98,8 +98,11 @@ def timed_recv_from_server(session, dest, byte_range): try: # headers are merged with the session ones, not overwritten. session.get(dest.url, headers=HTTP_GET_HEADERS, verify=dest.verify) - # Catch any `requests` exception, so that it can stored in the Result - except requests.exceptions.RequestException as e: + # All `requests` exceptions could be caught with + # `requests.exceptions.RequestException`, but it seems that `requests` + # does not catch all the ssl exceptions and urllib3 doesn't seem to have + # a base exception class. + except Exception as e: log.debug(e) return False, e end_time = time.time()

1 0

[sbws/master] fix: scanner: log traceback instead of print it
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit 0f33d3a4a5d21c00b1210148e624f7098d0ddf0e Author: juga0 <juga(a)riseup.net> Date: Tue Mar 26 13:28:34 2019 +0000 fix: scanner: log traceback instead of print it --- sbws/core/scanner.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py index fdb44eb..d3ed5b1 100644 --- a/sbws/core/scanner.py +++ b/sbws/core/scanner.py @@ -70,7 +70,7 @@ def dumpstacks(): for thread_id, stack in sys._current_frames().items(): log.critical("Thread: %s(%d)", thread_id2name.get(thread_id, ""), thread_id) - log.critical(traceback.print_stack(stack)) + log.critical(traceback.format_stack("".join(stack))) # If logging level is less than DEBUG (more verbose), start pdb so that # developers can debug the issue. if log.getEffectiveLevel() < logging.DEBUG: @@ -635,8 +635,10 @@ def force_get_results(pending_results): log.critical(FILLUP_TICKET_MSG) # If the exception happened in the threads, `log.exception` does # not have the traceback. - log.warning("traceback %s", - traceback.print_exception(type(e), e, e.__traceback__)) + # Using `format_exception` instead of of `print_exception` to show + # the traceback in all the log handlers. + log.warning("".join(traceback.format_exception( + type(e), e, e.__traceback__))) def run_speedtest(args, conf):

1 0

[sbws/master] fix: relaylist: catch exception accessing exit_policy
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit 32f283f75965bde77ff2b3e6208b8a9f106e7708 Author: juga0 <juga(a)riseup.net> Date: Tue Mar 26 09:24:42 2019 +0000 fix: relaylist: catch exception accessing exit_policy --- sbws/lib/relaylist.py | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py index febc717..289e0d2 100644 --- a/sbws/lib/relaylist.py +++ b/sbws/lib/relaylist.py @@ -228,9 +228,23 @@ class Relay: """ assert isinstance(port, int) # if dind't get the descriptor, there isn't exit policy - if not self.exit_policy: + # When the attribute is gotten in getattr(self._desc, "exit_policy"), + # is possible that stem's _input_rules is None and raises an exception + # (#29899): + # File "/usr/lib/python3/dist-packages/sbws/lib/relaylist.py", line 117, in can_exit_to_port # noqa + # if not self.exit_policy: + # File "/usr/lib/python3/dist-packages/stem/exit_policy.py", line 512, in __len__ # noqa + # return len(self._get_rules()) + # File "/usr/lib/python3/dist-packages/stem/exit_policy.py", line 464, in _get_rules # noqa + # for rule in decompressed_rules: + # TypeError: 'NoneType' object is not iterable + # Therefore, catch the exception here. + try: + if self.exit_policy: + return self.exit_policy.can_exit_to(port=port) + except TypeError: return False - return self.exit_policy.can_exit_to(port=port) + return False def is_exit_not_bad_allowing_port(self, port): return (Flag.BADEXIT not in self.flags and

1 0

[sbws/master] CI: disable doclinks temporally
by juga＠torproject.org 27 Mar '19

27 Mar '19

commit 4455b81c5bc3c872f262dba39ca266aff047bb26 Author: juga0 <juga(a)riseup.net> Date: Wed Mar 27 07:57:47 2019 +0000 CI: disable doclinks temporally since Travis is failing to check github.com with forbidden error, but it doesn't fail locally (https://travis-ci.org/torproject/sbws/jobs/511862070#L1938) --- .travis.yml | 2 +- tox.ini | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index aa477ef..1afe725 100644 --- a/.travis.yml +++ b/.travis.yml @@ -9,7 +9,7 @@ script: - tox # This is not in included in the tox envlist, in order to don't need Internet # when running tox - - tox -e doclinks + # - tox -e doclinks - tox -e clean before_install: diff --git a/tox.ini b/tox.ini index 7109cc5..4136036 100644 --- a/tox.ini +++ b/tox.ini @@ -4,7 +4,7 @@ envlist = py35, py36, inst, setup, unit, integration, lint, stats, doc [travis] python = - 3.5: py35, inst, setup, unit, integration, lint, doc, doclinks + 3.5: py35, inst, setup, unit, integration, lint, doc 3.6: py36, inst, setup, unit, integration [testenv]

1 0