April 2019 - tor-commits - lists.torproject.org

[snowflake/master] Ported log sanitization to client
by cohosh＠torproject.org 16 Apr '19

16 Apr '19

commit b4a4a256db08d14093edf442ab77e8a1763ee80b Author: Cecylia Bocovich <cohosh(a)torproject.org> Date: Fri Apr 12 09:52:23 2019 -0400 Ported log sanitization to client --- client/snowflake.go | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/client/snowflake.go b/client/snowflake.go index 006de6b..715f83f 100644 --- a/client/snowflake.go +++ b/client/snowflake.go @@ -16,6 +16,7 @@ import ( "git.torproject.org/pluggable-transports/goptlib.git" sf "git.torproject.org/pluggable-transports/snowflake.git/client/lib" + "git.torproject.org/pluggable-transports/snowflake.git/common/safelog" "github.com/keroserene/go-webrtc" ) @@ -77,6 +78,12 @@ func main() { webrtc.SetLoggingVerbosity(1) log.SetFlags(log.LstdFlags | log.LUTC) + // Don't write to stderr; versions of tor earlier than about + // 0.3.5.6 do not read from the pipe, and eventually we will + // deadlock because the buffer is full. + // https://bugs.torproject.org/26360 + // https://bugs.torproject.org/25600#comment:14 + var logOutput io.Writer = ioutil.Discard if *logFilename != "" { if *logToStateDir { stateDir, err := pt.MakeStateDir() @@ -91,15 +98,10 @@ func main() { log.Fatal(err) } defer logFile.Close() - log.SetOutput(logFile) - } else { - // Don't write to stderr; versions of tor earlier than about - // 0.3.5.6 do not read from the pipe, and eventually we will - // deadlock because the buffer is full. - // https://bugs.torproject.org/26360 - // https://bugs.torproject.org/25600#comment:14 - log.SetOutput(ioutil.Discard) + logOutput = logFile } + //We want to send the log output through our scrubber first + log.SetOutput(&safelog.LogScrubber{Output: logOutput}) log.Println("\n\n\n --- Starting Snowflake Client ---")

1 0

[stem/master] Log test tracebacks for all threads, and all active children
by atagar＠torproject.org 16 Apr '19

16 Apr '19

commit c3109eb9d081fe86981db963bfc9a92c8f910893 Author: teor <teor(a)torproject.org> Date: Thu Apr 11 14:42:09 2019 +1000 Log test tracebacks for all threads, and all active children Also: - ignore signals while in a signal handler, - log the current pid, - flush buffers after logging, and - terminate using os._exit() on ABRT, so the exception can't be caught. Closes 30122. --- run_tests.py | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 61 insertions(+), 6 deletions(-) diff --git a/run_tests.py b/run_tests.py index 3c1cde0e..e1b42744 100755 --- a/run_tests.py +++ b/run_tests.py @@ -6,6 +6,7 @@ Runs unit and integration tests. For usage information run this with '--help'. """ +import multiprocessing import os import signal import sys @@ -71,16 +72,71 @@ New capabilities are: """ +def enable_signal_handlers(): + """ + Enable signal handlers for USR1 and ABRT. + """ + signal.signal(signal.SIGABRT, log_traceback) + signal.signal(signal.SIGUSR1, log_traceback) + + +def disable_signal_handlers(): + """ + Ignore signals USR1 and ABRT. + """ + signal.signal(signal.SIGABRT, signal.SIG_IGN) + signal.signal(signal.SIGUSR1, signal.SIG_IGN) + + +def format_traceback(pid, ident, frame): + """ + Format the traceback for process pid and thread ident using the stack frame. + """ + if frame is not None: + return ('Traceback for thread %d in process %d:\n\n%s' % + (ident, pid, ''.join(traceback.format_stack(frame)))) + else: + return ('No traceback for thread %d in process %d.' % (ident, pid)) + + def log_traceback(sig, frame): """ - Signal handler that logs the present traceback, and aborts our process with - exit status -1 in the case of SIGABRT. + Signal handler that: + - logs the current thread id and pid, + - logs tracebacks for all threads, + - flushes stdio buffers, + - propagate the signal to multiprocessing.active_children(), and + - in the case of SIGABRT, aborts our process with exit status -1. + While this signal handler is running, other signals are ignored. """ - print('Signal %s received. Traceback:\n\n%s' % (sig, ''.join(traceback.format_stack(frame)))) + disable_signal_handlers() + + # format and log tracebacks + pid = os.getpid() + thread_tracebacks = [format_traceback(pid, ident, frame_) + for ident, frame_ in sys._current_frames().items()] + print('Signal %s received by thread %d in process %d:\n\n%s' % + (sig, threading.current_thread().ident, pid, + '\n\n'.join(thread_tracebacks))) + + # we're about to signal our children, and maybe do a hard abort, so flush + sys.stdout.flush() + + # propagate the signal to any multiprocessing children + pgid = os.getpgid(pid) + for p in multiprocessing.active_children(): + # avoid race conditions + if p.is_alive(): + os.kill(p.pid, sig) if sig == signal.SIGABRT: - sys.exit(-1) + # we need to use os._exit() to abort every thread in the interpreter, + # rather than raise a SystemExit exception that can be caught + os._exit(-1) + else: + # we're done: stop ignoring signals + enable_signal_handlers() def get_unit_tests(module_prefix = None): @@ -137,8 +193,7 @@ def main(): println('%s\n' % exc) sys.exit(1) - signal.signal(signal.SIGABRT, log_traceback) - signal.signal(signal.SIGUSR1, log_traceback) + enable_signal_handlers() test_config = stem.util.conf.get_config('test') test_config.load(os.path.join(test.STEM_BASE, 'test', 'settings.cfg'))

1 0

[stem/master] Note thread names rather than idents
by atagar＠torproject.org 16 Apr '19

16 Apr '19

commit a4751b2dbc692dd100cd678b23b51f1f5c08f1f9 Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Apr 16 09:09:29 2019 -0700 Note thread names rather than idents Thread ident numbers are unhelpful for troubleshooting hangs. If a thread hangs I need to know its name, not its internal numeric identifier. Also cleaning up the pid bits a tad. The pgid variable was unused, and there's no need to pass the pid of ourselves as an argument (it's static for the interpreter, so we can fetch it as needed). --- run_tests.py | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/run_tests.py b/run_tests.py index e1b42744..220208b4 100755 --- a/run_tests.py +++ b/run_tests.py @@ -88,15 +88,18 @@ def disable_signal_handlers(): signal.signal(signal.SIGUSR1, signal.SIG_IGN) -def format_traceback(pid, ident, frame): +def format_traceback(thread): """ Format the traceback for process pid and thread ident using the stack frame. """ + + frame = sys._current_frames().get(thread.ident, None) + if frame is not None: - return ('Traceback for thread %d in process %d:\n\n%s' % - (ident, pid, ''.join(traceback.format_stack(frame)))) + return ('Traceback for thread %s in process %d:\n\n%s' % + (thread.name, os.getpid(), ''.join(traceback.format_stack(frame)))) else: - return ('No traceback for thread %d in process %d.' % (ident, pid)) + return ('No traceback for thread %s in process %d.' % (thread.name, os.getpid())) def log_traceback(sig, frame): @@ -113,18 +116,16 @@ def log_traceback(sig, frame): disable_signal_handlers() # format and log tracebacks - pid = os.getpid() - thread_tracebacks = [format_traceback(pid, ident, frame_) - for ident, frame_ in sys._current_frames().items()] - print('Signal %s received by thread %d in process %d:\n\n%s' % - (sig, threading.current_thread().ident, pid, + thread_tracebacks = [format_traceback(thread) + for thread in threading.enumerate()] + print('Signal %s received by thread %s in process %d:\n\n%s' % + (sig, threading.current_thread().name, os.getpid(), '\n\n'.join(thread_tracebacks))) # we're about to signal our children, and maybe do a hard abort, so flush sys.stdout.flush() # propagate the signal to any multiprocessing children - pgid = os.getpgid(pid) for p in multiprocessing.active_children(): # avoid race conditions if p.is_alive():

1 0

[stem/master] Improve readability of thread stacktrace dumps
by atagar＠torproject.org 16 Apr '19

16 Apr '19

commit e1b90b5fc6df32d08bcc2c60ccfddf6fc22131ad Author: Damian Johnson <atagar(a)torproject.org> Date: Tue Apr 16 09:44:03 2019 -0700 Improve readability of thread stacktrace dumps Making a few adjustments to how we dump our stacktraces... * Adding dividers to improve readability. Here's an example of what our new dumps look like... ================================================================================ Signal SIGUSR1 received by thread MainThread in process 5819 -------------------------------------------------------------------------------- MainThread thread stacktrace File "./run_tests.py", line 434, in <module> main() File "./run_tests.py", line 316, in main task.join() # no-op if these haven't been run File "/home/atagar/Desktop/stem/test/task.py", line 276, in join self.result = self._background_process.join() File "/home/atagar/Desktop/stem/stem/util/system.py", line 281, in join self._process.join() File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join res = self._popen.wait(timeout) File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait return self.poll(0) File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll pid, sts = os.waitpid(self.pid, flag) File "./run_tests.py", line 86, in log_traceback for thread_name, stacktrace in test.output.thread_stacktraces().items(): File "/home/atagar/Desktop/stem/test/output.py", line 110, in thread_stacktraces stacktraces[thread.name] = ''.join(traceback.format_stack(frame)) ================================================================================ * Providing this output on stderr rather than stdout. Our test.output module's println() flushes, so there's no need to explicitly do that. * Noting the signal's name rather than its numeric identifier. * Dropping signal suspension. I'd be open to adding this back but without a usecase where it helps I'd prefer to bias toward keeping this simple. --- run_tests.py | 74 ++++++++++++++++++++-------------------------------------- test/output.py | 22 +++++++++++++++++ 2 files changed, 47 insertions(+), 49 deletions(-) diff --git a/run_tests.py b/run_tests.py index 220208b4..6d0951e9 100755 --- a/run_tests.py +++ b/run_tests.py @@ -72,62 +72,40 @@ New capabilities are: """ -def enable_signal_handlers(): - """ - Enable signal handlers for USR1 and ABRT. - """ - signal.signal(signal.SIGABRT, log_traceback) - signal.signal(signal.SIGUSR1, log_traceback) - - -def disable_signal_handlers(): - """ - Ignore signals USR1 and ABRT. - """ - signal.signal(signal.SIGABRT, signal.SIG_IGN) - signal.signal(signal.SIGUSR1, signal.SIG_IGN) - - -def format_traceback(thread): +def log_traceback(sig, frame): """ - Format the traceback for process pid and thread ident using the stack frame. + Dump the stacktraces of all threads on stderr. """ - frame = sys._current_frames().get(thread.ident, None) - - if frame is not None: - return ('Traceback for thread %s in process %d:\n\n%s' % - (thread.name, os.getpid(), ''.join(traceback.format_stack(frame)))) - else: - return ('No traceback for thread %s in process %d.' % (thread.name, os.getpid())) + # Attempt to get the name of our signal. Unfortunately the signal module + # doesn't provide a reverse mapping, so we need to get this ourselves + # from the attributes. + signal_name = str(sig) -def log_traceback(sig, frame): - """ - Signal handler that: - - logs the current thread id and pid, - - logs tracebacks for all threads, - - flushes stdio buffers, - - propagate the signal to multiprocessing.active_children(), and - - in the case of SIGABRT, aborts our process with exit status -1. - While this signal handler is running, other signals are ignored. - """ + for attr_name, value in signal.__dict__.items(): + if attr_name.startswith('SIG') and value == sig: + signal_name = attr_name + break - disable_signal_handlers() + lines = [ + '', # initial NL so we start on our own line + '=' * 80, + 'Signal %s received by thread %s in process %i' % (signal_name, threading.current_thread().name, os.getpid()), + ] - # format and log tracebacks - thread_tracebacks = [format_traceback(thread) - for thread in threading.enumerate()] - print('Signal %s received by thread %s in process %d:\n\n%s' % - (sig, threading.current_thread().name, os.getpid(), - '\n\n'.join(thread_tracebacks))) + for thread_name, stacktrace in test.output.thread_stacktraces().items(): + lines.append('-' * 80) + lines.append('%s thread stacktrace' % thread_name) + lines.append('') + lines.append(stacktrace) - # we're about to signal our children, and maybe do a hard abort, so flush - sys.stdout.flush() + lines.append('=' * 80) + println('\n'.join(lines)) # propagate the signal to any multiprocessing children + for p in multiprocessing.active_children(): - # avoid race conditions if p.is_alive(): os.kill(p.pid, sig) @@ -135,9 +113,6 @@ def log_traceback(sig, frame): # we need to use os._exit() to abort every thread in the interpreter, # rather than raise a SystemExit exception that can be caught os._exit(-1) - else: - # we're done: stop ignoring signals - enable_signal_handlers() def get_unit_tests(module_prefix = None): @@ -194,7 +169,8 @@ def main(): println('%s\n' % exc) sys.exit(1) - enable_signal_handlers() + signal.signal(signal.SIGABRT, log_traceback) + signal.signal(signal.SIGUSR1, log_traceback) test_config = stem.util.conf.get_config('test') test_config.load(os.path.join(test.STEM_BASE, 'test', 'settings.cfg')) diff --git a/test/output.py b/test/output.py index 8ca64372..0eedfa27 100644 --- a/test/output.py +++ b/test/output.py @@ -7,6 +7,8 @@ together for improved readability. """ import re +import threading +import traceback import sys import stem.util.enum @@ -92,6 +94,26 @@ def print_logging(logging_buffer): print('') +def thread_stacktraces(): + """ + Provides a dump of the stacktrace information for all active threads. + + :returns: **dict** that maps thread names to their stacktrace + """ + + stacktraces = {} + + for thread in threading.enumerate(): + frame = sys._current_frames().get(thread.ident, None) + + if frame: + stacktraces[thread.name] = ''.join(traceback.format_stack(frame)) + else: + stacktraces[thread.name] = 'No traceback available' + + return stacktraces + + def apply_filters(testing_output, *filters): """ Gets the tests results, possibly processed through a series of filters. The

1 0

[snowflake/master] Updated robots.txt handler on broker
by cohosh＠torproject.org 16 Apr '19

16 Apr '19

commit d4c5d4e65b89fb3da83826edfdd5cfd5bf4a0efb Author: Cecylia Bocovich <cohosh(a)torproject.org> Date: Mon Apr 15 16:43:21 2019 -0400 Updated robots.txt handler on broker Updated robots.txt file to disallow crawling as in ticket #29565. --- broker/broker.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/broker/broker.go b/broker/broker.go index b3a255e..7e2da8d 100644 --- a/broker/broker.go +++ b/broker/broker.go @@ -218,7 +218,7 @@ func debugHandler(ctx *BrokerContext, w http.ResponseWriter, r *http.Request) { func robotsTxtHandler(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "text/plain; charset=utf-8") - w.Write([]byte("User-agent: *\nDisallow:\n")) + w.Write([]byte("User-agent: *\nDisallow: /\n")) } func main() {

1 0

[tpo/master] Merge pull request #18 from torproject/master
by hiro＠torproject.org 16 Apr '19

16 Apr '19

commit 4c0501626f8c2650e71696763ef3711f7a6499a3 Merge: e168cfa 4743c46 Author: Antonela <hola(a)antonela.me> Date: Tue Apr 16 13:25:01 2019 -0300 Merge pull request #18 from torproject/master rebase databags/versions.ini | 10 ++++++++-- templates/download-android.html | 9 +++++---- templates/download-tor.html | 22 ++++++++++++++++++++-- templates/hero-download.html | 5 ++++- 4 files changed, 37 insertions(+), 9 deletions(-)

1 0

[tpo/master] Fix svg fallback
by hiro＠torproject.org 16 Apr '19

16 Apr '19

commit b4a7f8c0078e1b8f5f97d3c3d76274d2f1c4f0a2 Author: hiro <hiro(a)torproject.org> Date: Tue Apr 16 18:31:25 2019 +0200 Fix svg fallback --- templates/download.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/templates/download.html b/templates/download.html index 61ec23c..0e4befb 100644 --- a/templates/download.html +++ b/templates/download.html @@ -1,8 +1,8 @@ <div class="container py-3"> <div class="row"> <div class="col-sm-6"> - <div class="col-md-8 order-1"> - <img src="{{ '/static/images/download/get-connected.svg'|asseturl }}" class="" alt="{{ _("Get Connected") }}" /> + <div class="col-md-8 order-1 illo-container"> + <img src="{{ '/static/images/download/png/get-connected(a)3x.png'|asseturl }}" class="" alt="{{ _("Get Connected") }}" /> </div> <h3 class="text-primary">{{ _('Get connected') }}</h3> <div class="row pl-3 pr-5 text-tpo"> @@ -15,8 +15,8 @@ </div> </div> <div class="col-sm-6"> - <div class="col-md-8 order-1"> - <img src="{{ '/static/images/download/stay-safe.svg'|asseturl }}" class="" alt="{{ _("Stay safe") }}" /> + <div class="col-md-8 order-1 illo-container"> + <img src="{{ '/static/images/download/png/stay-safe(a)3x.png'|asseturl }}" class="" alt="{{ _("Stay safe") }}" /> </div> <h3 class="text-primary">{{ _('Stay safe') }}</h3> <div class="row pl-3 pr-5 text-tpo">

1 0

[tpo/master] Add png for svg fallback /download
by hiro＠torproject.org 16 Apr '19

16 Apr '19

commit e168cfa203d81a25deb40da7a23234380b974f81 Author: Antonela <hola(a)antonela.me> Date: Tue Apr 16 13:24:04 2019 -0300 Add png for svg fallback /download --- assets/static/images/download/png/get-connected.png | Bin 0 -> 15182 bytes assets/static/images/download/png/get-connected(a)2x.png | Bin 0 -> 35598 bytes assets/static/images/download/png/get-connected(a)3x.png | Bin 0 -> 56569 bytes assets/static/images/download/png/stay-safe.png | Bin 0 -> 23398 bytes assets/static/images/download/png/stay-safe(a)2x.png | Bin 0 -> 53831 bytes assets/static/images/download/png/stay-safe(a)3x.png | Bin 0 -> 84316 bytes .../static/images/download/{ => svg}/get-connected.svg | 0 assets/static/images/download/{ => svg}/stay-safe.svg | 0 8 files changed, 0 insertions(+), 0 deletions(-) diff --git a/assets/static/images/download/png/get-connected.png b/assets/static/images/download/png/get-connected.png new file mode 100644 index 0000000..6375bb8 Binary files /dev/null and b/assets/static/images/download/png/get-connected.png differ diff --git a/assets/static/images/download/png/get-connected(a)2x.png b/assets/static/images/download/png/get-connected(a)2x.png new file mode 100644 index 0000000..21f8283 Binary files /dev/null and b/assets/static/images/download/png/get-connected(a)2x.png differ diff --git a/assets/static/images/download/png/get-connected(a)3x.png b/assets/static/images/download/png/get-connected(a)3x.png new file mode 100644 index 0000000..e6f5370 Binary files /dev/null and b/assets/static/images/download/png/get-connected(a)3x.png differ diff --git a/assets/static/images/download/png/stay-safe.png b/assets/static/images/download/png/stay-safe.png new file mode 100644 index 0000000..0c0a224 Binary files /dev/null and b/assets/static/images/download/png/stay-safe.png differ diff --git a/assets/static/images/download/png/stay-safe(a)2x.png b/assets/static/images/download/png/stay-safe(a)2x.png new file mode 100644 index 0000000..f49df98 Binary files /dev/null and b/assets/static/images/download/png/stay-safe(a)2x.png differ diff --git a/assets/static/images/download/png/stay-safe(a)3x.png b/assets/static/images/download/png/stay-safe(a)3x.png new file mode 100644 index 0000000..00e057f Binary files /dev/null and b/assets/static/images/download/png/stay-safe(a)3x.png differ diff --git a/assets/static/images/download/get-connected.svg b/assets/static/images/download/svg/get-connected.svg similarity index 100% rename from assets/static/images/download/get-connected.svg rename to assets/static/images/download/svg/get-connected.svg diff --git a/assets/static/images/download/stay-safe.svg b/assets/static/images/download/svg/stay-safe.svg similarity index 100% rename from assets/static/images/download/stay-safe.svg rename to assets/static/images/download/svg/stay-safe.svg

1 0

[translation/tpo-web] Update translations for tpo-web
by translation＠torproject.org 16 Apr '19

16 Apr '19

commit f5af7ac8db13b8960b0c4b0cfc6743f9607a26c9 Author: Translation commit bot <translation(a)torproject.org> Date: Tue Apr 16 16:20:34 2019 +0000 Update translations for tpo-web --- contents+fr.po | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/contents+fr.po b/contents+fr.po index 2df0eb81f..f92fcd7f2 100644 --- a/contents+fr.po +++ b/contents+fr.po @@ -344,6 +344,11 @@ msgid "" "diverse interests and trust assumptions, and the software needed to be free " "and open to maximize transparency and separation." msgstr "" +"Depuis sa création dans les années 1990, le routage oignon a été conçu pour " +"s'appuyer sur un réseau décentralisé. Ce réseau avait besoin d'être manipulé" +" par des entités aux intérêts divers et des affectations de confiance, et le" +" logiciel se devait d'être gratuit et ouvert pour maximiser sa transparence " +"et sa séparation." #: https//www.torproject.org/about/history/ #: (content/about/history/contents+en.lrpage.body) @@ -351,6 +356,8 @@ msgid "" "That's why in October 2002 when the Tor network was initially deployed, its " "code was released under a free and open software license." msgstr "" +"C'est pour cela qu'en octobre 2002, lors du déploiement initial du réseau " +"Tor, son code fut publié sous une licence logicielle libre et ouverte." #: https//www.torproject.org/about/history/ #: (content/about/history/contents+en.lrpage.body) @@ -358,6 +365,8 @@ msgid "" "By the end of 2003, the network had about a dozen volunteer nodes, mostly in" " the U.S., plus one in Germany." msgstr "" +"Vers la fin 2003, le réseau avait une douzaine de nœuds volontaires, " +"majoritairement aux États-Unis, en plus d'un en Allemagne." #: https//www.torproject.org/about/history/ #: (content/about/history/contents+en.lrpage.body) @@ -375,6 +384,9 @@ msgid "" "address censorship, such as the need to get around government firewalls, in " "order for its users to access the open web." msgstr "" +"En 2007, l'organisation débuta le développement de ponts vers le réseau Tor " +"pour contourner la censure, comme par exemple les pare-feu gouvernementaux, " +"afin de permettre à tous ses utilisateurs d'accéder au Web libre." #: https//www.torproject.org/about/history/ #: (content/about/history/contents+en.lrpage.body)

1 0

[tpo/master] Merge remote-tracking branch 'anto/master'
by hiro＠torproject.org 16 Apr '19

16 Apr '19

commit 4743c462d280eb95fb1950586048e462c74b20f3 Merge: 3050853 fe37f17 Author: hiro <hiro(a)torproject.org> Date: Tue Apr 16 18:19:45 2019 +0200 Merge remote-tracking branch 'anto/master' templates/download.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)

1 0