September 2019 - tor-commits - lists.torproject.org

[tor/master] Use Doctests to test the behavior of annotate_ifdef_directives.
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit 6f0e697e4155ca567ddfd46a7f4e7c013287c42a Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 26 10:03:28 2019 -0400 Use Doctests to test the behavior of annotate_ifdef_directives. --- scripts/maint/annotate_ifdef_directives | 115 ++++++++++++++++++++++++++------ 1 file changed, 94 insertions(+), 21 deletions(-) diff --git a/scripts/maint/annotate_ifdef_directives b/scripts/maint/annotate_ifdef_directives index f88dd4fdf..514b5e58b 100755 --- a/scripts/maint/annotate_ifdef_directives +++ b/scripts/maint/annotate_ifdef_directives @@ -2,24 +2,60 @@ # Copyright (c) 2017-2019, The Tor Project, Inc. # See LICENSE for licensing information -# This script iterates over a list of C files. For each file, it looks at the -# #if/#else C macros, and annotates them with comments explaining what they -# match. -# -# For example, it replaces this: -# -# #ifdef HAVE_OCELOT -# // 500 lines of ocelot code -# #endif -# -# with this: -# -# #ifdef HAVE_OCELOT -# // 500 lines of ocelot code -# #endif /* defined(HAVE_OCELOT) */ -# -# Note that only #else and #endif lines are annotated. Existing comments -# on those lines are removed. +r""" +This script iterates over a list of C files. For each file, it looks at the +#if/#else C macros, and annotates them with comments explaining what they +match. + +For example, it replaces this kind of input... + +>>> INPUT = ''' +... #ifdef HAVE_OCELOT +... C code here +... #if MIMSY == BOROGROVE +... block 1 +... block 1 +... block 1 +... block 1 +... #else +... block 2 +... block 2 +... block 2 +... block 2 +... #endif +... #endif +... ''' + +With this kind of output: +>>> EXPECTED_OUTPUT = ''' +... #ifdef HAVE_OCELOT +... C code here +... #if MIMSY == BOROGROVE +... block 1 +... block 1 +... block 1 +... block 1 +... #else /* !(MIMSY == BOROGROVE) */ +... block 2 +... block 2 +... block 2 +... block 2 +... #endif /* MIMSY == BOROGROVE */ +... #endif /* defined(HAVE_OCELOT) */ +... ''' + +Here's how to use it: +>>> import sys +>>> if sys.version_info.major < 3: from cStringIO import StringIO +>>> if sys.version_info.major >= 3: from io import StringIO + +>>> OUTPUT = StringIO() +>>> translate(StringIO(INPUT), OUTPUT) +>>> assert OUTPUT.getvalue() == EXPECTED_OUTPUT + +Note that only #else and #endif lines are annotated. Existing comments +on those lines are removed. +""" import re @@ -38,6 +74,17 @@ class Problem(Exception): def close_parens_needed(expr): """Return the number of left-parentheses needed to make 'expr' balanced. + + >>> close_parens_needed("1+2") + 0 + >>> close_parens_needed("(1 + 2)") + 0 + >>> close_parens_needed("(1 + 2") + 1 + >>> close_parens_needed("(1 + (2 *") + 2 + >>> close_parens_needed("(1 + (2 * 3) + (4") + 2 """ return expr.count("(") - expr.count(")") @@ -47,6 +94,17 @@ def truncate_expression(expr, new_width): characters long. Try to return an expression with balanced parentheses. + + >>> truncate_expression("1+2+3", 8) + '1+2+3' + >>> truncate_expression("1+2+3+4+5", 8) + '1+2+3...' + >>> truncate_expression("(1+2+3+4)", 8) + '(1+2...)' + >>> truncate_expression("(1+(2+3+4))", 8) + '(1+...)' + >>> truncate_expression("(((((((((", 8) + '((...))' """ if len(expr) <= new_width: # The expression is already short enough. @@ -69,14 +127,23 @@ def truncate_expression(expr, new_width): return ellipsis def commented_line(fmt, argument, maxwidth=LINE_WIDTH): - - """ + # (This is a raw docstring so that our doctests can use \.) + r""" Return fmt%argument, for use as a commented line. If the line would - be longer than maxwidth, truncate argument. + be longer than maxwidth, truncate argument but try to keep its + parentheses balanced. Requires that fmt%"..." will fit into maxwidth characters. Requires that fmt ends with a newline. + + >>> commented_line("/* %s */\n", "hello world", 32) + '/* hello world */\n' + >>> commented_line("/* %s */\n", "hello world", 15) + '/* hello... */\n' + >>> commented_line("#endif /* %s */\n", "((1+2) && defined(FOO))", 32) + '#endif /* ((1+2) && defi...) */\n' + """ assert fmt.endswith("\n") result = fmt % argument @@ -208,6 +275,12 @@ def translate(f_in, f_out): raise Problem("Missing #endif") import sys,os + +if sys.argv[1] == "--self-test": + import doctest + doctest.testmod() + sys.exit(0) + for fn in sys.argv[1:]: with open(fn+"_OUT", 'w') as output_file: translate(open(fn, 'r'), output_file)

1 0

[tor/master] Rename annotate_ifdef_directives to end with .py.
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit f1e0665c934db49cc86936a17a3a1247db9e3337 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 26 15:43:40 2019 -0400 Rename annotate_ifdef_directives to end with .py. This allows the python doctest module to process it correctly when invoked as: python -m doctest -v annotate_ifdef_directives.py --- Makefile.am | 2 +- .../maint/{annotate_ifdef_directives => annotate_ifdef_directives.py} | 0 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile.am b/Makefile.am index 491b4c8f9..e52b1f742 100644 --- a/Makefile.am +++ b/Makefile.am @@ -477,7 +477,7 @@ version: .PHONY: autostyle-ifdefs autostyle-ifdefs: - $(PYTHON) scripts/maint/annotate_ifdef_directives $(OWNED_TOR_C_FILES) + $(PYTHON) scripts/maint/annotate_ifdef_directives.py $(OWNED_TOR_C_FILES) .PHONY: autostyle-ifdefs autostyle-operators: diff --git a/scripts/maint/annotate_ifdef_directives b/scripts/maint/annotate_ifdef_directives.py similarity index 100% rename from scripts/maint/annotate_ifdef_directives rename to scripts/maint/annotate_ifdef_directives.py

1 0

[tor/master] Annotate_ifdef_directives: doctest for 80-column lines.
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit 21c9f7c85e90f4d3ef539d41a36a24b2f26ad3d1 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 26 19:57:41 2019 -0400 Annotate_ifdef_directives: doctest for 80-column lines. --- scripts/maint/annotate_ifdef_directives.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/maint/annotate_ifdef_directives.py b/scripts/maint/annotate_ifdef_directives.py index 6ff9b8ec4..b4326f982 100755 --- a/scripts/maint/annotate_ifdef_directives.py +++ b/scripts/maint/annotate_ifdef_directives.py @@ -144,6 +144,18 @@ def commented_line(fmt, argument, maxwidth=LINE_WIDTH): >>> commented_line("#endif /* %s */\n", "((1+2) && defined(FOO))", 32) '#endif /* ((1+2) && defi...) */\n' + + The default line limit is 80 characters including the newline: + + >>> long_argument = "long " * 100 + >>> long_line = commented_line("#endif /* %s */\n", long_argument) + >>> len(long_line) + 80 + + >>> long_line[:40] + '#endif /* long long long long long long ' + >>> long_line[40:] + 'long long long long long long lon... */\n' """ assert fmt.endswith("\n") result = fmt % argument

1 0

[tor/master] annotate_ifdef_directives: Allow it to be imported as a module.
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit d229399e77f17a8ad19a793fcc7252027c0d3758 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 26 15:36:20 2019 -0400 annotate_ifdef_directives: Allow it to be imported as a module. --- scripts/maint/annotate_ifdef_directives | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/scripts/maint/annotate_ifdef_directives b/scripts/maint/annotate_ifdef_directives index 514b5e58b..6ff9b8ec4 100755 --- a/scripts/maint/annotate_ifdef_directives +++ b/scripts/maint/annotate_ifdef_directives @@ -274,14 +274,16 @@ def translate(f_in, f_out): if len(stack) or cur_level != whole_file: raise Problem("Missing #endif") -import sys,os +if __name__ == '__main__': -if sys.argv[1] == "--self-test": - import doctest - doctest.testmod() - sys.exit(0) + import sys,os -for fn in sys.argv[1:]: - with open(fn+"_OUT", 'w') as output_file: - translate(open(fn, 'r'), output_file) - os.rename(fn+"_OUT", fn) + if sys.argv[1] == "--self-test": + import doctest + doctest.testmod() + sys.exit(0) + + for fn in sys.argv[1:]: + with open(fn+"_OUT", 'w') as output_file: + translate(open(fn, 'r'), output_file) + os.rename(fn+"_OUT", fn)

1 0

[tor/master] Merge branch 'tor-github/pr/1339'
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit 4673cb8168cbacd817c38c3f1ca5c9e88d10b053 Merge: ffd857bbe fc1134e3e Author: George Kadianakis <desnacked(a)riseup.net> Date: Mon Sep 30 13:44:59 2019 +0300 Merge branch 'tor-github/pr/1339' Makefile.am | 2 +- changes/ticket31759 | 5 + scripts/maint/annotate_ifdef_directives | 74 ------- scripts/maint/annotate_ifdef_directives.py | 317 +++++++++++++++++++++++++++++ src/app/config/config.c | 6 +- src/lib/crypt_ops/compat_openssl.h | 2 +- src/lib/fs/dir.c | 2 +- src/lib/log/util_bug.h | 2 +- src/lib/math/fp.c | 2 +- src/lib/memarea/memarea.c | 2 +- src/lib/process/daemon.c | 2 +- src/lib/process/process.c | 2 +- src/lib/process/restrict.c | 2 +- src/lib/process/setuid.c | 2 +- src/lib/tls/tortls_openssl.c | 4 +- 15 files changed, 337 insertions(+), 89 deletions(-)

1 0

[tor/master] annotate_ifdef_directives: test edge-case of 80-char line
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit fc1134e3e59df2ca473e787f70a57ebf659d78f2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Sep 26 20:30:41 2019 -0400 annotate_ifdef_directives: test edge-case of 80-char line An 80-character line (79 characters if you don't count the newline) should not be truncated, and should not have a "..." insterted. --- scripts/maint/annotate_ifdef_directives.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/scripts/maint/annotate_ifdef_directives.py b/scripts/maint/annotate_ifdef_directives.py index b4326f982..102128bfa 100755 --- a/scripts/maint/annotate_ifdef_directives.py +++ b/scripts/maint/annotate_ifdef_directives.py @@ -156,6 +156,22 @@ def commented_line(fmt, argument, maxwidth=LINE_WIDTH): '#endif /* long long long long long long ' >>> long_line[40:] 'long long long long long long lon... */\n' + + If a line works out to being 80 characters naturally, it isn't truncated, + and no ellipsis is added. + + >>> medium_argument = "a"*66 + >>> medium_line = commented_line("#endif /* %s */\n", medium_argument) + >>> len(medium_line) + 80 + >>> "..." in medium_line + False + >>> medium_line[:40] + '#endif /* aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + >>> medium_line[40:] + 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa */\n' + + """ assert fmt.endswith("\n") result = fmt % argument

1 0

[tor/master] Add comments to annotate_ifdef_directives
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit 0b367f3386b0ec25f85716001690c95ae2e78c4d Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 18 10:41:05 2019 -0400 Add comments to annotate_ifdef_directives --- scripts/maint/annotate_ifdef_directives | 40 +++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/scripts/maint/annotate_ifdef_directives b/scripts/maint/annotate_ifdef_directives index ca267a865..15121652d 100755 --- a/scripts/maint/annotate_ifdef_directives +++ b/scripts/maint/annotate_ifdef_directives @@ -2,21 +2,54 @@ # Copyright (c) 2017-2019, The Tor Project, Inc. # See LICENSE for licensing information +# This script iterates over a list of C files. For each file, it looks at the +# #if/#else C macros, and annotates them with comments explaining what they +# match. +# +# For example, it replaces this: +# +# #ifdef HAVE_OCELOT +# // 500 lines of ocelot code +# #endif +# +# with this: +# +# #ifdef HAVE_OCELOT +# // 500 lines of ocelot code +# #endif /* defined(HAVE_OCELOT) */ +# +# Note that only #else and #endif lines are annotated. Existing comments +# on those lines are removed. + import re +# Any block with fewer than this many lines does not need annotations. LINE_OBVIOUSNESS_LIMIT = 4 class Problem(Exception): pass def uncomment(s): + """ + Remove existing trailing comments from an #else or #endif line. + """ s = re.sub(r'//.*','',s) s = re.sub(r'/\*.*','',s) return s.strip() def translate(f_in, f_out): - whole_file = [] + """ + Read a file from f_in, and write its annotated version to f_out. + """ + # A stack listing our current if/else state. Each member of the stack + # is a list of directives. Each directive is a 3-tuple of + # (command, rest, lineno) + # where "command" is one of if/ifdef/ifndef/else/elif, and where + # "rest" is an expression in a format suitable for use with #if, and where + # lineno is the line number where the directive occurred. stack = [] + # the stack element corresponding to the top level of the file. + whole_file = [] cur_level = whole_file lineno = 0 for line in f_in: @@ -24,6 +57,7 @@ def translate(f_in, f_out): m = re.match(r'\s*#\s*(if|ifdef|ifndef|else|endif|elif)\b\s*(.*)', line) if not m: + # no directive, so we can just write it out. f_out.write(line) continue command,rest = m.groups() @@ -43,6 +77,8 @@ def translate(f_in, f_out): cur_level = new_level f_out.write(line) elif command in ("else", "elif"): + # We stay at the same level on the stack. If we have an #else, + # we comment it. if len(cur_level) == 0 or cur_level[-1][0] == 'else': raise Problem("Unexpected #%s on %d"% (command,lineno)) if (len(cur_level) == 1 and command == 'else' and @@ -52,6 +88,7 @@ def translate(f_in, f_out): f_out.write(line) cur_level.append((command, rest, lineno)) else: + # We pop one element on the stack, and comment an endif. assert command == 'endif' if len(stack) == 0: raise Problem("Unmatched #%s on %s"% (command,lineno)) @@ -71,4 +108,3 @@ for fn in sys.argv[1:]: with open(fn+"_OUT", 'w') as output_file: translate(open(fn, 'r'), output_file) os.rename(fn+"_OUT", fn) -

1 0

[tor/master] annotate_ifdef_directives: introduce a function to make commented lines
by asn＠torproject.org 30 Sep '19

30 Sep '19

commit f36e743e5de92cecee81a64d2e78ce5ca3070f98 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 18 10:46:47 2019 -0400 annotate_ifdef_directives: introduce a function to make commented lines No functional change in this commit. --- scripts/maint/annotate_ifdef_directives | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/scripts/maint/annotate_ifdef_directives b/scripts/maint/annotate_ifdef_directives index 15121652d..fcd96aeb3 100755 --- a/scripts/maint/annotate_ifdef_directives +++ b/scripts/maint/annotate_ifdef_directives @@ -29,6 +29,12 @@ LINE_OBVIOUSNESS_LIMIT = 4 class Problem(Exception): pass +def commented_line(fmt, argument): + """ + Return fmt%argument, for use as a commented line. + """ + return fmt % argument + def uncomment(s): """ Remove existing trailing comments from an #else or #endif line. @@ -83,7 +89,8 @@ def translate(f_in, f_out): raise Problem("Unexpected #%s on %d"% (command,lineno)) if (len(cur_level) == 1 and command == 'else' and lineno > cur_level[0][2] + LINE_OBVIOUSNESS_LIMIT): - f_out.write("#else /* !(%s) */\n"%cur_level[0][1]) + f_out.write(commented_line("#else /* !(%s) */\n", + cur_level[0][1])) else: f_out.write(line) cur_level.append((command, rest, lineno)) @@ -96,9 +103,11 @@ def translate(f_in, f_out): f_out.write(line) elif len(cur_level) == 1 or ( len(cur_level) == 2 and cur_level[1][0] == 'else'): - f_out.write("#endif /* %s */\n"%cur_level[0][1]) + f_out.write(commented_line("#endif /* %s */\n", + cur_level[0][1])) else: - f_out.write("#endif /* %s || ... */\n"%cur_level[0][1]) + f_out.write(commented_line("#endif /* %s || ... */\n", + cur_level[0][1])) cur_level = stack.pop() if len(stack) or cur_level != whole_file: raise Problem("Missing #endif")

1 0

[gettor/master] Update check service script
by hiro＠torproject.org 30 Sep '19

30 Sep '19

commit b34700fdd2556fb6106d18d9e2a0bb928aa42df2 Author: hiro <hiro(a)torproject.org> Date: Mon Sep 30 12:33:19 2019 +0200 Update check service script --- scripts/check_service | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/check_service b/scripts/check_service index ed7f6a6..9a66053 100644 --- a/scripts/check_service +++ b/scripts/check_service @@ -66,7 +66,7 @@ def test_email_from_gmail(password): - + mail.close() return WARNING, "No emails from gettor found" except Exception as e: @@ -82,11 +82,11 @@ def send_email_from_gmail(password): """ % (sent_from, ", ".join(sent_to), subject, body) try: - server = smtplib.SMTP_SSL('smtp.gmail.com', 465) - server.ehlo() - server.login(sent_from, password) - server.sendmail(sent_from, sent_to, email_text) - server.close() + mail = smtplib.SMTP_SSL('smtp.gmail.com', 465) + mail.ehlo() + mail.login(sent_from, password) + mail.sendmail(sent_from, sent_to, email_text) + mail.close() except Exception as e: return UNKNOWN, str(e)

1 0

[gettor/master] Update upload files script
by hiro＠torproject.org 30 Sep '19

30 Sep '19

commit 5dccebef3bbaadfdcee9411d344c82c4a4aeea2b Author: hiro <hiro(a)torproject.org> Date: Mon Sep 30 12:31:35 2019 +0200 Update upload files script --- scripts/update_files | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/update_files b/scripts/update_files index 8d33075..7183d50 100755 --- a/scripts/update_files +++ b/scripts/update_files @@ -58,5 +58,5 @@ for f in $(ls); do # Update Google Drive rclone copy $f gdrive:releases # Update Internet Archive - ia upload TorBrowserRelease-${f} $f --remote-name=$f --metadata="title:New Tor Browser release ${f}." --metadata="mediatype:software" --metadata="collection:open_source_software" + ia upload ${f}-TorBrowserRelease $f --remote-name=$f --metadata="title:${f} - Tor Browser release" --metadata="mediatype:software" --metadata="collection:open_source_software" done

1 0