August 2020 - tor-commits - lists.torproject.org

[sbws/maint-1.1] new: Add gitchangelog template
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit 3bc35e6c09ded9f35f5aff15521fe5e49804db31 Author: juga0 <juga(a)riseup.net> Date: Sat Nov 24 14:21:53 2018 +0000 new: Add gitchangelog template --- .gitchangelogrst.tpl | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/.gitchangelogrst.tpl b/.gitchangelogrst.tpl new file mode 100644 index 0000000..46f02a3 --- /dev/null +++ b/.gitchangelogrst.tpl @@ -0,0 +1,34 @@ +% if data["title"]: +${data["title"]} +\n +${"=" * len(data["title"])} + + +% endif +% for version in data["versions"]: +<% +title = "%s (%s)" % (version["tag"], version["date"]) if version["tag"] else opts["unreleased_version_label"] + +nb_sections = len(version["sections"]) +%>${title} +${"-" * len(title)} +% for section in version["sections"]: +% if not (section["label"] == "Other" and nb_sections == 1): + +${section["label"]} +${"~" * len(section["label"])} +% endif +% for commit in section["commits"]: +<% +subject = "%s [%s]" % (commit["subject"], ", ".join(commit["authors"])) +entry = indent('\n'.join(textwrap.wrap(subject)), + first="- ").strip() +%>${entry} + +% if commit["body"]: +${indent(commit["body"])} +% endif +% endfor +% endfor + +% endfor

1 0

[sbws/maint-1.1] fix: scripts: Change Github by Gitlab
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit 58f71fabe181952e372a72123a47d9276efdaa3f Author: juga0 <juga(a)riseup.net> Date: Fri Jul 17 09:38:42 2020 +0000 fix: scripts: Change Github by Gitlab releases can live now in gitlab.tpo, instead of github.com and there is no need to check them since Gitlab is FLOSS and gitlab.tpo is hosted by Tor Project. Also, stop assuming which is the current branch and remote and do not push. Instead guide the maintainer to do it. --- scripts/maint/release.py | 55 +++++++++++++----------------------------------- 1 file changed, 15 insertions(+), 40 deletions(-) diff --git a/scripts/maint/release.py b/scripts/maint/release.py index 01e63e8..ee1c904 100755 --- a/scripts/maint/release.py +++ b/scripts/maint/release.py @@ -13,7 +13,7 @@ It will: commit it and commit changelog 4. Create a version with the tag and sign it 5. Push the commit and tag to the repository -6. Obtain the release tarball from Github +6. Obtain the release tarball 7. Sign the release tarball 8. Modify the program to the next prerelease version and commit it 9. Push the commit @@ -23,10 +23,8 @@ All in sequence and doesn't recover from any previous step. It assumes that: - the program version is in ``__init__.py.__version__`` - gitchangelog and semantic_version are installed -- we are in the master branch -- the remote repository to push is origin - the next prerelease version is the release version + "-dev0" -- the official releases tarballs are in Github (because no access to dist.tpo) +- the official tarball releases are at gitlab.torproject.org - the key to sign the release is only one and is available in the system """ @@ -130,45 +128,22 @@ def main(args): '-m', '"Release version {}."'.format(release_version)]) print("\n5. Push commit and tag") - print("----------------------") - print("\nPush now so that the Github tarball will have the" - " correct tag...") - input("Press enter when you are sure everything is correct.") - subprocess.call(['git', 'push', 'origin', 'master']) - - # While we use Github releases and not dist.tpo - print("\n6. Create release tarball") - print("-------------------------") - - print("\nCreating a release tarball...") - subprocess.call( - "git archive --format=tar.gz --prefix=sbws-{}/ " - "-o v{}.tar.gz v{}" - .format(release_version, release_version, release_version).split(' ') - ) - print("\nCreating tarball hash file...") - fd = open('v{}.tar.gz.sha256'.format(release_version), 'w') - subprocess.call("sha256sum v{}.tar.gz".format(release_version).split(' '), - stdout=fd) - fd.close() - - print("Obtaining Github tarball...") - # This will overwrite local tarball, but that's fine since the hash file - # won't be overwritten. + print("------------------------") + print("\nPush now so that the Gitlab creates the tarball from the new " + " commit and tag, eg:") + print("git push myremote mybranch") + print("git push myremote --tags") + input("Press enter when you are done.") + + print("\n6. Obtain the release tarball") + print("-------------------------------") + print("Obtaining Gitlab tarball...") subprocess.call( - "wget https://github.com/torproject/sbws/archive/v{}.tar.gz " + "wget https://gitlab.torproject.org/tpo/network-health/sbws/-/archive/v{}/sbws-v{… " "-O v{}.tar.gz" - .format(release_version, release_version).split(' ') + .format(release_version, release_version, release_version).split(' ') ) - print("Verifying Github tarball and local one are the same...") - try: - subprocess.check_call("sha256sum -c v{}.tar.gz.sha256" - .format(release_version).split(' ')) - except subprocess.CalledProcessError: - print("Tarballs are not the same") - sys.exit(1) - print("\n7. Create the tarball signature") print("-------------------------------") print("Creating detached signature...") @@ -178,7 +153,7 @@ def main(args): .format(keyid, release_version, release_version) .split(' ')) - print("\nUpload the signature manually to Github.") + print("\nUpload the signature manually to Gitlab.") input("Press enter when done.") print("\nRelease done!!!.")

1 0

[sbws/maint-1.1] fix: scripts: Stop bumping to next prerelease version
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit 25cbdaf1eb9fc94a0e6458a209f370a714b1d895 Author: juga0 <juga(a)riseup.net> Date: Fri Jul 17 12:43:00 2020 +0000 fix: scripts: Stop bumping to next prerelease version since it is now managed automatically by versioneer. Instead, suggest creating a "next" maintainance branch. But stop using `-` and `.` characters in it, to type it faster, since most of the new branches will be based on it. --- scripts/maint/release.py | 34 ++++++++++------------------------ 1 file changed, 10 insertions(+), 24 deletions(-) diff --git a/scripts/maint/release.py b/scripts/maint/release.py index 7433ce8..305f8ce 100755 --- a/scripts/maint/release.py +++ b/scripts/maint/release.py @@ -62,14 +62,6 @@ def obtain_release_version(version): sys.exit(1) -def obtain_next_prerelease_version(release_version): - # Assuming that we are only jumping from release to `-dev0` - next_prerelease_version = semantic_version.Version( - str(release_version.next_patch()) + "-dev0" - ) - return next_prerelease_version - - def main(args): print(__doc__) try: @@ -141,22 +133,16 @@ def main(args): input("Press enter when done.") print("\nRelease done!!!.") - print("\n8. Create next prerelease version") - print("---------------------------------") - next_prerelease_version = obtain_next_prerelease_version(release_version) - print("\nReplacing the version in the program with " - "the next prerelease version...") - replace_version(release_version, next_prerelease_version) - - print("\nCommitting the prerelease version...") - subprocess.call(['git', 'commit', - '-am', '"Bump to {}."'.format(next_prerelease_version)]) - - print("\n9. Push commit") - print("--------------") - input("Press enter when you are sure everything is correct.") - subprocess.call(['git', 'push', 'origin', 'master']) - print("Done!") + print("\n8. Create next prerelease branch") + print("----------------------------------") + print("\nIf this release happens in a maintainance branch, merge the " + "the commit to master and push, eg:" + "git checkout master" + "git merge --no-ff mybranch" + "git push myremote master") + next_branch_version = "maint{}".format(release_version) + print("And create the next prerelease branch, eg:") + "git checkout -b {}".format(next_branch_version) if __name__ == "__main__":

1 0

[sbws/maint-1.1] fix: scripts: Reformat sentence
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit ef2e35a8a4c587d875f41e6b87c62c40ab5b7429 Author: juga0 <juga(a)riseup.net> Date: Fri Jul 17 12:50:19 2020 +0000 fix: scripts: Reformat sentence --- scripts/maint/release.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/maint/release.py b/scripts/maint/release.py index 305f8ce..0c69f86 100755 --- a/scripts/maint/release.py +++ b/scripts/maint/release.py @@ -20,8 +20,8 @@ It will: All in sequence and doesn't recover from any previous step. It assumes that: -- gitchangelog and semantic_version are installed - the program version can be obtained with ``sbws.__version__`` +- ``gitchangelog`` and ``semantic_version`` python packages are installed - the official tarball releases are at gitlab.torproject.org - the key to sign the release is only one and is available in the system

1 0

[sbws/maint-1.1] fix: scripts: Stop changing version in __init__
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit b88eb394bdb9b1473289f8899d557fe6a084a139 Author: juga0 <juga(a)riseup.net> Date: Fri Jul 17 12:29:32 2020 +0000 fix: scripts: Stop changing version in __init__ Since it is now done by `versioneer`. --- scripts/maint/release.py | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/scripts/maint/release.py b/scripts/maint/release.py index ee1c904..fba204e 100755 --- a/scripts/maint/release.py +++ b/scripts/maint/release.py @@ -9,8 +9,7 @@ It will: 1. Ask which version to release 2. Update the changelog automatically with ``gitchangelog`` and leave the option to user to manually edit it -3. Modify the program version to release version, - commit it and commit changelog +3. Commit the changelog 4. Create a version with the tag and sign it 5. Push the commit and tag to the repository 6. Obtain the release tarball @@ -21,7 +20,6 @@ It will: All in sequence and doesn't recover from any previous step. It assumes that: -- the program version is in ``__init__.py.__version__`` - gitchangelog and semantic_version are installed - the next prerelease version is the release version + "-dev0" - the official tarball releases are at gitlab.torproject.org @@ -66,15 +64,6 @@ def obtain_release_version(version): sys.exit(1) -def replace_version(old_version, new_version): - with open(sbws.__file__, 'r+') as f: - text = f.read() - text = re.sub(str(old_version), str(new_version), text) - f.seek(0) - f.write(text) - f.truncate() - - def obtain_next_prerelease_version(release_version): # Assuming that we are only jumping from release to `-dev0` next_prerelease_version = semantic_version.Version( @@ -112,12 +101,9 @@ def main(args): print("\nRemoving the tag...") subprocess.call(['git', 'tag', '-d', 'v{}'.format(release_version)]) - print("\n3. Modify program version") + print("\n3. Commit the changelog") print("--------------------------") - print("\nReplacing __init__.py version with the release version...") - replace_version(current_version, release_version) - - print("\nCommiting __init__.py and CHANGELOG.rst...") + print("\nCommiting CHANGELOG.rst...") subprocess.call(['git', 'commit', '-am', '"Release version {}."'.format(release_version)])

1 0

[sbws/maint-1.1] fix: scripts: Stop releasing from -dev0 version
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit 12001ae613d0a9da6425229d396eee12e622960f Author: juga0 <juga(a)riseup.net> Date: Fri Jul 17 10:43:52 2020 +0000 fix: scripts: Stop releasing from -dev0 version since now sbws version is calculated from last release tag. --- scripts/maint/release.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/scripts/maint/release.py b/scripts/maint/release.py index fba204e..7433ce8 100755 --- a/scripts/maint/release.py +++ b/scripts/maint/release.py @@ -21,7 +21,7 @@ All in sequence and doesn't recover from any previous step. It assumes that: - gitchangelog and semantic_version are installed -- the next prerelease version is the release version + "-dev0" +- the program version can be obtained with ``sbws.__version__`` - the official tarball releases are at gitlab.torproject.org - the key to sign the release is only one and is available in the system @@ -44,8 +44,6 @@ import sbws def obtain_release_version(version): - # Ensure this is a prerelease version (eg 1.0.3-dev0) - assert version.prerelease release_type_dict = { 1: version.next_patch(), 2: version.next_minor(),

1 0

[sbws/maint-1.1] fix: scripts: Clarify the scope of the script
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit 11c8cb66cf683512522899b8298929f3929b2a94 Author: juga0 <juga(a)riseup.net> Date: Fri Jul 17 12:50:49 2020 +0000 fix: scripts: Clarify the scope of the script it should not take more effort than solving self-sbws issues. --- scripts/maint/release.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/scripts/maint/release.py b/scripts/maint/release.py index 0c69f86..b5fdf3e 100755 --- a/scripts/maint/release.py +++ b/scripts/maint/release.py @@ -4,7 +4,14 @@ usage: release.py GPG_KEY_IDENTIFIER -It will: +This is a fast and dirty script to help release new sbws versions. +It does not intent to cover all possible cases and errors because this is out +of the scope of sbws. +There are other tools [*] that help manage release tags, branches and +changelogs, but we should not add more requirements to sbws to just release a +new version. + +This script will: 0. Detect the current program version 1. Ask which version to release 2. Update the changelog automatically with ``gitchangelog`` @@ -17,7 +24,7 @@ It will: 8. Modify the program to the next prerelease version and commit it 9. Push the commit -All in sequence and doesn't recover from any previous step. +All is done in sequence and doesn't recover from any previous step. It assumes that: - the program version can be obtained with ``sbws.__version__`` @@ -25,6 +32,9 @@ It assumes that: - the official tarball releases are at gitlab.torproject.org - the key to sign the release is only one and is available in the system +[*] See for instance https://semantic-release.gitbook.io, + https://github.com/GitTools/GitVersion + """ import re import subprocess

1 0

[sbws/maint-1.1] fix: doc: Start using release script later
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit 20feac8136ddb05d0a170828bbdb38f3724750df Author: juga0 <juga(a)riseup.net> Date: Sat Jul 18 10:22:55 2020 +0000 fix: doc: Start using release script later Change the version from which the release script is used. Also explain the prefixes used in the commits. Closes #29294 --- docs/source/contributing.rst | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst index 3cb4bc1..fb313cc 100644 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@ -135,9 +135,19 @@ Commits Each commit should reference the Tor Project Trac ticket (example: ``#12345``) and possibly the bugfix version. +The commit message should contain ``Closes: #bugnumber``. -From version 1.0.3, the commit title should start by ``new`` or ``fix`` or -``change``. The commit message should contain ``Closes: #bugnumber``. +From version 1.0.2 we started to prefix the summary with the subpackage or +component, though we have not standarized the words to use, eg: ``scanner``, +``generate``, ``v3bwfile``, ``relaylist``, ``doc``, ``test``, ``CI``. + +From version 1.0.3, we also started to prefix the summary with ``new``, +``fix`` or ``chg``, so that `gitchangelog`_ automatically generates different +sections in the CHANGELOG. + +From version 1.1.0 we started to use the words ``new``, ``chg`` and ``fix``, +not in the sense `gitchangelog`_ use them, but to match sematic versioning +changes major, minor and patch. Try to make each commit a logically separate changes.:: @@ -261,7 +271,10 @@ Create a ./CHANGELOG.rst file. Each entry should reference the Tor Project Trac ticket (example: ``#12345``) and possibly the bugfix version. Until version 1.0.2 we have followed `keep a changelog`_ format. -From version 1.0.3, run ``./scripts/maint/release.py`` to create new releases. + +From version 1.1.x, run ``./scripts/maint/release.py`` to create new releases. +It uses `gitchangelog`_ to automatically add new CHANGELOG entries from the +commits' messages. .. _commit-msg: @@ -299,3 +312,4 @@ Example commit message .. _`keep a changelog`: https://keepachangelog.com/en/1.0.0/ .. _`semantic versioning`: https://semver.org/ .. _`vulture`: https://pypi.org/project/vulture/ +.. _`gitchangelog`: https://github.com/vaab/gitchangelog

1 0

[sbws/maint-1.1] Merge remote-tracking branch 'gitlab/merge-requests/17' into maint-1.1
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit b9f92100102cdffb0c4ee289a6b40efb808c20fd Merge: 9ecff4e 3fa7af0 Author: Georg Koppen <gk(a)torproject.org> Date: Sun Aug 9 18:14:02 2020 +0000 Merge remote-tracking branch 'gitlab/merge-requests/17' into maint-1.1 AUTHORS.md | 7 +++- DEPLOY.rst | 14 +++---- INSTALL.rst | 2 +- README.md | 14 +++---- docs/source/contributing.rst | 73 ++++++++++++++++++++++--------------- docs/source/man_sbws.ini.rst | 2 +- docs/source/man_sbws.rst | 2 +- docs/source/tor_bandwidth_files.rst | 4 +- sbws/core/scanner.py | 4 +- sbws/lib/v3bwfile.py | 2 +- 10 files changed, 71 insertions(+), 53 deletions(-) diff --cc docs/source/contributing.rst index fb313cc,b5259c8..0d41689 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@@ -133,21 -145,8 +145,21 @@@ but not all Commits ~~~~~~~~~ - Each commit should reference the Tor Project Trac ticket (example: ``#12345``) + Each commit should reference the Tor Project Gitlab issue (example: ``#12345``) and possibly the bugfix version. +The commit message should contain ``Closes: #bugnumber``. + +From version 1.0.2 we started to prefix the summary with the subpackage or +component, though we have not standarized the words to use, eg: ``scanner``, +``generate``, ``v3bwfile``, ``relaylist``, ``doc``, ``test``, ``CI``. + +From version 1.0.3, we also started to prefix the summary with ``new``, +``fix`` or ``chg``, so that `gitchangelog`_ automatically generates different +sections in the CHANGELOG. + +From version 1.1.0 we started to use the words ``new``, ``chg`` and ``fix``, +not in the sense `gitchangelog`_ use them, but to match sematic versioning +changes major, minor and patch. Try to make each commit a logically separate changes.::

1 0

[sbws/maint-1.1] fix: relaylist: filter out private networks
by juga＠torproject.org 11 Aug '20

11 Aug '20

commit ca20d8287c956e5f8224133e225fb1c8b6c9754d Author: juga0 <juga(a)riseup.net> Date: Sat Aug 1 13:37:20 2020 +0000 fix: relaylist: filter out private networks when checking exit policies to know whether an exit can exit to a port. Closes: #40010 --- sbws/lib/relaylist.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py index ab7455e..dfd8b76 100644 --- a/sbws/lib/relaylist.py +++ b/sbws/lib/relaylist.py @@ -181,6 +181,10 @@ class Relay: """ Returns True if the relay has an exit policy and the policy accepts exiting to the given portself or False otherwise. + + The exits that are IPv6 only or IPv4 but rejecting some public networks + will return false. + On July 2020, there were 67 out of 1095 exits like this. """ assert isinstance(port, int) # if dind't get the descriptor, there isn't exit policy @@ -199,7 +203,12 @@ class Relay: if self.exit_policy: # Using `strict` to ensure it can exit to ALL domains # and ips and that port. See #40006. - return self.exit_policy.can_exit_to(port=port, strict=True) + # Using `strip_private` to ignore reject rules to private + # networks. + return ( + self.exit_policy.strip_private() + .can_exit_to(port=port, strict=True) + ) except TypeError: return False return False

1 0