commit 9e165ce63acf8edc805736dad963528c24033ef7 Author: juga0 <juga@riseup.net> Date: Wed Apr 14 08:35:17 2021 +0000 Prepare release version 1.2.0. Update changelog, updated automatically, manually fixed typos and and new lines. Closes #40004. --- CHANGELOG.rst | 607 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 607 insertions(+) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index bc55b15..7969a47 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -7,6 +7,613 @@ The format is based on `Keep a Changelog <http://keepachangelog.com/en/1.0.0/>`__ and this project adheres to `Semantic Versioning <http://semver.org/spec/v2.0.0.html>`__. +1.2.0 +----- + +New +~~~ +- Docs: Include script on how to release. +- Scripts: Add script to help new releases. +- Add gitchangelog template. +- Add gitchangelog configuration file. +- Docs: Add bwauths list image. +- Relaylist: Keep relays not in last consensus. + Keep the relays that are not in the the last consensus, but are not + "old" yet. +- Util: Add function to know if timestamp is old. + Part of #30727. + +Changes +~~~~~~~ +- Stem: Set default torrc options. + when connecting to an external tor and they are not already set. +- Generate, cleanup: Use 28 days of measurements. + When generating the Bandwidth File as Torflow, use 28 days of past raw + measurements instead of 5, by default. + Also keep the raw measurements for that long before compressing or + deleting them. + And stop checking whether the compression and delete + periods are valid, without checking defaults first and based on + arbitrary values. +- Stem: Add function to connect or start tor. + Move initialization via existing socket to this new function and start + tor only when it fails. +- Stem, scanner: Change args initializing controller. + to check whether the external control port configuration is set. + There is no need to assert all argument options nor to return the error. +- Config: Add option to connect to external tor. + via control port. +- Circuitbuilder: Remove not used attributes. + and make argument optional. +- Circuitbuilder: Simplify building circuit. + Since sbws is only building 2 hop paths, there is no need to add random + relays to the path, or convert back and forth between fingerprint and + ``Relay`` objects. + This will eliminate the circuit errors: + - Tor seems to no longer think XXX is a relay + - Can not build a circuit, no path. + - Can not build a circuit with the current relays. + If a relay is not longer running when attempting to build the circuit, + it will probably fail with one of the other circuit errors: TIMEOUT, + DESTROYED or CHANNEL_CLOSED. +- Scanner: Stop storing recent_measurement_attempt. + because it stores a timestamp for each attempt, which makes state.dat + grow thousand of lines (json). + Closes #40023, #40020 +- V3bwfile: Exclude relays without observed bw. + and without consensus bw from scaling. + Part of #33871, closes #33831. +- V3bwfile: Percentage difference with consensus. +- V3bwfile: Calculate hlimit from scaled sum bw. + instead of bw before scaling. + Tests have finally correct value. + For 1 result, only when the cap is 1, the value will be equal to the + rounded bw because the cap does not limit it. +- V3bwfile: Obtain consensus values from last consensus. +- V3bwfile: Round scaled bandwidth after capping. + Make tests pass because the high limit change the expected values, + but the final value still needs to be fixed. +- V3bwfile: Change logic obtaining min bandwidth. + Take either the consenus bandwidth or the descriptor bandwidth if + one of them is missing, do not scale when both are missing and + ignore descriptor average and burst when they are missing. +- V3bwfile: Scale relays missing descriptor bws. + Scale relays without average or observed bandwidth. + Later it will be check what to do if their values are None or 0 +- V3bwfile: Stop making mean minimum 1. +- V3bwfile: Calculate filtered bandwidth. + for each relay, calculate the filtered mean for all relays and + calculate the filtered ratio for each relay. +- Scaling: Add filtered bandwidth function. + to calculate the filtered bandwidth for each relay. +- Bwfile: Test KeyValues in a bandwidth file. + Added: + - library to check whether the KeyValues make sense + - test an example bandwidth file + - a command to check an arbitrary bandwidth file + Finally, doing something with all these KeyValues! + (Quarantine day 7th) +- V3bwfile: Count recent relay's monitoring numbers. + using timestamps class. + Also add one more result to the tests data and change the + test accordingly. +- Tests: Remove `_count` from attr. +- Resultdump: Add missing attrs to errors. +- Resultdump: Remove `_count` from attributes. + Tests wont' pass with this commit, they'll be fixed in the next commits +- Relayprioritizer: Count priorities with timestamps. + in RelayPrioritizer: + + - Rename recent_priority_list_count to recent_priority_list when + there is no counting + - Rename recent_priority_relay_count to recent_priority_relay + when there is no counting + - Use the timestamps class to manage/count priority lists/relays +- Relaylist: Count measurements with timestamps. + in RelayList: + + - Rename recent_measurement_attempt_count to recent_measurement_attempt when + there is no counting + - Use the timestamps class to manage/count measurement attempts +- Relaylist, v3bwfile: Count consensus with timestamps. + in RelayList: + + - Rename consensus_timestamps to recent_consensus + - Rename recent_consensus_count to recent_consensus when there is + no counting + - Use the timestamps class to manage/count consensuses + - Remove method not needed anymore +- V3bwfile: Convert datetime to str. +- Resultdump: Use custom json encoder/decoder. +- State: Encode/decode datetimes. +- Json: Create custom JSON encoder/decoder. + to be able to serialize/deserialize datetime in the state file. +- Timestamps: Add module to manage datetime sequences. +- State: Add method to count list values. + +Fix +~~~ +- Clarify release script dependencies. +- Use rst changelog template. + and put in the same entry commit subject and body removing new lines. +- Correct network stream and filtered bw. + because Torflow is not using them by relay type. +- V3bwfile: network means without relay type. + This reverts commit fc3d3b992ada601a6255f8a6889179abd4b7e55e and partially + reverts a82c26184097bea3ca405ae19773de7c4354a541. + It was a mistake to think torflow was using the means by relay type, + it actually sets the same networks means for all relay types. + Closes #40080. +- Semi-automatic correction of typos. + Closes #33599. +- Tests: Add codespell configuration. +- Tests: Additional security tests. +- CI: Use all tox environments for python 3.8. +- 2nd round of automatic format. + black insists to keep one long line and flake complain, therefore make + flake to ignore it. +- Flake8 errors. +- Reorder imports with isort. +- Reformat all with black. +- Move to declarative setup.cfg. + Also: + - Update versioneer + - And include other source distribution files in MANIFEST.in + - Add project URLs + - Add formatter and linter dependencies and configurations. + - tox: Remove travis, fix python environments + - tox: Remove extra coverage options and add them in .coveragerc. +- Indent by default to 2 except python files. + also uncomment final newline. Can be commented again in case it fails +- V3bwfile: network means by relay type. + Calculate network stream and filtered bandwidth averages per relay + type, to obtain bandwidth weights the same way as Torflow. + Closes #40059. +- Scaling: Return mean if no bw >= mean. +- Scaling: Stop returning 1 as the means minima. + since they are used as the numerator when calculating the ratio and the + rounding already returns a minimum of 1. +- Scaling: Return if there are no measurements. + it should not be the case because the measurements come from + successful results, but otherwise it'd throw an exception. +- Tests: Add bw filtered from results. +- Scaling: round bandwidth filtered. + because Torflow does it. +- Scanner: Return from measure if no helper. + After refactoring in #40041, it was forgotten to return the error in + the case a helper was not found, what can happen in test networks. + Closes #40065. +- Tests: debug log for tests by default. + and fix test that didn't consider that there might be other logs from + other threads. + Closes #33797. +- Scanner: Log times kept. + not only the times that are not kept. + Closes #40060 +- CI: Temporal workaround for #40072. +- Relalist: Use the consensus timestamp. + to the relay consensus timestamps list, so that it can be + tested it was in a concrete consensus. +- CI: Exit from integration script. + when any of the commands fail. +- CI: Update Python versions. + Closes #40055. +- CI: Update tor versions. +- System physical requirements. + After fixing #40017, the datadir files are compressed after 29 days and + deleted after 57. However the total used disk space is less than 3G, + leaving 3G as precaution. + Closes #40044. +- Scanner: Return from measure if no helper. + After refactoring in #40041, it was forgotten to return the error in + the case a helper was not found, what can happen in test networks. + Closes #40065. +- Update differences Torflow/sbws. + Closes #40056 +- Reorganize Torflow aggregation. + - reorganize sections + - add diagrams and links + - add pseudocode + - remove math + - correct statements + So that it's more accurate and easier to understand. +- Docs: Rename section, add diagrams. +- Separate Torflow/sbws differences. + into a new file. +- Add target to call plantuml. + and generate .svg from .puml files. + Do not add to the html target since the generated svg images are + not deterministic and will change every time `plantuml` is call. +- Separate how scanner and generator work. + in different files and link to each other. +- Add missing new lines. +- CI: Make wget quiet. + to avoid many lines of non useful text the CI. +- Scanner: Rm condition assigning helper. +- Scanner: Move as_entry/as_exit into one function. + since they're similar code +- Scanner: remove relay to measure as helper. +- Scanner: log exit policy when stream fails. +- Relaylist: Remove duplicated can exit methods. + After refactoring and making clear when we were using exit(s) that can + exit to all public IPs (and a port) or only some, refactor them + removing the duplicated code and adding the `strict` argument. +- Add relay measure activity diagram. +- Scanner: extract method on circuit error. + At some point all possible errors should be exceptions. +- Scanner: extract method for not helper case. +- Scanner: extract method to create paths. + because `measure_relay` method is too long, confusing and we have had + several bugs in this part of the code. +- Relaylist: Add methods to obtain exits that. + can exit to some IPs. + To use them in the cases it will be more convenient. +- Relaylist: rename exits_not_bad_allowing_port. + see previous commit +- Relaylist: rename is_exit_not_bad_allowing_port. + see previous commit +- Relaylist: rename can_exit_to_port. + to can_exit_to_port_all_ips, because it's using `strict`, which means + that it allows to exit to all IPs. + It seems more convenient to try first with exits that allow to + exit to some IPs and only try a second time if that fails, because + there are more. +- Resultdump: Check that the error has a circuit. + Because if the error is not a circuit error, it does not have that + attribute. +- Tests: Run integration tests with chutney. + and adapt the tests to pass. + \o/ +- Add chutney configuration. + and scripts to run the integration tests with chutney. + It does not replace yet the way integration tests are run. +- Stem: Move torrc option that does not depend on config. + It seems we forgot this option when refactoring in #28738. +- Stem: Remove torrc option that is the default. + to avoid conflict when comparing the options that should be set and the + ones are set, since the SocksPort will be differently in chutney. +- Resultdump: Log if relay was measured as exit. + or entry. + Closes #40048 +- Relaylist: Stop measuring relays not in the consenus. + as this might cause many circuit errors. + They're already added to the generator. + Also adapt the number in test_init_relays. +- Sphinx warnings when creating documentation. + This should give us at least a clean html, text, and man build + experience. + Closes #40036. +- Add forgotten image from consensus health. + It was referenced by 6e6a8f3ba534cbd93b830fe3ffd5ce40abe8e77d. Since that + image was wrong, created a new screenshot from the current "past 90 + days" at consensus-health.tpo. +- Stem: Add possible exception cause. +- Stem: Remove unused code. +- Stem: Exit on failure connecting to control port. + because when trying to connect to an external tor (chutney), it does + not make sense to start own tor. + Also log how the connection has been made. +- Update values in config_tor.rst + clean-up. + Closes #40035. +- Update default values in man_sbws.ini.rst. + Closes #40034. +- Clean up config.rst. + Closes #40033. +- Scanner: Retry to measure exit as exit. + if it fails to be measured as entry. + Mayb closes: #40029. +- Relaylist: Comment on IPv6 exit policy. + that could be also checked, increasing the chances that the exit can + exit to our Web servers. + But if it could not, then we need to retry to measure it as 1st hop. +- Config: Increment circuit build timeout. + setting it to the default, 60secs. + Since many relays fail to be measured cause of circuit timeout. + Maybe closes #40029. +- Bump bandwidth file version to 1.5.0. + after removing KeyValue recent_measurement_attempt_count in #40023. + Changed also torspec, issue #20. +- V3bwfile: Tor version added in bandwidth v1.4.0. + since, by mistake, the bandwidth file version here was never updated + to v1.5.0. + This patch only changes the constants names, but logic remains the same. + Related to torspec#35. +- Add the bwauths timeline wiki. + Closes #40013. +- Add bwauthealth tool. +- Add consensus health page. + about bwauths measured relays. +- Move consensus weight to top. + and explain what to check. +- V3bwfile: Take all measurements when IP changes. + Previously, when a relay changes IP, only the measurements with the + last IP were considered. + Relays with dynamic IP could get unmeasured that way. + Now, all the measurements are considered. +- V3bwfile: Avoid statistics without data. + If mean or median argument is empty, they throw an exception. + This can happen when the scanner has stopped and the result is + stored as successful without any downloads. +- No need to use Travis anymore. +- Clarify branch to use when contributing. +- Maint: Fix linter error after merging #29294. +- Tests: Stop converting boolean key to int. + Conversion only happens when parsing a bandwidth file in the + integration tests. +- Relaylist: filter out private networks. + when checking exit policies to know whether an exit can exit to a port. +- Update authors. +- Replace docs links from Github to Gitlab. +- Update reviewers. +- Replace Github review process to Gitlab. + Replace also Github terminology to Gitlab. +- Replace Trac, ticket by Gitlab, issue. +- Replace links from Trac to Gitlab. +- Start using release script later. + Change the version from which the release script is used. + Also explain the prefixes used in the commits. + Closes #29294 +- Scripts: Clarify the scope of the script. + it should not take more effort than solving self-sbws issues. +- Scripts: Reformat sentence. +- Scripts: Stop bumping to next prerelease version. + since it is now managed automatically by versioneer. + Instead, suggest creating a "next" maintenance branch. + But stop using `-` and `.` characters in it, to type it faster, since + most of the new branches will be based on it. +- Scripts: Stop releasing from -dev0 version. + since now sbws version is calculated from last release tag. +- Scripts: Stop changing version in __init__ + Since it is now done by `versioneer`. +- Scripts: Change Github by Gitlab. + releases can live now in gitlab.tpo, instead of github.com and + there is no need to check them since Gitlab is FLOSS and gitlab.tpo is + hosted by Tor Project. + Also, stop assuming which is the current branch and remote and do not + push. Instead guide the maintainer to do it. +- CI: Add .gitlab-ci.yml to run tests in Gitlab. +- Relaylist: Check exit to all domains/ips. + When an exit policy allows to exit only to some subnet, it is not + enough to check that it can exit to a port, since it can, but it might + not be able to exit to the domain/ip of the sbws Web servers. + To ensure that without having to check whether it can exit to a + specific domain/ip, we can query the exit policy with `strict`. + Closes #40006. Bugfix v1.0.3. +- V3bwfile: Count relay priority lists. + and measurement attempts from all the results. + Until they get properly updated. + Also change dates in tests, so that timestamps are counted correctly +- Recomment maint-1.1 for production. +- Recommend using a CDN, + add link to it and rephrase some sentences. +- Increase RAM required. + ahem, because of all json it has to manage in memory. +- Recommend pip only for development. + or testing and add links. +- Update supported Python versions. +- Comment on Debian/Ubuntu releases. + because sometimes the package might not be in Debian stable or testing + and we are not checking Ubuntu releases. +- Tests: Remove all the `\t` in torrc files. + at the beginning of the line and in empty lines. They are not needed. +- Tests: Create new authority keys. + because they expired. + They will expire again in a year. + Implementing #33150 and using chutney would avoid to update keys. + Closes #34394. +- V3bwfile: linter error with new flake version. +- Add differences between Torflow and sbws. + Closes #33871. +- Update/clarify Torflow aggregation. +- Docs: Remove unneeded linter exception. +- Docs: Move torflow scaling docstring to docs. + so that it has its own page as it is too long as docstring and is + harder to write latex with the docstring syntax. +- Unrelated linter error. +- V3bwfile: Remove unneeded minimum 1. + since rounding already returns 1 as minimum. +- V3bwfile: Use cap argument to clip scaled bw. + Make test pass, though the value is not correct since it needs to be + rounded after clipping +- V3bwfile: cap is never None. +- V3bwfile: Warn about None bandwidth values. + since they are probably due a bug. +- Check that log prints a number. + and not a list of timestamps. +- Assert that caplog messages were found. +- Explain changes in the previous commits. +- Tests: Check the files generated in test net. + Test that the results, state and bandwidth file generated by running + the scanner and the generator in the test network are correct. +- Tests: Add tests loading results. + in ResultDump and incrementing relay's monitoring KeyValues. +- Tests: Add results incrementing relays' + monitoring KeyValues. +- V3bwfile: Stop calculating failures with 0 attempts. +- Relaylist: Count recent relay's monitoring numbers. + using timestamps class. + Additionally: + - fix: relayprioritizer: Replace call relay priority + - fix: scanner: Replace call relay measurement attempt +- State: Let json manage data types. + Since state uses json and json will raise an error when it can't + decode/encode some datatype. +- State: Read file before setting key. + Otherwise, if other instance of state set a key, it's lost by the + current instance. + Bugfix v0.7.0. +- Tests: Test state file consistency. + Test that two different instances of state don't overwrite each other. + This test don't past in this commit, will pass in the next bugfix. + Bugfix v0.7.0, which claimed 100% test coverage on state. +- Tests: linter error cause missing nl. +- Relaylist: Update relay status before consensus. + Update relay status before updating the consensus timestamps + Timestamps that are not old yet were getting removed because the + document.valid_after timestamp was still the one from the previous + consensus. + Closes #33570. +- Tests: Test the number of consensus in Relay. + This test does not pass in this commit, but in the next bugfix. +- Relaylist: Use is_old fn removing consensus. + since the logic is the same and the there were two bugfixes on the + same logic. +- Relaylist: Use seconds removing consensuses. + by default days is passed to timedelta, what was making the oldest + date be thousands of days in the past. + Bugfix 1.1.0. +- Tests: Add relaylist test. + Tests don't pass in this commit, they're fixed in the next commits. +- Tests: Add mocked controller fixture. + to be able to unit test all the code that needs a controller. +- Tests: Add test for remove old consensus ts. + Tests don't pass in this commit, it's fixed in the next commits. +- Timestamp: measurements period is in seconds. + by default days is passed to timedelta, what was making the oldest + date be thousands of days in the past. +- Timestamp: Old timestamps are minor than older. + Old timestamps are minor than the older date, not major. +- Relaylist: Stop passing argument to self.is_old. +- Tests: Add test timestamp.is_old. + The tests don't pass in this commit, it's fixed in the next ones. +- V3bwfile: Reformat to don't get flake8 errors. + Part of #30196 +- V3bwfile: Move keys to correct constant. + Part of #30196. +- V3bwfile: Add comment about bwlines v1.3. + Part of #30196. +- V3bwfile: Add tor_version KeyValue. + + - Create new KeyValues constants for the new v1.5.0 KeyValues + - Instantiate State in Header.from_results so that there is no need + to create new methods for all the header KeyValues that are read + from the state file + - Add tor_version to the kwargs to initialize the Header + - Write tor_version in the state file when the scanner is started +- V3bwfile: Add constant for ordered key/values. + to build the list of all keys from it and ensure no key is missing. +- V3bwfile: Reformat to don't get flake8 errors. + After the automatic constants renaming, fix the flake8 errors by + reformatting automatically with `black`, only the lines that had + errors. + Part of #30196 +- Document why ersioneer to obtain version. +- Add at build time the git revion to version. + Instead of having a hardcoded version, calculate the version at build + time making use of `git describe --tags --dirty --always`. + This way, even if the program is not running from inside a git + repository it still can know which was the git revision from the + source it was installed from. + If the program is launched from a path that is a git repository, it + does not gives the git revision of that other repository. + If's also able to get the version when installed from a tarball. + It does not add the git revision when it's being install from a git + tag. + `versioneer` external program is only needed the first time, because + it copies itself into the repository. So it does not add an external + dependency. + There're no changes needed to the `--version` cli argument nor to the + code that generates the bandwidth file, since they both use the + variable `__version__`. + The version previous to this commit was `1.1.1-dev0`, after + this commit, it becomes `1.1.0+xx.gyyyyyyyy`, ie. xx commits after + `1.1.0` plus the git short hash (yyyyyyyy). +- Tests: Test maximum retry delta in destination. +- Destination: Replace constant name. + to make it consistent with others and shorter. + Part of #33033. +- Destination: Set maximum to retry a destination. + otherwise the time to retry a destination could increase too much, + as it's being multiplied by 2, and a destination would not recover. + Patch submitted by tom. +- Relaylist: linter error after after merge. + Fix linter error after merging #30733 and #30727. +- CI: Cache pip, run tox stats after success. + and do not require sudo. +- CI: Test all supported python versions. + As in chutney and stem: + - Test all supported python versions + - Test all supported tor versions + Differences between chutney, stem and sbws: + - in sbws we run directly, not an script that calls tox + - we're not using chutney for integration tests (yet) and therefore we're not testing it with different networks + - we don't have shellcheck tests + - we don't support osx nor windows +- Relaylist: Update the relays' descriptors. + when fetching new consensuses. + Part of #30733. +- Globals: Fetch descriptors early. + and useless descriptors, so that sbws detect early changes in the relay + descriptors and continue downloading them even when Tor is idle. + +Other +~~~~~ +- Wip: rm me, temporally change release url. + to personal fork, to test the release process +- Fixup! minor: Change info logs to debug or warning. +- Major: Change default log level to info. + also change formatting to show thread. +- Minor: Change log warning to info or debug. + when it contains sensitive information. +- Minor: Change info logs to debug or warning. + when they contain sensitive information, eg. Web server or are too + verbose for the debug level. + Also add log to indicate when the main loop is actually started. +- Revert "fix: stem: Remove torrc option that is the default" + This reverts commit 15da07d6a447d8310354124f6020b4cf74b75488. + Because it's not the default. No additional changes are needed in the + tests. + Closes #40064. +- Minor: scanner: Change logic creating the path. + When the relay is not an exit, instead of choosing exits that can + exit to all IPs, try with exits that can exit to some IPs, since the + relay will be measured again with a different exit in other loop. + When the relay is an exit, instead of ensuring it can exit all IPs, try + using it as exit if it can exit to some IPs. + If it fails connecting to the Web server, then try a 2nd time using it + as entry to avoid that it will fail in all loops if there is only one + Web server, cause it will be used again as an exit. + Also, the helper exits don't need to be able to exit all IPs. When a + helper exit fails to exit (maybe cause it can not exit to the Web + sever IP), it's not a problem cause in a next loop other exit will be + chosen. + This change of logic also solves the bug where non exits were being + used as exits, because we were trying to measure again a relay that + was used as entry, because it could not exit all IPs, which includes + also the non exits. +- Minor: scanner: move checking helper to methods. + `helper` variable is only used to return error, therefore move it to + the methods that create the path and return the error there. + `our_nick` is not useful for the log, since it is always the same, but + not removing it here. +- Vote on the relays with few or close measurements. + to vote on approximately the same numbers of relays as Torflow. + Torflow does not exclude relays with few or close measurements, though + it is possible that because of the way it measures, there are no few + or close measurements. + Closes #34393 +- Doc: fix: Update sbws availabity in OS and links. +- Bug 33009: Require minimum bandwidth for second hop. +- Use freeze_time() in other parts of our tests, too. + When using `_relays_with_flags()` and similar methods it's possible + that tests start to hang without time freezing. See bug 33748 for more + details. We work around this by providing the necessary `freeze_time()` + calls meanwhile. +- Bug 33600: `max_pending_results` is not directly used in `main_loop` +- Fixup! fix: CI: Test all supported python versions. +- Relaylist: stop using the current time when a consensus is downloaded twice. + Instead: + * use the consensus valid-after time, or + * use the supplied timestamp, or + * warn and use the current time. + This should fix the occasional CI failure, when the current time is 1 second + later than the test consensus time. (Or it should warn, and we can fix the + test code.) + Fixes bug 30909; bugfix on 1.1.0. +- V3bwfile: skip relay results when required bandwidths are missing. + Fixes bug 30747; bugfix on 1.1.0. +- Bump to version 1.1.1-dev0. + v1.1.0 (2019-03-27) -------------------