- tor-project - lists.torproject.org

Analyzing bwauth disagreements
by Tom Ritter 10 Oct '17

10 Oct '17

In which I pretend to know how to do statistics or data analysis; but really hope I spur people to point out what I'm doing wrong =) Methodology: 1) Run two bwauths on the same box 2) Collect votes (and raw bwauth votes where available) for a month from all bwauths (including the duplicate one) 3) Stick the tuple (vote timestamp, relayid, bwauthid, bw value) into a database 4) Measure the percent difference in bandwidth for the same relay at the same vote time between two bwauths as (abs(r1.bw - r2.bw) / ((r1.bw + r2.bw) / 2) 5) Average that value for all relays for a given vote time to get the overall percent difference for that vote time 6) Graph it for multiple bwauth pairs 7) Get a slight flavor difference of the above value by limiting it to relays with a bandwidth value of at least 100, to see if there are any differences Code: Download and archive votes and raw bwauth votes: https://github.com/tomrittervg/bwauth-tools/blob/master/download_files.py cron it up, run it every hour at 15 minutes after the hour. MySQL Database schema: https://github.com/tomrittervg/bwauth-tools/blob/master/schema.sql Process the files and put them in the database: https://github.com/tomrittervg/bwauth-tools/blob/master/files_to_database.py Run the last query to get the data: https://github.com/tomrittervg/bwauth-tools/blob/master/queries.sql Plot it: https://github.com/tomrittervg/bwauth-tools/blob/master/plot_data.py Optionally run something like grep -v "\\\\N,\\\\N,\\\\N,\\\\N,\\\\N,\\\\N,\\\\N,\\\\N" query.csv > non-blank-lines.csv, or change non-blank-lines.csv to query.csv ~ One month of data is 29,194,717 rows or ~4GB of data, and you want to run the query and the plotting overnight. Conclusions: The data!! https://raw.githubusercontent.com/tomrittervg/bwauth-tools/master/data.png x axis: unix epoch of the vote time y axis: percent disagreement between maatuska's bwauth and the indicated bwauth The two bwauths run from maatuska agree quite consistently. However they have the only instance of DISagreement between the 'all' set of relays and the '>100' set. maatuska and moria agree eerily closely. two bwauths agree with each other to the same amount consistently over time, although how much they agree depends on the two bwauths being compared the 'base' agreement, how much two identical bwauths will agree, is around 35% there is not much difference between 'all relays' and 'relays > 100' Follow Up: Is my data correct? maatuska and moria being that close together is suspicious. Compare moria<->longclaw, moria<->faravahar, and longclaw<->faravahar Change the bandwidth limit to >1000 and see if they produces more or less disagreement I can start applying bwauth patches, and run the new bwauth code, from the same vantage point and use this methodology to determine if new code has altered the results. IF it has altered the results: are they better results or worse? Don't know! I should measure the disagreements between bwauths when grouping relays by country. Because I have raw vote data from maatuska and moria, I can test the hypothesis that relays fall between gaps in scanners. Maybe. I could probably get positive proof this occurs but maybe not proof it _doesn't_ occur. -tom

4 8

Damian's Status Report - September 2017
by Damian Johnson 09 Oct '17

09 Oct '17

Hi all, here's what I've been up to this last month... http://blog.atagar.com/september2017/ Cheers! -Damian

1 0

September 2017 report for the metrics team
by Karsten Loesing 06 Oct '17

06 Oct '17

Hello Tor, hello world! Below you'll find the highlights of Tor metrics team work done in September 2017. On behalf of the Tor metrics team, Karsten Deployed a number of improvements for Atlas including fixing a long standing issue with bandwidth sorting in search results by finally considering units [1], showing the first recorded time a relay was seen [2], and displaying DNS names on the details pages. [1] https://trac.torproject.org/projects/tor/ticket/23137 [2] https://trac.torproject.org/projects/tor/ticket/21958 Moved CollecTor's directory listings to the CollecTor page on the Tor Metrics Website [3] in an attempt to provide a single human-facing website for all aspects around Tor Metrics. [3] https://metrics.torproject.org/collector.html Deployed a second back-end server for the Onionoo service [4] to remove yet another single point of failure. [4] https://metrics.torproject.org/onionoo.html Released CollecTor 1.3.0, metrics-lib 2.1.0, and Onionoo 4.1-1.5.0 [5] which all include upgrades to Java 8. [5] https://lists.torproject.org/pipermail/tor-dev/2017-September/012443.html Did some housekeeping in Trac's Metrics/* components [6], reducing the list of open tickets [7] from around 300 to slightly more than 200. [6] https://lists.torproject.org/pipermail/metrics-team/2017-September/000443.h… [7] https://trac.torproject.org/projects/tor/query?status=!closed&component=^Me… Held a 1-day in-person team meeting in Berlin to draft a roadmap for the next 12 months and to do a ticket triage.

1 0

Comments on Donor FAQ page?
by Tommy Collison 06 Oct '17

06 Oct '17

Hello world! I'm reviewing Tor's donor FAQ page [1] and making some copy edits. If anyone has comments or thinks that the FAQ page is missing something, give me a holler. Cryptographically yrs, Tommy [1] https://donate.torproject.org/donor-faq.html

2 1

Notes from October 5 2017 Vegas team meeting
by Karsten Loesing 06 Oct '17

06 Oct '17

Notes for October 5 2017 meeting: Karsten: 1) Wrote a roadmap draft for metrics team work until Q3/2018 that we'd like to share with tor-dev@ on Friday and ask for feedback on missing goals or possibly wrong priorities. If you can, please give it a look and tell us whether that's a bad idea for some reason (or be the first to provide feedback!) 2) What's the current process for adding people to tor-internal@? Conclusion, updated after the meeting: https://gitweb.torproject.org/community/policies.git/tree/membership.txt#n46 Georg: 1) Mozilla All Hands coordination: sent mails to Ethan 2) Continued hiring process for Sponsor8: we are almost done for the browser positions I think Nick: 1) Alpha released: now ahf and isis have the experience of doing it. Planning to add more people as we go along. 2) Trying to manage questions of stability vs bugfixing in 0.3.2. Seem to be doing okay, but risk remains. 3) Going to New England Systems Verification Day at MIT tomorrow; will present some of our verification wishlist. 4) We have found that, although our test coverage is improving as a percentage, our number of uncovered lines is not falling. We need to evaluate strategies here to improve our coverage. 5) Rust is a fun programming language. We'll be talking about Rust stuff at the Montreal meeting a lot. Shari: 1) Getting ready for Montreal meeting 2) Mark Your Calendars: next Tor Meeting dates are March 11-15 in Rome; we're signing contract with hotel this week 3) Any topics to discuss for open or closed board meeting? 4) Was in San Francisco last week; lots of meetings. Isabela: 1) Sent some orientation to network team for roadmap and retrospective sessions at teams meeting day 2) Got all the stuff UX team needs to print for Montreal - sending to Jon today (hopefully is not too late) 3) Finished the designer proposal with Tommy - hopefully we will get it 4) Working with steph and hiro on the newsletter - final reviews and tests 5) Met with Nick and coordinated some stuff including Chile :) I emailed folks there to confirm the presentation will update speakers list when I get more information from Chile people (and copy Nick of course) 6) Excited with all the sessions in Montreal - please reinforce the note taking :) so folks like me can follow up 7) Preparing for including modularization work as part of sponsor8 contract - me, shari and tommy are meeting tomorrow about that Steph: 1) Lots of sharing / promotion: blog posts and releases, upcoming events 2) Checking over final changes with Isa and Hiro to have our newsletter ready 3) Working with GR + comms team on fundraising campaign - have major designs set! 4) Will start to more thoroughly track upcoming events and will try hand at coordinating sessions

1 0

September 2017 Report for the Tor Browser Team
by Georg Koppen 05 Oct '17

05 Oct '17

Hi! In September the Tor Browser team made three releases: Tor Browser 7.0.5[1], 7.0.6[2], and 7.5a5[3]. Tor Browser 7.0.5 fixed an incompatibility of HTTPS-Everywhere with higher security slider levels in Tor Browser.[4] In fact, it made Tor Browser compatible with any WebExtension not just that one. 7.0.6 and 7.5a5 picked up new Tor versions (0.3.1.7 and 0.3.2.1-alpha respectively) and above all the usual Firefox security updates by including Firefox 52.4.0esr. Worth noting for both versions as well is a fix for a crash bug that could kill all tabs if one closed a single tab.[5] Tor Browser 7.5a5 contains for the first time content sandboxing in Firefox enabled on Windows and Linux. Please give it a test as we want to have this feature on the stable channel as soon as we can. Included in this alpha, too, is a patch that will hopefully help our users who run into crashes on Windows by backporting Mozilla code for blocking libraries that are known to interfere with Firefox processes.[6] Apart from the release related work we focused on remaining Sponsor4 items. In particular, we finally switched to our new, rbm-based build system which we used for the first time for official builds when preparing the packages for Tor Browser 7.5a5.[7] We made progress as well working on 64-bit builds for Windows[8][9] and testing UBSan for finding bugs that are the result of undefined behavior[10]. Our improvements to the Tor Launcher UI are starting to emerge, too, thanks to the collaboration with the UX team.[11][12] The full list of tickets closed by the Tor Browser team in September is accessible using the `TorBrowserTeam201709` keyword in our bug tracker.[13] In October we plan to finish our Tor Launcher UI improvements, get our Security Slider update done[14] and will hopefully have some Tor Browser nightly builds for 64bit Windows available. We'll continue testing further Tor Browser hardening techniques like UBSan, STACK[15], and Intel's MPX[16] and start experimenting with improving our bridge distribution in Tor Launcher[17]. All tickets on our radar for this month can be seen with the `TorBrowserTeam201710` keyword in our bug tracker.[18] Georg [1] https://blog.torproject.org/tor-browser-705-released [2] https://blog.torproject.org/tor-browser-706-released [3] https://blog.torproject.org/tor-browser-75a5-released [4] https://trac.torproject.org/projects/tor/ticket/23258 [5] https://trac.torproject.org/projects/tor/ticket/23393 [6] https://trac.torproject.org/projects/tor/ticket/23582 [7] https://trac.torproject.org/projects/tor/ticket/23213 [8] https://trac.torproject.org/projects/tor/ticket/23228 [9] https://trac.torproject.org/projects/tor/ticket/23229 [10] https://trac.torproject.org/projects/tor/ticket/12418 [11] https://trac.torproject.org/projects/tor/ticket/23262 [12] https://trac.torproject.org/projects/tor/ticket/23261 [13] https://trac.torproject.org/projects/tor/query?status=closed&keywords=~TorB… [14] https://trac.torproject.org/projects/tor/ticket/23409 [15] https://trac.torproject.org/projects/tor/ticket/12420 [16] https://trac.torproject.org/projects/tor/ticket/16352 [17] https://trac.torproject.org/projects/tor/ticket/23136 [18] https://trac.torproject.org/projects/tor/query?status=accepted&status=assig…

1 0

Mozilla "All Hands" coordination
by Roger Dingledine 05 Oct '17

05 Oct '17

Hi folks, Some of us have been invited to the upcoming Mozilla All-Hands meeting in Austin in December: https://wiki.mozilla.org/All_Hands/Austin Georg (gk@torproject) is coordinating our plans, to be sure to have enough of the right people there. We think it would be wise to get a good cross-section of the Tor world there -- not just a pile of browser devs. So, if you (a) got an invite, or (b) would like to go as a Tor person, please let Georg know. The deadline for answering their invites is Oct 5, so be sure to do it with enough time before that deadline. (Technically it is not an invite, but rather a nomination for an invite, and then on Oct 5 they'll do some internal process to decide actual invites.) --Roger

3 3

September 2017 Transifex report
by Colin Childs 04 Oct '17

04 Oct '17

Hi everyone, Below is the monthly Transifex report for September, 2017: ### Report Attached are daily, weekly and monthly translation graphs. The Y axis is "source words". 13.02K source words 3,429 collaborators 154 languages 41 Project resources 6 languages at 100% completion (across all 42 project resources): Bulgarian (bg) French (fr) Norwegian Bokmål (nb) Spanish (es) Turkish (tr) -- Colin Childs Tor Project https://www.torproject.org Twitter: @Phoul

1 0

Grants update, Sept 2017
by Tommy Collison 03 Oct '17

03 Oct '17

## Grants - Awarded a $250,000 general operating support grant from MDF. \o/ - Submitted general support grant to the Rose Foundation. - Invited to submit a full proposal to OTF to make usability improvements to Tor Browser. - Invited to submit a full grant proposal to DRL to modularize the Tor network codebase. - Wrote general language about Tor, onion services, and bridges for use in grants, blog posts, fundraising campaigns, song lyrics, wedding vows, etc. etc. ## Other - Coordinated Tor’s participation in the current Outreachy internship round [1]. - Worked on copy for Tor’s end-of-year fundraising campaign, headlined Powering Digital Resistance. - Wrote about upcoming open days and the relay meetup in Montreal [2]. - Wrote about a bug bounty payout and a call to action for technical folks to help test next-gen onion services. - Wrote about new Tor KIST developments and added my own pun to the canon [4]. As a reminder, you can help me gather information on Tor projects that could benefit from grant money by filling out the grant pipeline [5]. (I’m not the person who makes decides what to fund, but it’s good to know what people are working on.) Tommy [1] https://trac.torproject.org/projects/tor/wiki/Outreachy [2] https://blog.torproject.org/all-are-welcome-tors-open-hack-days, https://blog.torproject.org/relay-operators-come-hang-out-montreal [3] https://blog.torproject.org/kist-and-tell-tors-new-traffic-scheduling-featu… [4] https://blog.torproject.org/report-bugs-get-money-like-atechdad, https://blog.torproject.org/we-want-you-test-next-gen-onion-services [5] https://pipeline.torproject.net

1 0

Network Team meeting notes, 2 Oct 2017
by Nick Mathewson 02 Oct '17

02 Oct '17

Hi, all! You can see the logs from this week's network team meeting at http://meetbot.debian.net/tor-meeting/2017/tor-meeting.2017-10-02-16.59.html . Below are the notes we typed into the meeting pad. ============= Tor network team meeting pad, 2 Oct 2017 Note new meeting location: #tor-meeting on OFTC! (See https://lists.torproject.org/pipermail/tor-project/2017-September/001459.ht… for background.) Last week's notes: * https://lists.torproject.org/pipermail/tor-project/2017-September/001457.ht… Discussion topics: Things to do before Montreal: * Teams rotation for October: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/TeamRot… * RSVP for team dinner in montreal! See network-team ML for info. * Learn some Rust! https://github.com/chelseakomlo/rust_kata * Sleep; take care of yourselves; try not to show up burned-out. ;) * RUST HACKATHON STARTS 10:30am on Monday; please arrive by 10:00am! Nick: * Last week * Wrote a bunch of unit tests * Did some research on async io in Rust * Hunted a few bugs (including 8185, 23690); helped with some others * Helped some with upcoming 0.3.2.2-alpha * Reviewed and merged various tickets * Looked at test coverage diffs between 0.3.1 and 0.3.2: we have some problems. Summary: although our coverage rate is up, we have ~500 more uncovered lines than we had before. To find out where they are, have a look at https://people.torproject.org/~nickm/volatile/coverage-diff-summary . For epecific lines, see https://people.torproject.org/~nickm/volatile/coverage-diff.xz . For history: * 0.2.9: 30014 uncovered; 27157 covered: 47.50% covered * 0.3.0: 29520 uncovered; 30288 covered: 50.64% covered * 0.3.1: 30629 uncovered; 34805 covered: 53.19% covered * 0.3.2: 31138 uncovered; 38385 covered: 55.21% covered * This week * Meet with isabela for misc planning * PETS review * Encourage people to defer, fix, fix, defer, etc. * Time permitting, look forward to sponsor8 stuff. * New England Systems Verification Day (Friday) * Montreal prep TBD * Help more with 0.3.2.2-alpha stuff * Prep for 0.3.1.8 as needed? dgoulet: * Last week: - Work on prop224 tickets. We've postponed the IPv6 + Single onion service discussion until Montreal. - Spent some time debugging scheduler KIST issues which turned out to be more deeper issues within tor: #23709, #23710, #23711, #23712 - Worked on ticket from the 032 milestone including review/triage/patches. Nothing big though, most of it was hs/sched related. * This week: - Continue the 032 milestone work. Some prop224 and scheduler stuff needs to be done for the stable. - I want to prep for the IPv6 hackfest in Montreal next week with teor. Many issues in the hidden service subsystem that I want to document properly before that. - Need some prep as well for the circuit building expiring logic for hidden service for which I want/need to sit down with Mike for this in Montreal (mostly related to #3733). teor: (offline, leave this week) Last week: * finish off a bunch of experimental PrivCount features and bug fixes * review Shamir secret sharing proposal * prepare to travel to Montreal (so many bits of paper!) * I think there's some tor coding I forgot, too This week: * week off: rest, casual coding, enjoy Montreal ahf: Last week: Sponsor 4: - Fixed Zstd regression in #23551 Sponsor 8: - Roger found an issue with the HS we used to do our download test with. We fixed this using JavaScript to download the file in the browser and report back the results to the server. Release work: - Meeting with Isis and Nick about what needs to be done before the release. - Reviewed: #23691, #23690 This week: - Roll 0.3.2.2-alpha together with Isis. - Prepare for Montreal meeting. Leaving on Friday (and will by in the air almost the entire day). - Finish Rust toy project. - Update the Sponsor 8 HS to allow the testers to specify their country when doing the testing. - Go over things I haven't gotten around to in the last week: go over old bugs, end of month tasks asn Last week: - Reviewed #23662 and #23603 for prop224. Worked on #23493. - Reviewed #23100. - Worked on #23672. Now ready for merge.. - Did some more debugging on #21969 ("missing primary guard descriptors"). This bug is still alive and a more edge-casey than before so harder to debug. It occured on my service a few days ago, and left it hanging for 28 hours until I restarted it. We also received more reports of this on trac. Needs more hardcore debugging + #23671 + #23670. - Helped tommy with the upcoming blog post calling for prop224 testing. - Revamped the prop224 wiki page some more: https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions This week: - More prop224 testing/bugfixing/planning. - Prepare for Montreal meeting. - More reviews as needed. catalyst: last week (2017-W39): - bug triage - reviewed some patches (including #23643 -- Type-check struct members that are passed to confparse) - more #20532 investigation - Tor Launcher progress bar meeting - fixed a leak in tests (#23691) this week (2017-W40): - somewhat slowed down by respiratory virus - Montreal travel prep - patches for #20532 - more bootstrap progress reporting improvements komlo: last week: - Submitted #22840 (review for this would be great before Montreal) this week: - prep for rust hackfest pastly: last week: more kist bug wrangling with dgoulet isabela: Last week: - Worked on progress bar error msgs and synced with catalyst+mcs https://docs.google.com/document/d/1jZNFYXzii7A6gCTJtvpO-PAO-WNZa2RuEEXYJwx… - Got tests for our network speed done in a some of our target countries, hope to have 100% coverage for october tests (was important to have measurements for september tho) This week: - organize prep work for team roadmap session in montreal - get devs survey done in form format and send it to developers: https://storm.torproject.org/shared/_OX6xP4qVbt5BObbZ5EhCSWOh7snk3O_5iqoaes… isis: last week: - preparation for releasing 0.3.2.2-alpha - more work on the captcha server to fix some bugs found while writing tests #15697 - started on moat backend #22871 - organising meeting sessions this week: - finish #22871 - go over komlo's rust protover patches on #22840

1 0