- tor-project - lists.torproject.org

Tor powers digital resistance with help from Mozilla
by Tommy Collison 26 Oct '17

26 Oct '17

Hello world! Today, we’re launching our end-of-year crowdfunding campaign, highlighting the ways that Tor powers digital resistance and protects essential human rights around the world. We’re especially excited about this campaign because our friends at Mozilla are matching donations up to $500,000, meaning that donations go twice as far. More info (and links to donate) are on our blog: https://blog.torproject.org/powering-digital-resistance-help-mozilla Tor’s done amazing work in 2017, and I’m proud of what we’ve achieved. This crowdfunding campaign is an opportunity to start 2018 strong (and to get some awesome swag -- check out https://donate.torproject.org/) So, please donate if you can. If you can’t right now, please spread the word about this campaign. Tell your friends that, because of Mozilla’s matching program, there's never been a better time to donate to our favorite anonymity network. Thanks, everyone! TC

4 6

Re: [tor-project] Analyzing bwauth disagreements
by dawuud 25 Oct '17

25 Oct '17

Hi Nusenu, Tor Project, Tor community, tl;dr The Tor network is *highly* partitioned. The scanner code is located here: https://github.com/david415/tor_partition_scanner HOWEVER, the scanner needs a redesign... I have been using the WRONG methodology for scanning the Tor network for partitions in that I select relays which I think are interesting and then see how much connectivity there is between them. I instead would like to scan ALL the Tor relays (repeatedly at different times of day) and then later perform queries against the consensus to see which relays show up in interesting combinations of failures. The other failure in this approach to scanning is that it uses a fixed set of relays... and therefore the Tor consensus file used will become old and contain relays no longer in the consensus well before the scan is complete. And lastly, don't get gamed! Scanning from a single IP is a mistake and this is an obvious way to get gamed. For all these reasons my colleague Katharina and I plan to write a new better scanner of Tor network partions that is distributed, and that has a central dispatcher so that the new consensus documents will be used to inform "worker machines" which circuits to try and build. All that having been said, let me show you what my naive scan looked like a few weeks ago: 1. setup a machine running Tor and expose its control port as either a tcp port or unix domain socket with no authentication *edit* /etc/tor/torrc blah blah easy rtfm 2. install tor_partition_scanner virtualenv virtenv-orscanner . ./virtenv-orscanner/bin/activate mkdir -p code; cd code git clone https://github.com/david415/tor_partition_scanner.git cd tor_partition_scanner pip install -e . 3. get a recent consensus file Use consensus files from collector if you want others to be able to reproduce your results. here --> https://collector.torproject.org/recent/relay-descriptors/consensuses/ wget https://collector.torproject.org/recent/relay-descriptors/consensuses/2017-… 4. choose which relays you want in your scan Here I am intentionally NOT scanning 50 million tor circuits using the entire consensus. Instead I am using a simple python program written using the Stem library to parse the consensus file and give us all the realys with the Stable and Fast flags; among those we choose the top 100 in terms of consensus bandwidth. ./helpers/query_fingerprints_from_consensus_file.py 2017-09-21-23-00-00-consensus > top100.relays 5. perform scan of top 100 relays detect_partitions.py --tor-control tcp:127.0.0.1:9051 --log-dir ./ --status-log ./status_log \ --relay-list top100.relays --secret secretTorEmpireOfRelays --partitions 1 --this-partition 0 \ --build-duration .25 --circuit-timeout 60 --log-chunk-size 1000 --max-concurrency 100 9,900 two hop tor circuits are being built. As the scan runs you can tail -f the status_log to make sure its working. Only circuit build failures if any will be logged in the json log file. When the scan completes the status_log should display something like this: 2017-09-22T00:05:44+0000 [-] $BD4C647508162F59CB44E4DFC1C2B2B8A9387CCA -> $DD808ECE4F2E24F377CBE11E335ECDA196FE3B78 2017-09-22T00:05:44+0000 [-] $0966A24977A0B0DB62546C6F18F9578D97FE86F0 -> $AD00FB62A133F91009AD5F6503E5F21F594BC4C6 2017-09-22T00:05:50+0000 [orscanner#info] Finished writing measurement values to ./2017-09-22T00:05:50.492698-scan.json. 2017-09-22T00:05:50+0000 [-] Main loop terminated. 6. Load circuit build failures into sqlite db file ./bin/load.py --dbfile scan1.db -p 2017-09-22T00:03:31.610096-scan.json \ -p 2017-09-22T00:05:42.886622-scan.json \ -p 2017-09-22T00:05:50.492698-scan.json 7. Count the results echo "select first_hop, second_hop from scan_log;" | sqlite3 scan1.db | wc -l 2014 8. Attempt to eliminate false positives by retesting the failed circuits mkdir scan1 mv *.json scan1 echo "select first_hop, second_hop from scan_log;" | sqlite3 scan1.db > scan2.circuits detect_partitions.py --tor-control tcp:127.0.0.1:9051 --log-dir ./ --status-log ./status_log \ --relay-list relays_for_scan1 --build-duration .25 --circuit-timeout 60 --log-chunk-size 1000 \ --max-concurrency 100 --circuit-file scan2.circuits ./bin/load.py --dbfile scan2.db -p 2017-09-22T00:59:31.017246-scan.json -p 2017-09-22T01:04:35.491908-scan.json echo "select first_hop, second_hop from scan_log;" | sqlite3 scan2.db | wc -l 1947 still 1947 circuit build failures! Now tell the database to show us the relays involved in a circuit build timeout AND count the number of failures by first hop: sqlite> select first_hop,count(first_hop) from scan_log where status = 'timeout' group by second_hop; $4198BD138E5E11B15B05C826B427148CED7D99FE|2 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $7C0AA4E3B73E407E9F5FEB1912F8BE26D8AA124D|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $A6B0521C4C1FB91FB66398AAD523AD773E82E77E|18 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|2 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $DAA3D3F6FDA962885072537E3F315086B003A6E3|3 $B5212DB685A2A0FCFBAE425738E478D12361710D|1 $0593F5255316748247EBA76353A3A61F62224903|4 $7C0AA4E3B73E407E9F5FEB1912F8BE26D8AA124D|2 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $C818C0EA0BAD90F5432DBA4EE662BCBEC39D2668|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $B5212DB685A2A0FCFBAE425738E478D12361710D|1 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 sqlite> A6B0521C4C1FB91FB66398AAD523AD773E82E77E is certainly an outlier! It looks like this relay hits it's bandwidth limit several times per day... and Roger and I suspect it is failing circuit builds at certain times because it is overloaded during specific times of day due to traffic spikes. The fix of course would be to adjust the tor consensus and perhaps cap this relay's bandwidth capacity in the consensus: https://atlas.torproject.org/#details/A6B0521C4C1FB91FB66398AAD523AD773E82E… Circuit build failures also have outliers where a few relays fail more than others: sqlite> select first_hop,count(first_hop) from scan_log where status = 'failure' group by second_hop; $1AF72E8906E6C49481A791A6F8F84F8DFEBBB2BA|10 $1AF72E8906E6C49481A791A6F8F84F8DFEBBB2BA|10 $A571351082A9E04F14A0A3DF27E0637231D57B84|10 $B204DE75B37064EF6A4C6BAF955C5724578D0B32|11 $F6740DEABFD5F62612FA025A5079EA72846B1F67|10 $BF0FB582E37F738CD33C3651125F2772705BB8E8|10 $A571351082A9E04F14A0A3DF27E0637231D57B84|10 $1D3F937E2053E58C18E18D43FA5153E2A9F4DC77|99 This outlier relay 1D3F937E2053E58C18E18D43FA5153E2A9F4DC77 is also quite interesting. It also seems to hit its bandwidth limit several times per day and it's also interesting to note that it's located in the UK: https://atlas.torproject.org/#details/1D3F937E2053E58C18E18D43FA5153E2A9F4D… Show me all the tor relays that failed ALL of their circuit builds: echo "select first_hop,count(first_hop) from scan_log where status = 'failure' group by second_hop;" | sqlite3 scan2.db | grep 99 $1D3F937E2053E58C18E18D43FA5153E2A9F4DC77|99 $C793AB88565DDD3C9E4C6F15CCB9D8C7EF964CE9|99 $EF6591754F9079DD122EFC2C4B52917F625A8E5B|99 $55C7554AFCEC1062DCBAC93E67B2E03C6F330EFC|99 $87C08DDFD32C62F3C56D371F9774D27BFDBB807B|99 $BD4C647508162F59CB44E4DFC1C2B2B8A9387CCA|99 $EF6591754F9079DD122EFC2C4B52917F625A8E5B|99 $1D3174338A1131A53E098443E76E1103CDED00DC|99 $734EDDC2C04B1C0184178167ABD23AE85413212F|99 $7C0AA4E3B73E407E9F5FEB1912F8BE26D8AA124D|99 99 is used because here we scanned the top 100 and therefore each relay is involved in 99 circuit builds since relay A cannot build a circuit to itself. I hope it rings loud and clear that this is a *huge* problem for the health of the Tor network that many of the top 100 Tor relays with the Fast and Stable flags cannot even build a single Tor circuit to any of the other top 100 relays! In conclusion, many of these circuit build failures are very likely NOT the fault of the relay operators but instead this points to the failure of the current Tor Bandwidth Authority system. Not only is it old and broken, even if it was to work "properly" it would still be broken by design if: a. it's not performing circuit build tests b. it's not distributed and thus more easily gameable Katharina and I plan to conduct a more formal research project in this area and scan the entire Tor network (50 million 2-hop Tor circuits) several times at some point in the near future using a new better methodology. However even before we do that, I can tell you right now you will not like the results. It's bad news. (but fret not, Tor Project is committed to fixing the Bandwidth Authority system) And yes, I am well aware that correlating these scan results with BGP ASNs is interesting. There are many other interesting queries we can do after we've collected the data to try and find malicious/intentional network partitions. One possible positive outcome of our research could perhaps be the addition of a partition scanning component to the Bandwidth Authority System. However, as it stands right now Tor Project is prioritizing bandwidth measurements NOT partition scanning, since that is currently a maintenance pain point for the Directory Authority operators that must deal with the torflow tool. Sincerely, David Stainton On Wed, Oct 11, 2017 at 09:55:00PM +0000, nusenu wrote: > Hi David, > > dawuud: > > Also I have recently done a few small scans for 2-hop circuit > > connectivity and shared some of the results with Roger on > > #tor-dev. One theory is that many of the circuit failures are due to > > traffic spikes at certain hours of the day. > > where can I find more about that if I havn't been on #tor-dev at the time? > thanks, > nusenu

3 5

Booth during the FOSDEM 2018
by syl 25 Oct '17

25 Oct '17

Hello it's a bit late (deadline for booth proposals is 3 November) but I want to propose a booth for the next FOSDEM (if we are enough). Last year, we have a common booth with 4 organisations (Tor, Tails, Frënn vun der Ënn and Nos oignons) and 11 persons. Someone are interested to participate this year? syl (from Nos oignons)

6 5

Monthly Community Council Update (23 Oct 2017)
by Linda Naeun Lee 23 Oct '17

23 Oct '17

Hello Tor! The community council had its first meeting after Montreal on Monday 10/23. We met to discuss the discussion of the case handling policy during the parallel sessions and the work done on the code of conduct during the open hack days. We plan on working on these two policies in the following month, meeting weekly. Thanks, The Community Council

1 0

Network team meeting notes, 23 Oct 2017.
by Nick Mathewson 23 Oct '17

23 Oct '17

Hi! You can see logs of the weekly network team meeting at http://meetbot.debian.net/tor-meeting/2017/tor-meeting.2017-10-23-16.59.html ======================= Network team meeting pad, 23 October 2017 Note the meeting location: #tor-meeting on OFTC! (See https://lists.torproject.org/pipermail/tor-project/2017-September/001459.ht… for background.) The last time we met, it was in Montreal. See the various session notes at: * https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Montreal/Not… Announcements: * Remember, this meeting is 1700 UTC, no matter what local time is. That's about to get confusing again. * master is now 0.3.3; use the maint-0.3.2 branch for 0.3.2. * review-group-24 exists! * Team rotation: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/TeamRot… Discussion topics: * What do we do with our roadmap? [conclusion: continue to update, integrate/combine with other teams in early Nov, per Isabela's roadmap roadmap.] * What was that other review-group we wanted? * Cut down rotations? [conclusion: let's survive with ticket triage, coverity, and jenkins] Updates: (Please use boldface to indicate discussion topics) Nick: Last week: - Went to an internews PT meeting in Boston. Argued (successfully, I hope) for issue tracker & for focus on IPC API stability. Worked on threat modeling some. - Hacked on some children of #23684 ("Make it easier for mobile app developers to embed tor") - Took a couple of days off to destress, and make up for having worked over the weekend at the meeting. This week: - Open review-group-24 keyword. ✓ - Triage bugs in stable versions; make sure people are aware of must-fix things. - Meeting with Isa on wed/thu. - Merging patches on 0.3.2, 0.3.1, and maybe 0.3.3. - Backporting things for older versions. - Should I do releases? [dgoulet: I would maybe wait for a new 032 alpha until we have the big missing MD bug resolved #21969?] [asn: wrt md/guard stuff: I think by the end of this week we will have branches for #23862 and maybe other child tickets of #21969.] [ahf: do we have a release out of 0.3.1 with the fixes for the compression non-fatal assertion issue, yet? If you do a release maybe outsource it to others again?] - Working on October items in roadmap: measurements with ahf (?), examining dirauth authentication behavior [for #20532]. - Isabela: Last week: - met with tommy and shari to start working on modularization proposal - start a inter-teams roadmap sharing process (proposed to Vegas Leads last week: https://storm.torproject.org/shared/e0u8jYHQA_6I733RnO68uM6YroskSc21xXWFQ8A… This week: - working on organizing the modularization proposal work in different docs: timeline, logic model, M&E (measurement and evaluation) etc. - pinging people to do network speed test for October - check on the first deadline of inter-teams roadmap sharing - question to networkers: what is the difference between tbb-wants and tbb-needs and similar tags at the team roadmap? [nickm:there is not currently a consistent difference.] - question to networkers: October ends soon - how are we doing with the items we added on the roadmap? should we move things around? Also, november has a lot of things without ownership, should we move them too? https://docs.google.com/spreadsheets/d/1Ufrun1khEo5Cwd6OwngERn829wU3W3eskdr… [nickm: october looks okay; we added ownership for most nov things.] - request to networkers: if you are working on a ticket that is associated with some NSF sponsor work (M, V, Q, 2, 3) please add the right sponsor code to your ticket. We are trying to keep a better track of NSF work :) thanks! [asn: Moved guard ticket #21969 to SponsorV from SponsorU. I hope that's alright. I will move more guard tickets to SponsorV this week.] [dgoulet: All scheduler and KIST related tickets (keyword: tor-sched) have been tagged for SponsorV as discussed in Montreal with Roger. Let me know if they should be elsewhere.] [nickm: I went over everything in 0.3.1 through 0.3.3 that didn't have a sponsor earlier.] - meeting Nick on Wed/Thur komlo (offline): This week: - Submit updates for #22840 based on code review at tor dev (ideally this will be done tomorrow) - Look at next Rust tickets and create any new ones based on conversations at tor dev. asn: [Might be offline during meeting. Depends on dinner timing.] Last week: - Montreal meeting happened. - Some post-meeting TODO organization. This week: - More post-meeting TODO organization. As always after a meeting, I got tons of stuff on my plate that I need to figure out how to handle. - Work on the microdescriptor issues, which is my top priority wrt technical tasks right now (#21969 and children tickets) - Get hackerone under control. - Some non-technical onion service things coming up: Start thinking about press strategy for onion services, need better tools to setup onion services. - Got some important personal stuff on Thursday and Friday this week. dgoulet: Last week: - Montreal meeting - Went over the prop224 open tickets with asn. - Recovered. This week: - Address dgoulet assigned 032 tickets (currently 16 of them...) - If time allows it, I'll try to plan out the changes discussed with pastly that needs to happen in the scheduler for 033. It won't be simple changes so early merge would be ideal for better stabilization. ahf: Last week: Misc: - Montreal meeting + recovery. - Used some of the recovery time to get more into Shadow and Rust. This week: Sponsor 8: - Create bugs based on tasks we decided to do in our roadmapping in Montreal. - Talk with Nick and Isa about "new" mobile(maybe?) tasks that was found in the mobile sessions in montreal (OONI + Nathan+Mike's sessions). - Finish baseline measurement for initial s8 tasks. Misc: - Go over small todo tasks from Montreal (mostly around some of the tasks found in different notes). catalyst: last week (2017-W42): - Montreal meeting - recovery - CoC - GutHub mirroring research this week (2017-W43): - look at hackerone - continue #20532 work - write up more stuff about GitLab and GitHub mirroring and CI for developers isis: last week: - montréal and travelling home - spec patch and patch for #18329 (making it so that bridges can request how to be distributed, also mark their descriptors as bridge descriptors) - register for mozilla all-hands in december and begin planning for it - took time off to recooperate and de-stress this week: - make more tickets for things in the roadmap - finish moat server (#22871) - type up more meeting notes

1 0

Notes from October 19 2017 Vegas team meeting
by Karsten Loesing 23 Oct '17

23 Oct '17

Notes for October 19 2017 meeting: Georg: 1) Preparing Tor Browser 7.0.7 and 7.5a6 with the updated donation campaign code 2) Working on getting the roadmap updated with all the team dependencies Nick: 0) What a great meeting! 1) Internews PT meeting on monday/tuesday seemed to go okay; I think they're likely to adopt practices resulting in convergence on a stable PT 2.x design. 2) Probably not here today; after 9 days of meetings, I need a break. 3) Network team did a draft roadmap. We included items that depend on, or that will affect, other teams. Arturo: 1) Published a report on censorship in Pakistan in collaboration with Bytes for All: https://ooni.torproject.org/post/pakistan-internet-censorship/ 2) We are migrating our email from runbox to GSuite 3) Updated 4 tests lists: https://github.com/citizenlab/test-lists/pull/241, https://github.com/citizenlab/test-lists/pull/242 4) We are going to publish tomorrow a blog post about the OONI team gathering 5) Started experimenting with gatsby and contentful for revamping our website: https://github.com/openobservatory/ooni-website/tree/gatsby, found a bunch of bugs in gatsby, may not end up using it. 6) Created a repository containing licenses for OONI software: https://github.com/openobservatory/legal. Does anybody know some good friendly lawyers we could reach out to regarding the licensing of data? 7) Began adding support for the new OONI API to OONI Explorer: https://github.com/TheTorProject/ooni-explorer/pull/119 8) Following discussion with metrics and TBB teams we started adding support for extracting tor related metrics inside of ooni-pipeline: https://github.com/TheTorProject/ooni-pipeline/pull/98 9) My interview for an Italian TV program is available here: https://www.youtube.com/watch?v=EDZNqYiiYfo. I also explain what Tor is and how it works. (retweet if you wish: https://twitter.com/OpenObservatory/status/920941902408626178) 10) Making some progress on OONI Probe 3.0: https://github.com/OpenObservatory/ooniprobe-cli 11) The deployment process of our website has levelled up. When a branch is merged into master, travis will automatically build it and push it to staging. Deploying it is then just a matter of running a simple script (this could be improved) isabela: 1) followed via jitsi some of the workshops and sessions of Montreal meeting 2) worked on responding OTF design proposal (deadline oct 24th) - getting sponsor8 stuff updated and adding modularization proposal to it (deadline next friday 27th) 3) Organizing inter-team roadmap review / dependencies check: https://storm.torproject.org/shared/85fNa4NXH0lxi1SRMrDNTaDQCyJIuNgHiSgfsiz… Need to know if team leads here agrees with it. [Geko: sounds good to me, Arturo: sgtm, karsten: Oct 23 is reaaaally soon] [alison: I also would like a little more time but overall I think it's a good plan] 4) adam is getting our sponsor4 amendment for a NCE changing the deadline till end of January 5) all day meeting with Shari today here in NYC! Karsten: 1) Recovered from Montreal, worked off some tickets from my inbox, reviewed quite some code, ignored the rest. Shari: 1) talking with fellow travelers about safety in our community 2) working on modularization proposal with Isa and Tommy 3) working on end-of-year blog post and other eoy stuff 4) Tor meeting feedback? Steph: 1) working w hiro to finalize newsletter portal and to make some design changes to the blog 2) working on fundraising campaign, launching monday 3) tidying up comms retrospective, adding to 2018 strategy, incl notes from montreal. will be working in seattle in november with shari and tommy to finalize 4) tommy is taking on compiling positive onion services use cases 5) IFF submissions are due Oct 31 Brad: 1) Working on several follow-up items from Montreal this week 2) Submitted paperwork necessary to open brokerage account for accepting stock donations 3) Held discussions with employees in Montreal about updated time budgets, and have been rolling out the new budgets this week. 4) Compliance reports for Sponsor 8 due 10/31, so please submit Sept. time sheets asap if you haven't already done so. Alison 1) I hope everyone had a good meeting! I did. I was really pleased to see how many people reported feeling happier/healthier at this meeting. 2) I hope to get a post-meeting survey out soon (need to check in with Jon and gunner) 3) September 2018 meeting -- Brazil? 4) The implicit bias session was great. We should do more plenaries like these. 5) We should also do a roadmapping-share session with all teams. 6) Can we talk about how possible it would be to organize childcare at future meetings?

9 9

Hosting opportunity for open source projects
by Nima Fatemi 17 Oct '17

17 Oct '17

Hey there, At GSoC the conversation about hardware needs of open source projects came up and someone mentioned "Open Source Lab"[1] has managed and unmanaged hosting services. I'm not sure if this has come up before in conversations before but judging from their 'hosting details'[2] page, I'm wondering if we can take advantage of their services to save some money off hosting or make the life of our students a bit easier. Just wanted to make sure that people know this exists as it seems to be a useful thing. I hope you're having a good time at the dev meeting. [1] https://osuosl.org/ [2] https://osuosl.org/services/hosting/details/ -- Nima 0X58C4B928A3E218F6 | @mrphs "I disapprove of what you say, but I will defend to the death your right to say it" --Evelyn Beatrice Hall

4 4

Cease and Desist: Copyright Infringement.
by Yawning Angel 12 Oct '17

12 Oct '17

Dear Mr. Nathan Freitas, It has come to my attention that you have made an unauthorized use of my copyrighted work entitled obfs4proxy (the "Work") in the preparation of a work derived therefrom. I have reserved all rights in the Work, which was first published on 2014. Your work entitled Orbot, which appears as a binary distribution available on the Google Play store and F-Droid contains a copy of the Work, without including copies of the Work's copyright notice, conditions, and disclaimer as set out in the Work's software license. Therefore, I believe you have willfully infringed my rights under 17 U.S.C. Section 101 et seq. and could be liable for statutory damages as high as $150,000 as set forth in Section 504(c)(2) therein. I demand that you either comply with the terms set forth in the Work's software LICENSE, or immediately cease the use and distribution of all infringing works derived from the Work, and all copies, including electronic copies, of same. And furthermore that you desist from this or any other infringement of my rights in the future. If I have not received an affirmative response from you by October 17th, 2017 indicating that you have fully complied with these requirements, I shall consider taking any and all legal remedies available to rectify this situation at any time without further notice to you. This letter is without prejudice to my rights and claims, which are expressly reserved. Sincerely yours, Yawning Angel

2 3

Analyzing bwauth disagreements
by Tom Ritter 10 Oct '17

10 Oct '17

In which I pretend to know how to do statistics or data analysis; but really hope I spur people to point out what I'm doing wrong =) Methodology: 1) Run two bwauths on the same box 2) Collect votes (and raw bwauth votes where available) for a month from all bwauths (including the duplicate one) 3) Stick the tuple (vote timestamp, relayid, bwauthid, bw value) into a database 4) Measure the percent difference in bandwidth for the same relay at the same vote time between two bwauths as (abs(r1.bw - r2.bw) / ((r1.bw + r2.bw) / 2) 5) Average that value for all relays for a given vote time to get the overall percent difference for that vote time 6) Graph it for multiple bwauth pairs 7) Get a slight flavor difference of the above value by limiting it to relays with a bandwidth value of at least 100, to see if there are any differences Code: Download and archive votes and raw bwauth votes: https://github.com/tomrittervg/bwauth-tools/blob/master/download_files.py cron it up, run it every hour at 15 minutes after the hour. MySQL Database schema: https://github.com/tomrittervg/bwauth-tools/blob/master/schema.sql Process the files and put them in the database: https://github.com/tomrittervg/bwauth-tools/blob/master/files_to_database.py Run the last query to get the data: https://github.com/tomrittervg/bwauth-tools/blob/master/queries.sql Plot it: https://github.com/tomrittervg/bwauth-tools/blob/master/plot_data.py Optionally run something like grep -v "\\\\N,\\\\N,\\\\N,\\\\N,\\\\N,\\\\N,\\\\N,\\\\N" query.csv > non-blank-lines.csv, or change non-blank-lines.csv to query.csv ~ One month of data is 29,194,717 rows or ~4GB of data, and you want to run the query and the plotting overnight. Conclusions: The data!! https://raw.githubusercontent.com/tomrittervg/bwauth-tools/master/data.png x axis: unix epoch of the vote time y axis: percent disagreement between maatuska's bwauth and the indicated bwauth The two bwauths run from maatuska agree quite consistently. However they have the only instance of DISagreement between the 'all' set of relays and the '>100' set. maatuska and moria agree eerily closely. two bwauths agree with each other to the same amount consistently over time, although how much they agree depends on the two bwauths being compared the 'base' agreement, how much two identical bwauths will agree, is around 35% there is not much difference between 'all relays' and 'relays > 100' Follow Up: Is my data correct? maatuska and moria being that close together is suspicious. Compare moria<->longclaw, moria<->faravahar, and longclaw<->faravahar Change the bandwidth limit to >1000 and see if they produces more or less disagreement I can start applying bwauth patches, and run the new bwauth code, from the same vantage point and use this methodology to determine if new code has altered the results. IF it has altered the results: are they better results or worse? Don't know! I should measure the disagreements between bwauths when grouping relays by country. Because I have raw vote data from maatuska and moria, I can test the hypothesis that relays fall between gaps in scanners. Maybe. I could probably get positive proof this occurs but maybe not proof it _doesn't_ occur. -tom

4 8

Damian's Status Report - September 2017
by Damian Johnson 09 Oct '17

09 Oct '17

Hi all, here's what I've been up to this last month... http://blog.atagar.com/september2017/ Cheers! -Damian

1 0