tor-project October 2017

tor-project@lists.torproject.org

42 participants
24 discussions

"Capture the onion" + Tor village at next Defcon
by Roger Dingledine 17 Jan '18

17 Jan '18

tl;dr I would like to (A) design a "capture the onion" contest to get people trying to break the next-gen onion service protocol and code, and run the contest at the next Defcon; (B) craft a funding proposal to help us do A well; and (C) run a Tor village at the next Defcon too. Part A: We have this new onion service design, and one of its features is that we hope it is now impossible to discover onion addresses through in-protocol flaws. How do we know that we designed it right? How do we know that we implemented it right? Imagine if we deploy a local Tor network on the Defcon network, where random Defcon attendees can run relays. The anonymity properties would be terrible, because the trust isn't distributed over a large area and because Sybil attacks and other shenanigans are likely. But new-style onion services should be able to advertise themselves, and have their onion address remain private, even in this terrible environment. The goals of the contest would be getting people to (1) beat on our new onion service design, and (2) learn more about the Tor design and code. A proper contest has a variety of puzzles at a variety of difficulty levels. (If the only challenge were to uncover our private new-style onion addresses, most people would fail, and they would quickly become bored.) So we would want to include intermediate challenges. One example would be to run old-style onion services too, since there are known attacks against the old design. In an ideal world we'd brainstorm 15 or 20 puzzles, not all onion service related, to keep the attention of a variety of skill levels, and more importantly to raise the amount of Tor clue for people at each level. (The way you actually "capture" an onion could be by visiting the website behind it, and being given a token that proves you went to it.) Many fun contest designs include both offense and defense, i.e. they pit teams against each other rather than against the game itself. I don't know how to make use of this point, but maybe we will come up with ideas. I also pondered whether we could team up with an existing contest, e.g. ask the capture-the-flag people to be a module in their bigger plans. But capture-the-flag has been going through contortions to keep the offense from being too easy -- for example, this year they surprised people with a 27-bit architecture that uses 9-bit bytes and is middle-endian. Sounds fun if you're into that, but maybe not so useful for our needs. :) Part B: Our SponsorR program manager has expressed interest in helping to make this idea happen. In fact, it started out as his idea. He's asked us to put together proposals for the low end, the medium end, and the high end of what it would cost. (I asked him for hints on what he meant by each category, and he wouldn't commit, saying instead "whatever it would cost".) Piece one that would want attention is contest design, i.e. coming up with the puzzles, and figuring out how one would implement them, and also balancing the game so it's fun -- everybody can make progress at something, there aren't any puzzles that will destabilize the whole thing, it's clear who made the most progress, etc. Piece two would be sorting through how to harden a Tor network to survive being run in this terrible environment. We'd probably want to make the directory authorities more robust, like by making them onion services themselves, and/or making them "push" the consensus to the fallbackdirs rather than being able for pulls. This piece could involve large amounts of design and coding, so we'd want to make a roadmap and prioritize items. (We will want to declare certain types of attack out of scope for the contest, and frowned upon if people do them, but also we will want to show up with a robust network.) Piece three would be the actual operations of the contest, including deploying the network and sandboxing the components, but also including running everything during the weekend. The first year will be a learning experience all around, but the more we can learn surprising things rather than expected things, the better. :) We might also be wise to deploy the network, and parts or all of the game, beforehand for playtesting. And I bet some of the puzzles will work better if people know about them, and prepare for them, beforehand. Part B2: Now, there is a strong argument that if what we want most is an exploit on the new code, we should sign up for Pwn2Own or the like, and offer a $10k+ prize. After all, weekend contests are fun for raising interest and getting people to learn more, but in isolation they may not be an efficient way of getting people to discover complicated bugs. I think it's worth exploring both approaches, and maybe combining them in some cool way if we can think of how best to do it. Note that we already have our bug bounty program in place, and it would be very straightforward to highlight and advertise this category of bug for the top tier $3k payout, or even a higher payout if we wanted. But coordinating with a higher profile zero-day event could still bring more attention to the topic. Part C: A bunch of people I ran into at Defcon expressed interest in a Tor village next year. Rather than taking on all the logistics issues ourselves, I've instead been coordinating with organizers from two existing villages, who each have no plans to use their village space from 5pm onward each evening. This way we could have a designated "hang out and answer Tor questions" spot, with quite low logistics costs. And if we wanted to make it into something more we could. Eight years ago, when I was last at Defcon, I saw many many Tor shirts. This year I saw almost none. We have definitely been paying less attention to the American hacker scene lately than we should have been. I think it would be really great to get a huge pile of Tor shirts there next year, perhaps with a table in the Tor village, and perhaps also at already established booths like EFF and Calyx. --Roger

2 2

Intent to Minimise Effort: Fallback Directory Mirrors
by teor 09 Jan '18

09 Jan '18

Hi, Fallback directory mirrors help Tor clients find the tor network, and take load off Tor directory authorities. [0] Since Tor 0.2.8, I've generated a fallback directory mirror list every 6 months or so. I'd like to minimise the amount of effort I put into this in future. (But I think it's valuable, so I won't abandon it.) I'm happy to spend half a day every 6 months to update the input lists and run automated checks on a bunch of relays [1]. But contacting relay operators to agree to go on the list (opt-in), or check their relay details, is a time-consuming process. Here's what you can do to help: * We ask relay operators to opt-in. We could ask them to opt-out (contact us to be taken off the list) instead. The opt-in list hasn't been updated for about a year, so we will have to switch to opt-outs soon to get enough good fallbacks. - If we do opt-ins, who is going to contact relay operators? - If we do opt-outs, how do we make sure operators know they can opt-out? - (And how do we make sure we get stable relays?) * I wrote down the opt-in process in [1]. Maybe it's too complicated. - Can you design a better process? * If you're good at mail merges or scripting mail, that's one way we could contact a bunch of relay operators without it being time-consuming and error-prone. - Can you do mail merges? (I can't!) * We can improve the fallback script to make it easier to use. (See [1] and [2].) - Can you write python? If anyone wants to help out with any of these things, let me know. I'm going to do a fallback directory mirror refresh some time in the next few weeks. About 12% of fallbacks have failed, and it looks like this is placing extra load on the directory authorities. [3] There's already a ticket for it [4], and I'm happy to do a minimal refresh. If we don't get 150 fallbacks from a minimal refresh, I'll switch to opt-out and see how that goes. [0]: https://trac.torproject.org/projects/tor/wiki/doc/FallbackDirectoryMirrors [1]: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectory… [2]: https://trac.torproject.org/projects/tor/query?status=accepted&status=assig… [3]: https://lists.torproject.org/pipermail/tor-relays/2017-August/012886.html [4]: https://trac.torproject.org/projects/tor/ticket/22271 T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------

4 7

Future of Tor Messenger
by Sukhbir Singh 04 Nov '17

04 Nov '17

Hi, As some of you may have already noticed, the Instantbird team (the chat client on which Tor Messenger is based) has announced that it will no longer maintain the user interface for Instantbird and instead will work towards improving the existing chat support in Thunderbird: http://blog.queze.net/post/2017/10/18/Thunderbird-is-the-next-version-of-In… So far, we have been maintaining two products: Tor Messenger and TorBirdy, and this change is an opportunity to consolidate those efforts into one application, which we are tentatively calling "Tor Communicator". By doing this, we can continue maintaining Tor Messenger for our users and adding new features to it, and also prototype the "Tor Mail" bundle as has been requested in the past. (This change will only take effect at the next ESR major version bump, in March 2018.) We would still like to stay within the Mozilla ecosystem so that we can continue making use of technologies like Tor Launcher and the secure automatic updater patches from Tor Browser. While we have not done builds for Tor Communicator yet, we expect no restrictions on the technical side as we have released "Tor Mail" bundles in the past. Tor Messenger releases have been based on Thunderbird's ESR schedule as Instantbird releases have been infrequent. This has been possible because Thunderbird and Instantbird share a common source tree in the comm-central repository. There is also the question of the future of Thunderbird itself, summarized in the "A Bright Future" section in the blog post detailing Thunderbird's future: https://blog.mozilla.org/thunderbird/2017/05/thunderbirds-future-home/ By making this switch: - Users will get access to secure chat from Tor Messenger and secure email from TorBirdy, in a single application - Thunderbird has a single-window chat interface which is different from Instantbird (and better) - We can continue supporting Tor Messenger as a chat client for IRC/Jabber/Twitter - Users will get a Tor Mail application which has been requested for a while The idea behind this email is to brainstorm this switch and solicit feedback from the community, so please share your thoughts! -- Sukhbir

12 22

Network-team meeting notes, 30 Oct 2017
by Nick Mathewson 30 Oct '17

30 Oct '17

Hi! You can see today's meeting transcript here: http://meetbot.debian.net/tor-meeting/2017/tor-meeting.2017-10-30-16.59.html Here are the updates from our pad: Network team meeting pad, 23 October 2017 Welcome to our meeting! Every monday at 1700 UTC on #tor-meeting on OFTC. (This channel is logged while meetings are in progress.) Want to participate? Awesome! Here's what to do: 1. If you have updates, enter them below, under your name. 2. If you see anything you want to talk about in your updates, put them in *boldface*! 3. Show up to the IRC meeting and say hi! Note the meeting location: #tor-meeting on OFTC! (See https://lists.torproject.org/pipermail/tor-project/2017-September/001459.ht… for background.) Meeting notes from last week: * http://5jp7xtmox6jyoqd5.onion/p/QKh0bLHntAJ1 * https://pad.riseup.net/p/QKh0bLHntAJ1 *Announcements:* * Remember, this meeting is 1700 UTC, no matter what local time is. Watch for daylight saving changes in October or November. * Team rotation: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/TeamRot… * Review-group-24 still has unreviewed tickets! https://trac.torproject.org/projects/tor/query?status=accepted&status=assig… *Discussion topics:* * Do we want to plan a hackfest in 2018? I sent some options to the list. (teor) * *Do we want/need any additional proposals for guard-discovery other than Prop247 updates?* (See the SponsorV page for lists of guard discovery tickets and a rough roadmap - this should help us decide) * Notes from network team working sessions- is there a place to find these? [I still have the notes from the Firefox-uplifting-TB-patches-and-eventually-maybe-shipping-tor-in-private-browsing-mode meeting, and I've forgotten to type them up. I'll do that this morning and send to notes@. —isis] [note that you can also probably just photograph them and send them to notes@, if you run out of time to type. -nm] Updates: (Please use *boldface* to indicate discussion topics) teor (offline): Last week: - * - I keep seeing "Attempt to open a stream on first hop of circuit" warnings in chutney in recent master and 0.3.2, anyone know why? (#24011**)* - [NM: dgoulet notes that we removed the AllowSingleHopExits option recently. Could that be at fault?] - Proposed moving IPv6 ORPorts to the microdesc consensus (prop#283, #20916 and children), and discussed the draft proposal with the dir-auth list - Tried to help arthuredelstein diagnose Tor Browser connection timeouts: we think it's caused by overloaded exits, which probably needs some bandwidth authorities to change location, or more bandwidth authorities (#21394) - Ripped out buggy IPv6-specific v3 single onion service code, but kept support for IPv4 v3 single onion services (#23820) - We want to reduce relay bandwidth stats intervals, to make client guards harder to find using these stats (#23856) - More fun with missing microdescriptors! (#21969 and children) - Post-travel admin - Experimental PrivCount deployments, bug fixes, and feature development This week: - Experimental PrivCount onion service client descriptor fetches, intro and rend failure counts - Nudge the prop#283 thread, and once we're sure we want it, get the code reviewed mikeperry (offline - biological timezone and sleep issues): Last Week: - Recovering from illness, slowly but surely. - Wrote a plan for Guard discovery: https://trac.torproject.org/projects/tor/wiki/org/sponsors/SponsorV - *Please comment and/or update*it! I will check scrollback. [NM: Poked asn, teor, armadev about this.] This week: - Finishing up #23100 and #23114 (basically they need tests and asn had some trac comments as well). - If time/energy, start writing an experiment plan for the prop247 performance simulator. Nick: Last Week: - Six releases. Whee! - Reviewed and merged a bunch - Including the protover-in-rust code! - - *Are the doc/HACKING/*Rust.md documents up-to-date*? - CK: I am planning on reviewing these this week to fix up anything that isn't - [ I'm also happy to go over them. —isis] - Met with Isabela for roadmapping, planning, catch up - Helped a little with modularization proposal - worked on hsdescv3 fuzzing a little This week: - Wrap up october tasks; plan more about november. *(Remember the roadmap?)* - Review and merge more stability fixes for <= 0.3.2 - Write two or three proposals/design docs: - Improved APIs for control on mobile/embedded systems - Privcount with shamir-style secret splitting (if somebody can help figure out the math) - Ed25519 ID -> Consensus proposal - Coordinate w/ other teams on roadmap dependency issues - Follow up to Tim's n-t-s email. komlo (offline): Last week: - - Helped with the protover-in-rust merge! This week: - Work on fixing up some post-review improvements (#24029 and #24032) haxxpop: - Last week: - nothing - This week: - - Try to learn Rust and contribute something - - Maybe, I will take a look on fuzzing some http request - Tabish: This week: -Try to figure out how tor works and look at some easy tickets to find an interesting one. ahf: Last week: Sponsor 8: - Reviewed ticket #23845, #23848, and #23900. - Upstreamed some small patches to help profiling of Tor on Android to Orbot: github.com/n8fr8/orbot/pull/91 - Meeting with Isabela + Nick about the modularization proposal. - Looked into an issue in Orbot: github.com/n8fr8/orbot/issues/90 - Looked into simpleperf again with some more luck. Some sample reports can be seen at https://people.torproject.org/~ahf/sponsor8/sample-reports/ - Rebased my Orbot repository with Nathan's recent changes to upstream Orbot. - Opened #24061 (master) + #24062, #24063, #24064 and #24065 to track S8 performance work and added them to milestone spreadsheet. Misc: - Random small catch ups from notes from Montreal. This week: Sponsor 8: - Profiling of memory + CPU + battery baseline, discuss results and figure out where we should dive in first in November (#24065). - Update build instructions and hopefully get someone else to try it out. Sponsor 4: - Look into Nick's review in #22275 that got lost in all the BornHack stress in August/September. Misc: - Bug triage duty. - End of the month tasks: Harvest + Montreal reimbursement. dgoulet: Last week: - Worked on all my assigned 032 tickets which was most of my week. - Investigated the possible reasons on why we've lost ~500 Running relays in the network. I didn't want to exclude a tor bug. - Did some scheduler design work, no code yet. (#23993) - Writing an email for tor-dev@ that should go out soon about the state of connection/channel/scheduler for us to understand better the problem we are facing with the scheduler subsystem. This week: - Continue the work on 032 tickets and hopefully review some! - More progress for the scheduler work (#23993) because it might not be that small and 033 merge window is open! - Need to tackle the HS circuit timeout situation basically write down current state and discuss at least with Mike what to fix or change. catalyst: last week (2017-W43): - HackerOne triage - we might want to refine our HackerOne policy somewhat - reviewing #23816 (jittered backoff) - reviewing #23532 (add_laplace_noise()) this week (2017-W44): - follow up on some HackerOne stuff - more review of #23816 and #23532 - *how hard do we want to investigate why 0.2.9 seems not to be vulnerable to the #20532 bridge bypass?* - [NM: If we are fairly sure that the bypass doesn't happen, let's ascribe it to the big guard rewrite in 0.3.0] isis: last week: - reviewed/patched #18329 and #23594, so BridgeDB will now not distribute bridges that request not to be distributed - looked into an issue with some bridges not marked running (#23958) - started a patch for #16564 to keep bridges (with a "bridge-distribution-request" server descriptor line) out of the consensus - signed up for Tor's HackerOne * * would someone be willing to answer some questions about triage? * * * there's TB bugs in there, do we triage those? is there another account for TB? * * * there's non-security bugs in there, do i just make normal tickets for those? * * * how does giving people credit for finding bugs work? do i just credit them for any bug, even if it's not a vuln? * this week: - getting integration tests for moat to pass (#15957) - - more roadmap, making tickets, triaging - - spec writing Isabela: Last week: - submitted the modularization proposal - met with Nick to talk about the team roadmap - specially november and december tasks. This week: - working with team leads on inter-team roadmap sharing - will send emails on this later today - sponsor8 report for august and september

1 0

Notes from October 26 2017 Vegas team meeting
by Karsten Loesing 27 Oct '17

27 Oct '17

Notes for October 26 2017 meeting: Alison: 1) The code of conduct draft is being reviewed by the Community Council and Erin W right now. Anyone else who wants to review it before the proposal period on tor-internal can reach out to me directly. 2) I'd like to put together a Tor talk for RightsCon (I don't need to be the one to deliver it). Ideas about what talk we could give there? https://www.rightscon.org/submission-guidelines/ 3) Yay, the Core Contributor guidelines are in effect and people are using them! 4) I'm going to be reaching out to more people to join the Speakers Bureau. If you know of someone who wants to give Tor talks, ideally someone who speaks a language other than English, please send them my way. 5) Phoul and I are working on a blog post to encourage more relay operators. 6) Trying to help out where I can with the Primavera Hacker Tor Meetup 7) Finishing the Community Team roadmap! 8) Surveys are pouring in from the meeting. I'll send a reminder today. Nick: 1) Stable releases are out, again; new alpha to follow. 2) Team dividing energy between 0.3.2 stabilization stuff and 0.3.3 stuff. 3) Have been hanging out with Isabela yesterday and today to do planning, roadmapping, etc. 4) Currently under review 5) We should have optional rust code that actually _does_ something in Tor within the next couple of weeks 6) Several branches for mobile improvements implemented; waiting for review. Mobile developers, do the patches for #23845 and #23900 look like they will do what you want? Isabela: 1) working on the inter-teams roadmap sharing plan: https://storm.torproject.org/shared/3nodOOVmxGzmtuIxSiZoyO4jWtqmTyzqBpvE1bp… 2) finishing up Modularization proposal - deadline is tomorrow (friday) 3) got NCE for sponsor4 - new deadline is Jan 31st 4) had a great time meeting with Nick here at his house cover a lot of stuff related to team organizing and work planning. 5) network speed test metrics collection time - need to spam ppl to do that Arturo: 1) Preparing a blog post documenting the design and architecture decisions for the OONI Probe desktop apps to be released tomorrow 2) Making progress on the desktop apps development front 3) Did a weekend doing design stuff with Elio in Rome 4) Published an article on using OONI data on Data Driven Journalism: http://datadrivenjournalism.net/resources/ooni_data Karsten: 1) Finished our team roadmap draft for other teams to review and tell us what else we're missing and for us to learn from other team roadmaps what else we should be doing. 2) Working on patch releases for most of the metrics tools due to an alternate Tor implementation publishing (valid) descriptors that are unlike the ones that Tor publishes. Steph: 1) We launched our fundraising campaign. Tommy has been a huge help! 2) Outreachy application period has closed, so we’re going through applications. We’ll make a decision by Tues 3) Planning a blog post on the new dirauth with Tommy. 4) RightsCon subs are due Nov 24 Shari: 1) end-of-year campaign 2) what to do about Tor messenger 3) new office for Seattle 4) DRL submission for modularization 5) meeting with other groups on dealing with harassment 6) going through Outreachy applications Brad: 1) Reviewing expense reports from Montreal and analyzing final invoice from venue 2) Closing books through September 30 and filing compliance reports 3) Working with CPA firm to finalize audit & tax return 4) Working with Jon to implement new system for managing swag inventory 5) Progress Report for Sponsor 8 Georg: 1) Rightscon brainstorming 2) Roadmapping work 3) Tor Browser releases for donation campaign

1 0

Tor powers digital resistance with help from Mozilla
by Tommy Collison 26 Oct '17

26 Oct '17

Hello world! Today, we’re launching our end-of-year crowdfunding campaign, highlighting the ways that Tor powers digital resistance and protects essential human rights around the world. We’re especially excited about this campaign because our friends at Mozilla are matching donations up to $500,000, meaning that donations go twice as far. More info (and links to donate) are on our blog: https://blog.torproject.org/powering-digital-resistance-help-mozilla Tor’s done amazing work in 2017, and I’m proud of what we’ve achieved. This crowdfunding campaign is an opportunity to start 2018 strong (and to get some awesome swag -- check out https://donate.torproject.org/). So, please donate if you can. If you can’t right now, please spread the word about this campaign. Tell your friends that, because of Mozilla’s matching program, there's never been a better time to donate to our favorite anonymity network. Thanks, everyone! TC

4 6

Re: [tor-project] Analyzing bwauth disagreements
by dawuud 25 Oct '17

25 Oct '17

Hi Nusenu, Tor Project, Tor community, tl;dr The Tor network is *highly* partitioned. The scanner code is located here: https://github.com/david415/tor_partition_scanner HOWEVER, the scanner needs a redesign... I have been using the WRONG methodology for scanning the Tor network for partitions in that I select relays which I think are interesting and then see how much connectivity there is between them. I instead would like to scan ALL the Tor relays (repeatedly at different times of day) and then later perform queries against the consensus to see which relays show up in interesting combinations of failures. The other failure in this approach to scanning is that it uses a fixed set of relays... and therefore the Tor consensus file used will become old and contain relays no longer in the consensus well before the scan is complete. And lastly, don't get gamed! Scanning from a single IP is a mistake and this is an obvious way to get gamed. For all these reasons my colleague Katharina and I plan to write a new better scanner of Tor network partions that is distributed, and that has a central dispatcher so that the new consensus documents will be used to inform "worker machines" which circuits to try and build. All that having been said, let me show you what my naive scan looked like a few weeks ago: 1. setup a machine running Tor and expose its control port as either a tcp port or unix domain socket with no authentication *edit* /etc/tor/torrc blah blah easy rtfm 2. install tor_partition_scanner virtualenv virtenv-orscanner . ./virtenv-orscanner/bin/activate mkdir -p code; cd code git clone https://github.com/david415/tor_partition_scanner.git cd tor_partition_scanner pip install -e . 3. get a recent consensus file Use consensus files from collector if you want others to be able to reproduce your results. here --> https://collector.torproject.org/recent/relay-descriptors/consensuses/ wget https://collector.torproject.org/recent/relay-descriptors/consensuses/2017-… 4. choose which relays you want in your scan Here I am intentionally NOT scanning 50 million tor circuits using the entire consensus. Instead I am using a simple python program written using the Stem library to parse the consensus file and give us all the realys with the Stable and Fast flags; among those we choose the top 100 in terms of consensus bandwidth. ./helpers/query_fingerprints_from_consensus_file.py 2017-09-21-23-00-00-consensus > top100.relays 5. perform scan of top 100 relays detect_partitions.py --tor-control tcp:127.0.0.1:9051 --log-dir ./ --status-log ./status_log \ --relay-list top100.relays --secret secretTorEmpireOfRelays --partitions 1 --this-partition 0 \ --build-duration .25 --circuit-timeout 60 --log-chunk-size 1000 --max-concurrency 100 9,900 two hop tor circuits are being built. As the scan runs you can tail -f the status_log to make sure its working. Only circuit build failures if any will be logged in the json log file. When the scan completes the status_log should display something like this: 2017-09-22T00:05:44+0000 [-] $BD4C647508162F59CB44E4DFC1C2B2B8A9387CCA -> $DD808ECE4F2E24F377CBE11E335ECDA196FE3B78 2017-09-22T00:05:44+0000 [-] $0966A24977A0B0DB62546C6F18F9578D97FE86F0 -> $AD00FB62A133F91009AD5F6503E5F21F594BC4C6 2017-09-22T00:05:50+0000 [orscanner#info] Finished writing measurement values to ./2017-09-22T00:05:50.492698-scan.json. 2017-09-22T00:05:50+0000 [-] Main loop terminated. 6. Load circuit build failures into sqlite db file ./bin/load.py --dbfile scan1.db -p 2017-09-22T00:03:31.610096-scan.json \ -p 2017-09-22T00:05:42.886622-scan.json \ -p 2017-09-22T00:05:50.492698-scan.json 7. Count the results echo "select first_hop, second_hop from scan_log;" | sqlite3 scan1.db | wc -l 2014 8. Attempt to eliminate false positives by retesting the failed circuits mkdir scan1 mv *.json scan1 echo "select first_hop, second_hop from scan_log;" | sqlite3 scan1.db > scan2.circuits detect_partitions.py --tor-control tcp:127.0.0.1:9051 --log-dir ./ --status-log ./status_log \ --relay-list relays_for_scan1 --build-duration .25 --circuit-timeout 60 --log-chunk-size 1000 \ --max-concurrency 100 --circuit-file scan2.circuits ./bin/load.py --dbfile scan2.db -p 2017-09-22T00:59:31.017246-scan.json -p 2017-09-22T01:04:35.491908-scan.json echo "select first_hop, second_hop from scan_log;" | sqlite3 scan2.db | wc -l 1947 still 1947 circuit build failures! Now tell the database to show us the relays involved in a circuit build timeout AND count the number of failures by first hop: sqlite> select first_hop,count(first_hop) from scan_log where status = 'timeout' group by second_hop; $4198BD138E5E11B15B05C826B427148CED7D99FE|2 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $7C0AA4E3B73E407E9F5FEB1912F8BE26D8AA124D|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $A6B0521C4C1FB91FB66398AAD523AD773E82E77E|18 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|2 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $DAA3D3F6FDA962885072537E3F315086B003A6E3|3 $B5212DB685A2A0FCFBAE425738E478D12361710D|1 $0593F5255316748247EBA76353A3A61F62224903|4 $7C0AA4E3B73E407E9F5FEB1912F8BE26D8AA124D|2 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $C818C0EA0BAD90F5432DBA4EE662BCBEC39D2668|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $B5212DB685A2A0FCFBAE425738E478D12361710D|1 $578E007E5E4535FBFEF7758D8587B07B4C8C5D06|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 $4198BD138E5E11B15B05C826B427148CED7D99FE|1 sqlite> A6B0521C4C1FB91FB66398AAD523AD773E82E77E is certainly an outlier! It looks like this relay hits it's bandwidth limit several times per day... and Roger and I suspect it is failing circuit builds at certain times because it is overloaded during specific times of day due to traffic spikes. The fix of course would be to adjust the tor consensus and perhaps cap this relay's bandwidth capacity in the consensus: https://atlas.torproject.org/#details/A6B0521C4C1FB91FB66398AAD523AD773E82E… Circuit build failures also have outliers where a few relays fail more than others: sqlite> select first_hop,count(first_hop) from scan_log where status = 'failure' group by second_hop; $1AF72E8906E6C49481A791A6F8F84F8DFEBBB2BA|10 $1AF72E8906E6C49481A791A6F8F84F8DFEBBB2BA|10 $A571351082A9E04F14A0A3DF27E0637231D57B84|10 $B204DE75B37064EF6A4C6BAF955C5724578D0B32|11 $F6740DEABFD5F62612FA025A5079EA72846B1F67|10 $BF0FB582E37F738CD33C3651125F2772705BB8E8|10 $A571351082A9E04F14A0A3DF27E0637231D57B84|10 $1D3F937E2053E58C18E18D43FA5153E2A9F4DC77|99 This outlier relay 1D3F937E2053E58C18E18D43FA5153E2A9F4DC77 is also quite interesting. It also seems to hit its bandwidth limit several times per day and it's also interesting to note that it's located in the UK: https://atlas.torproject.org/#details/1D3F937E2053E58C18E18D43FA5153E2A9F4D… Show me all the tor relays that failed ALL of their circuit builds: echo "select first_hop,count(first_hop) from scan_log where status = 'failure' group by second_hop;" | sqlite3 scan2.db | grep 99 $1D3F937E2053E58C18E18D43FA5153E2A9F4DC77|99 $C793AB88565DDD3C9E4C6F15CCB9D8C7EF964CE9|99 $EF6591754F9079DD122EFC2C4B52917F625A8E5B|99 $55C7554AFCEC1062DCBAC93E67B2E03C6F330EFC|99 $87C08DDFD32C62F3C56D371F9774D27BFDBB807B|99 $BD4C647508162F59CB44E4DFC1C2B2B8A9387CCA|99 $EF6591754F9079DD122EFC2C4B52917F625A8E5B|99 $1D3174338A1131A53E098443E76E1103CDED00DC|99 $734EDDC2C04B1C0184178167ABD23AE85413212F|99 $7C0AA4E3B73E407E9F5FEB1912F8BE26D8AA124D|99 99 is used because here we scanned the top 100 and therefore each relay is involved in 99 circuit builds since relay A cannot build a circuit to itself. I hope it rings loud and clear that this is a *huge* problem for the health of the Tor network that many of the top 100 Tor relays with the Fast and Stable flags cannot even build a single Tor circuit to any of the other top 100 relays! In conclusion, many of these circuit build failures are very likely NOT the fault of the relay operators but instead this points to the failure of the current Tor Bandwidth Authority system. Not only is it old and broken, even if it was to work "properly" it would still be broken by design if: a. it's not performing circuit build tests b. it's not distributed and thus more easily gameable Katharina and I plan to conduct a more formal research project in this area and scan the entire Tor network (50 million 2-hop Tor circuits) several times at some point in the near future using a new better methodology. However even before we do that, I can tell you right now you will not like the results. It's bad news. (but fret not, Tor Project is committed to fixing the Bandwidth Authority system) And yes, I am well aware that correlating these scan results with BGP ASNs is interesting. There are many other interesting queries we can do after we've collected the data to try and find malicious/intentional network partitions. One possible positive outcome of our research could perhaps be the addition of a partition scanning component to the Bandwidth Authority System. However, as it stands right now Tor Project is prioritizing bandwidth measurements NOT partition scanning, since that is currently a maintenance pain point for the Directory Authority operators that must deal with the torflow tool. Sincerely, David Stainton On Wed, Oct 11, 2017 at 09:55:00PM +0000, nusenu wrote: > Hi David, > > dawuud: > > Also I have recently done a few small scans for 2-hop circuit > > connectivity and shared some of the results with Roger on > > #tor-dev. One theory is that many of the circuit failures are due to > > traffic spikes at certain hours of the day. > > where can I find more about that if I havn't been on #tor-dev at the time? > thanks, > nusenu

3 5

Booth during the FOSDEM 2018
by syl 25 Oct '17

25 Oct '17

Hello it's a bit late (deadline for booth proposals is 3 November) but I want to propose a booth for the next FOSDEM (if we are enough). Last year, we have a common booth with 4 organisations (Tor, Tails, Frënn vun der Ënn and Nos oignons) and 11 persons. Someone are interested to participate this year? syl (from Nos oignons)

6 5

Monthly Community Council Update (23 Oct 2017)
by Linda Naeun Lee 23 Oct '17

23 Oct '17

Hello Tor! The community council had its first meeting after Montreal on Monday 10/23. We met to discuss the discussion of the case handling policy during the parallel sessions and the work done on the code of conduct during the open hack days. We plan on working on these two policies in the following month, meeting weekly. Thanks, The Community Council

1 0

Network team meeting notes, 23 Oct 2017.
by Nick Mathewson 23 Oct '17

23 Oct '17

Hi! You can see logs of the weekly network team meeting at http://meetbot.debian.net/tor-meeting/2017/tor-meeting.2017-10-23-16.59.html ======================= Network team meeting pad, 23 October 2017 Note the meeting location: #tor-meeting on OFTC! (See https://lists.torproject.org/pipermail/tor-project/2017-September/001459.ht… for background.) The last time we met, it was in Montreal. See the various session notes at: * https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Montreal/Not… Announcements: * Remember, this meeting is 1700 UTC, no matter what local time is. That's about to get confusing again. * master is now 0.3.3; use the maint-0.3.2 branch for 0.3.2. * review-group-24 exists! * Team rotation: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/TeamRot… Discussion topics: * What do we do with our roadmap? [conclusion: continue to update, integrate/combine with other teams in early Nov, per Isabela's roadmap roadmap.] * What was that other review-group we wanted? * Cut down rotations? [conclusion: let's survive with ticket triage, coverity, and jenkins] Updates: (Please use boldface to indicate discussion topics) Nick: Last week: - Went to an internews PT meeting in Boston. Argued (successfully, I hope) for issue tracker & for focus on IPC API stability. Worked on threat modeling some. - Hacked on some children of #23684 ("Make it easier for mobile app developers to embed tor") - Took a couple of days off to destress, and make up for having worked over the weekend at the meeting. This week: - Open review-group-24 keyword. ✓ - Triage bugs in stable versions; make sure people are aware of must-fix things. - Meeting with Isa on wed/thu. - Merging patches on 0.3.2, 0.3.1, and maybe 0.3.3. - Backporting things for older versions. - Should I do releases? [dgoulet: I would maybe wait for a new 032 alpha until we have the big missing MD bug resolved #21969?] [asn: wrt md/guard stuff: I think by the end of this week we will have branches for #23862 and maybe other child tickets of #21969.] [ahf: do we have a release out of 0.3.1 with the fixes for the compression non-fatal assertion issue, yet? If you do a release maybe outsource it to others again?] - Working on October items in roadmap: measurements with ahf (?), examining dirauth authentication behavior [for #20532]. - Isabela: Last week: - met with tommy and shari to start working on modularization proposal - start a inter-teams roadmap sharing process (proposed to Vegas Leads last week: https://storm.torproject.org/shared/e0u8jYHQA_6I733RnO68uM6YroskSc21xXWFQ8A… This week: - working on organizing the modularization proposal work in different docs: timeline, logic model, M&E (measurement and evaluation) etc. - pinging people to do network speed test for October - check on the first deadline of inter-teams roadmap sharing - question to networkers: what is the difference between tbb-wants and tbb-needs and similar tags at the team roadmap? [nickm:there is not currently a consistent difference.] - question to networkers: October ends soon - how are we doing with the items we added on the roadmap? should we move things around? Also, november has a lot of things without ownership, should we move them too? https://docs.google.com/spreadsheets/d/1Ufrun1khEo5Cwd6OwngERn829wU3W3eskdr… [nickm: october looks okay; we added ownership for most nov things.] - request to networkers: if you are working on a ticket that is associated with some NSF sponsor work (M, V, Q, 2, 3) please add the right sponsor code to your ticket. We are trying to keep a better track of NSF work :) thanks! [asn: Moved guard ticket #21969 to SponsorV from SponsorU. I hope that's alright. I will move more guard tickets to SponsorV this week.] [dgoulet: All scheduler and KIST related tickets (keyword: tor-sched) have been tagged for SponsorV as discussed in Montreal with Roger. Let me know if they should be elsewhere.] [nickm: I went over everything in 0.3.1 through 0.3.3 that didn't have a sponsor earlier.] - meeting Nick on Wed/Thur komlo (offline): This week: - Submit updates for #22840 based on code review at tor dev (ideally this will be done tomorrow) - Look at next Rust tickets and create any new ones based on conversations at tor dev. asn: [Might be offline during meeting. Depends on dinner timing.] Last week: - Montreal meeting happened. - Some post-meeting TODO organization. This week: - More post-meeting TODO organization. As always after a meeting, I got tons of stuff on my plate that I need to figure out how to handle. - Work on the microdescriptor issues, which is my top priority wrt technical tasks right now (#21969 and children tickets) - Get hackerone under control. - Some non-technical onion service things coming up: Start thinking about press strategy for onion services, need better tools to setup onion services. - Got some important personal stuff on Thursday and Friday this week. dgoulet: Last week: - Montreal meeting - Went over the prop224 open tickets with asn. - Recovered. This week: - Address dgoulet assigned 032 tickets (currently 16 of them...) - If time allows it, I'll try to plan out the changes discussed with pastly that needs to happen in the scheduler for 033. It won't be simple changes so early merge would be ideal for better stabilization. ahf: Last week: Misc: - Montreal meeting + recovery. - Used some of the recovery time to get more into Shadow and Rust. This week: Sponsor 8: - Create bugs based on tasks we decided to do in our roadmapping in Montreal. - Talk with Nick and Isa about "new" mobile(maybe?) tasks that was found in the mobile sessions in montreal (OONI + Nathan+Mike's sessions). - Finish baseline measurement for initial s8 tasks. Misc: - Go over small todo tasks from Montreal (mostly around some of the tasks found in different notes). catalyst: last week (2017-W42): - Montreal meeting - recovery - CoC - GutHub mirroring research this week (2017-W43): - look at hackerone - continue #20532 work - write up more stuff about GitLab and GitHub mirroring and CI for developers isis: last week: - montréal and travelling home - spec patch and patch for #18329 (making it so that bridges can request how to be distributed, also mark their descriptors as bridge descriptors) - register for mozilla all-hands in december and begin planning for it - took time off to recooperate and de-stress this week: - make more tickets for things in the roadmap - finish moat server (#22871) - type up more meeting notes

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

tor-project October 2017