May 2017 - tor-project - lists.torproject.org

Weekly network-team meeting: notes and logs from 5/22
by Nick Mathewson 22 May '17

22 May '17

Hi, all! You can find the transcript of our regular network team meeting here: http://meetbot.debian.net/tor-dev/2017/tor-dev.2017-05-22-17.00.html Below are the contents of our meeting pad. ========================= Network team status updates, 2017 May 22 Discussion topics: - What is blocking 0.3.1.1-alpha, if anything? - What do we want to know about onion services? (teor is writing code to collect onion service stats on relays) - How do we make sure that the bugs get fixed? (in 0.3.1.x) (and in 0.3.0.x) - We should do more to test out oniongit.eu. What? - Can we publicize our 'easy' tickets for new/volunteer devs, and put more tickets in that category? - And remove ones that aren't so "easy" ======== Updates ======== teor: Last Week - Feature freeze for PrivCount 1.0.0 (PrivCount collects tor stats and aggregates them securely.) Paper: http://www.robgjansen.com/talks/privcount-ccs-20161027.pdf Source: https://github.com/privcount - So much PrivCount testing, fixed several counting and reliability bugs - Found a PrivCount scaling issue that will probably need a protocol redesign to fix - Refresh the fallback directory mirror list: https://lists.torproject.org/pipermail/tor-relays/2017-May/012285.html This Week - Update the PrivCount README (set up a live privcount network and make sure it works) - All going well, release PrivCount 1.0.0 - Start configuration and deployment for the next PrivCount collection run (maybe?) - Do some tor or chutney work: anyone have any suggestions? I might revise the chutney DNS thing - https://trac.torproject.org/projects/tor/ticket/21903 - Work on the PrivCount Onion Service collection design: what do we want to know about onion services? Nick Last week: * released 0.3.0.7 * froze 0.3.0 * dubious merges wrt timing: #1922, rust. Should I not have? * prepared for 0.3.1.1-alpha (changelog, testing, cleanup, etc etc) * worked on bugfixes * tried to get proposal status right on more proposals * distracted by AC installation (heat pumps are nifty) This week: * release 0.3.1.1-alpha * try to get us focused on bugfix-mode for 0.3.1.x * start prep for 0.3.2.x * Try to get more progress w/ catalyst on the Tor side of UX improvements. Roger: This weekend: I started plowing through old trac tickets, to give opinions on many of them so people can move forward. This week: Continue doing more trac tickets, especially around the new guard-spec design, and related future guard designs. asn [Will not be around for the actual meeting :(. Have to attend a birthday dinner. Feel free to leave stuff for me on IRC and I will pick it up tomorrow] [*] Last week: - Lots of debugging/testing on #21969. Finally submitted a patch for the bridge case today. - Posted a list of open topics for HS guard discovery research: https://lists.torproject.org/pipermail/tor-dev/2017-May/012260.html - Worked with Mike to make schedule for the guard parts of the June network team meetup. Need more feedback/brainstorming/thought. - More work on prop224 client-side descriptor fetching (#21403). - Updated guard-spec.txt based on feedback by Roger: https://lists.torproject.org/pipermail/tor-dev/2017-May/012265.html [*] This week: - Finalize #21403, or bring it very close to the end - Hopefully debug/fix the non-bridge case of #21969 as well. - More preparation for the network team hackfest. catalyst: * Last week: - based on IRC discussion #7869 needs some more work, including spec changes for a new consensus method - reviewed #21872 fixup patch - helped toby_ with test stuff. discovered that test-stem doesn't really work; filed #22301 about that - found a shell arithmetic syntax error in chutney proposed a patch in #22302 - during some background research on bootstrapping found glossary needed some additional definitions; added in #22296 - looked at some torbrowser-launcher stuff (sig verification and maybe pushing fixes for that into Debian/Ubuntu) * This week: - review #21969 stuff some more - document bootstrap process - make more progress on bootstrap UX improvements - sort out some apparently conflicting mental models of bootstrap, e.g., #22266 - give feedback on #7869 ahf: Last week (unordered): Sponsor 4: - Got the consdiff and directory code to work together and landed it (#21667). - Fixed issue: #22305. - Did a lot of testing with Chutney locally with current Tor HEAD + older releases. - Started to look into merging prop #278 changes to dir-spec.txt - Reviewed some patches for prop#140. Misc: - Read up on prop #247 + its discussion as preparation for the upcoming meeting. - Got trip booked for the upcoming team meeting. This week (ordered): Sponsor 4: - Finish the merging of prop #278 to dir-spec.txt (#22275) - Fix issues with #22286 - Testing/bugfixing of 0.3.1-alpha release. Misc: - Need to spend some additional time wrapping my head around the guard proposal for the upcoming meeting. komlo: - Last week: - Created a separate branch for rust_smartlist for review/to build upon. - Worked with manishearth for a POC to handle strings between Rust/C. This uses malloc and free, which would keep the Rust/C boundary more decoupled. We can re-review as we get further into Rust implementation: github.com/Manishearth/freestring - This week: - Finish implementing protover_is_supported_here and protover_compute_for_old_tor (will be ready for review after these) isis: Last Week: - Emails about contracting/employment - Spec'd #18329 and tried to move it forward - Did other things in my time off This Week: - Doing other things, let me know if help is needed somewhere urgently haxxpop: Previously: I didn't do anything 'cause my paid job. This Week: I think I have more time now. I will find something to do. pastly: - boring new-hire NRL travel stuff. Hopefully will get done before Wilmington - running experiments to determine if high priority traffic is correctly being prioritized over low priority traffic. Inconclusive so far. Is it network variance? Shadow is so nice. So hard to be sure of things on a live relay. - for the above point, looking at EWMA cell_count values vs Tor application queuing delay. Also looking at TCP information. This week: - more of the same prioritization verification - time to make serious efforts at documentation/tests for kist code mike: Last week: - Worked on Guard meeting agenda with asn - Discussion of catalyst and nick's bootstrap doc - Planning travel - Padding branch work This week: - Moar travel logistics - Padding Sebastian: - reviewed a bunch of Rust code, preparing future branches for smartlists/consdiff - not that much time atm :/ isabela: - last week: was sick for most of it, but sent out sponsor4 report and worked on invoices for sponsor4 and R - this week: i worked last week on my notes about the tor launcher whiteboarding session i had with mike and isis, i hope to finish that this week and share with people (from the launcher automation email thread) and schedule a meeting to pick up on that, discuss my notes and hopeflly answer some open questions I have.

1 0

Notes from May 18 2017 Vegas team meeting
by Karsten Loesing 22 May '17

22 May '17

Notes for May 18 2017 meeting: Nick: 1) UX discussions seem to be confusing on our end; requirements still unclear. Progress being made, though. 2) 0.3.0.7 released, 0.3.1.1-alpha coming soon. Freeze in effect. 3) My house now has enough AC for me to work. 4) Initial rust support branch is looking pretty nice 5) Going to work, per arma's suggestion, on cleaning up tor-core security issue policy. Anybody else interested in adopting it? Mike: 1) Working on Guard proposal and meeting agenda 2) Padding work 3) Conference with Mozilla about IoT 4) Figuring out several upcoming travel arrangement Alison: 1) Not able to make it; teaching a workshop. Arturo: 1) Quite some progress done on the orchestration system. Now the web admin panel has ability to login, add scheduled measurements, list clients and list scheduled jobs: https://github.com/TheTorProject/proteus/pull/2. 2) Updated the Iranian testing list: https://github.com/citizenlab/test-lists/pull/161 3) We released a new version of ooniprobe-mobile that is now out on the stores supporting Arabic, Farsi, Russian, Greek and supporting http-header-field-manipulation. Georg: 1) Tor Browser 7.0a4 got out; there are about 2-3 weeks left to get Tor Browser 7.0 into stable shape 2) Attended SIF17 Karsten: 1) Released metrics-lib 1.7.0, made plans for releasing 1.8.0, maybe 1.9.0, and 2.0.0 in the next six weeks. 2) Making plans for completing improvements to in-memory statistics without breaking them. Shari: 1) board meeting and followup 2) making appointments with foundation funders 3) interviewing and resume review for communications directors 4) getting ready for Tommy to start next week; lining up initial assignments 5) researching strategic planners

1 0

VKontakte recommending Tor for circumvention, links sketchy app for iOS
by David Fifield 21 May '17

21 May '17

I was reading about Ukraine's May 16 blocking of Russian web sites and the resulting increase in Tor users: https://www.rt.com/business/388502-ukraine-bans-vk-yandex/ https://metrics.torproject.org/userstats-relay-country.html?start=2017-05-0… https://metrics.torproject.org/userstats-bridge-country.html?start=2017-05-… One of the blocked sites is VKontakte, a service like Facebook. I found that they have a page on circumvention that recommends e.g. Opera VPN, Tunnelbear, and Tor Browser. https://vk.com/page-22884714_52979730 (https://web.archive.org/web/20170520222918/https://vk.com/page-22884714_529…) Their links for Windows, macOS, and Android look good, but for iOS they are linking to some weird third-party app "VPN Browser - TOR-powered unlimited VPN": https://itunes.apple.com/ru/app/vpn-browser-tor-powered-unlimited-vpn/id961… Perhaps someone could get them to link something more reasonable?

1 0

Core Tor Team work for sponsor 4 in April
by isabela 16 May '17

16 May '17

Core Tor Team April 2017 Report: We finished the remaining tasks related to the implemention of Proposal 140 [1], "Provide diffs between consensuses". * #21646 - Prop140/compression: Refactor "directory request" code [2] * #21647 - Prop140: directory caches cache multiple past diffs or consensuses [3] * #21869 - Labeled storage backend for consensus docs and diffs [4] * #21648 - Prop140: Caches generate diffs as appropriate [5] * #22096 - Increase MALLOC_MP_LIM in sandbox.c [6] We continue to work on Proposal 278 [7], "Directory Compression Scheme Negotiation", finishing the following tasks: * #21664 - Prop278: Make the current 'torgzip' module a submodule of a new 'compression' module [8] * #21662 - Prop278: Add support for LZMA2 and/or Zstandard [9] * #22066 - Add memory measurement code in LZMA and Zstandard compression backends [10] * #21665 - Prop278: Establish an upper-bound for LZMA2 memory usage [11] * #22051 - Use streaming compression API in the non-streaming compression API [12] * #21663 - Prop278: Refactor the torgzip module to support additional compression schemes [13] * #22065 - Prop278: Parse the Accept-Encoding header and pass it to "get" handlers [14] * #21667 - Prop278: Handle new headers in directory.c [15] * #22085 - Refactor and simplify compression tests [16] We also hosted a follow-up meeting with Metrics Team regarding the impact of these efforts on the their work. Notes from this meeting can be found at: https://pad.riseup.net/p/chikeuBN1OUh [1] https://gitweb.torproject.org/torspec.git/tree/proposals/140-consensus-diff… [2] https://trac.torproject.org/projects/tor/ticket/21646 [3] https://trac.torproject.org/projects/tor/ticket/21647 [4] https://trac.torproject.org/projects/tor/ticket/21869 [5] https://trac.torproject.org/projects/tor/ticket/21648 [6] https://trac.torproject.org/projects/tor/ticket/22096 [7] https://gitweb.torproject.org/torspec.git/tree/proposals/278-directory-comp… [8] https://trac.torproject.org/projects/tor/ticket/21664 [9] https://trac.torproject.org/projects/tor/ticket/21662 [10] https://trac.torproject.org/projects/tor/ticket/22066 [11] https://trac.torproject.org/projects/tor/ticket/21665 [12] https://trac.torproject.org/projects/tor/ticket/22051 [13] https://trac.torproject.org/projects/tor/ticket/21663 [14] https://trac.torproject.org/projects/tor/ticket/22065 [15] https://trac.torproject.org/projects/tor/ticket/21667 [16] https://trac.torproject.org/projects/tor/ticket/22085

1 0

Accidental email triggered during drupal migration
by Silvia [Hiro] 16 May '17

16 May '17

Hi everyone, Unfortunately during a migration, some emails were triggered asking old tor-blog users to activate their accounts. Please ignore that. And sorry for the inconvenience. -silvia/hiro

1 0

Notes from weekly network team meeting, 15 May 2017
by Nick Mathewson 15 May '17

15 May '17

Hello! Our weekly meeting logs are available here: http://meetbot.debian.net/tor-dev/2017/tor-dev.2017-05-15-16.59.html Below is a copy of the pad where we wrote our status reports: ===== Network team status updates, 2017 May 15 FEATURE FREEZE TODAY* Discussion topics: - How have the stabilization periods worked historically ? [AHF] - And how should they work this time? [nm] - #21969 seems important! What to do?[asn] - upcoming 0.3.0 and 0.3.1 releases [nm] - hackfest date is June 12 through June 16 -- we might miss isis and aaron j. with those should we try for 26-30 week? - Bootstrap doc? #22265 ======== Updates ======== dgoulet is out this week. nickm: Last week - worked on reviewing merging a whole lot of tickets - worked on consensus compression and storage - today: just review and merge. - Need somebody to review my pending 0.3.1 patches, today if possible? This week - new 0.3.0 and 0.3.1 releases ahf: Last week: Sponsor4: - Prop#278 changes, testing and finishing. - Reviewed various patches and some spec changes. - Helped out with an issue in compression issue that affected moria1 This week: - Get things in for 0.3.1 freeze today :-) - Stabilization period? - Look into next Sponsor4 tasks? pastly: vacation Sebastian: Doing some light rust stuff. Would still appreciate review on the autoconf stuff asn: This week: - Prepared skeleton implementation of prop224 client-side desc fetching #21403. Code needs more tests, docs, beauty. - Put #22006 in needs_review after addressing Yawning's review. - Revised the e2e #21859 branch based on David's comments. It's now ready to be integrated to the larger service-side branch. - Tested the TROVE-2017-002 patch and confirmed it fixes the bug. - Posted a plausible theory in #21969 about what the bug could be (see last comment). - Did some SponsorR bookkeeping Next week: - More work on client-side descriptor fetching/caching #21403. - More work on #21969. - More work on guard discovery stuff. komlo: - Last week - Finished protover_compute_vote, getting closer to a full protover Rust implementation - Helped out with #21846 - This week - Continue protover Rust implementation (hopefully should be done soon) - Look into overriding CString to use malloc and free Mike: Last week: - Reviewed on netflow testing ticket issues - Worked with Isabela and Isis on Bridge discovery plans + UX - Concerned about Guard discovery hackfest This Week: - Reviewing bootstrap doc - Planning lots of travel - Padding catalyst: - Last week * [Coverity] #22221 (CID 1405983) fixed dead code in tests by swapping two loops * [Coverity] #22231 (CID 1397192) an old issue that i figured out the underlying problem for * found that a Linux VM with 2GB RAM seems to make --enable-fragile-hardening behave better in the test suite; was previously getting OOM-killed in slow/crypto/s2k_pbkdf2 * [sponsorM] my obfs4 boostrap failures probably related to #21969 and directory guard stuff. added some comments and uploaded a log. * [sponsorM] reviewing nickm's writeup about bootstrap state modeling; also correspondence with Linda draft at https://pad.riseup.net/p/OroadsO1qBgA * [sponsorM] reproduced jump-to-80% on Linux with obfs4 and existing state. also some 120 second timeout before giving up on a down bridge? maybe this should be shorter? * [sponsorM] opened #22232 to gather info about how Tor Launcher uses the control channel; also read some of the Tor Launcher implementation and have some confirmation that it uses the raw progress/phase numbers as percent completion on the progress bar - This week * draft a wiki page with a high-level overview of the bootstrapping process (#22265) * outline some incremental changes to bootstrap progress reporting that can make a large UX improvement * look some more into #21969 * bug triage

1 0

Linda in Stockholm this Tues-Friday
by Linda Naeun Lee 15 May '17

15 May '17

I am going to the Stockholm internet forum. I will be checking email periodically but may miss meetings.

1 0

We are upgrading our blog!
by Silvia [Hiro] 15 May '17

15 May '17

Hi all, In the next few days we will be in the process of upgrading blog.torproject.org. While we get prepared to refurbish the space, the current blog will be frozen for new content, starting now. This includes posts and comments and, if everything goes as planned, it should only a few days. Thanks in advance for your patience. See you on the other side. -hiro/silvia

1 0

Notes from May 11 2017 Vegas team meeting
by Karsten Loesing 15 May '17

15 May '17

Notes for May 11 2017 meeting: Nick: 1) Wrapping up 0.3.1.x features; on to bugfixes next. 2) Progress wrt personnel Alison: 1) Lots of grantwriting 2) Trying to make it to Seattle at the end of this month/beginning of next. 3) LFP just initiated a big survey to many of our participating libraries to see what they've done wrt privacy practices and policies. 4) Community team continues to work on support portal content. 5) The trainers we're working with in Pakistan just wrapped up a big youth summit in Peshawar. Waiting for the report back on that! 6) Community team members will be helping with the mirror list redesign (#22178) Arturo: 1) Releasing tomorrow a new version of ooniprobe mobile with 4 additional languages: Arabic, Farsi, Russian, Greek 2) Wrote/published OONI team status report for April 3) We now have much better monitoring of OONI infrastructure based on prometheus with alerting to slack, IRC and email when pieces of the infrastructure go down 4) On Sunday Maria and Arturo are going to be speaking at Open Source Conference Albania (https://oscal.openlabs.cc/) Shari: 1) Reviewing applications for Communications Director. Setting up initial interviews for next week. 2) Preparing for board meeting. 3) Begun researching strategic planning firms. 4) Edited grant proposals. 5) Getting ready for Tommy (writer) to start. Lining up things for him to do. 6) Preparing for a bunch of people coming to Seattle in a couple of weeks. Mike: 1) Met with Isa an Isis about bridge distribution 2) Discussed new hires for DRL with Isa; we still need a little budget clarity 3) Working on padding branches Isabela: 1) sent reviewed DRL proposal to Laura - sent out deliverables to stakeholders involved in the proposal 2) working on organize network team hackfest 3) met with mike and isis on building the experience for tor launcher automation (notes to come) 4) writing sponsor4 april report (little late with this one) 5) hiro should freeze the blog today and we will start the migration to the new blog - we will have a blog post up and email tor-project about that 6) linda is working on wireframes of the new portals with antonella and will be at SIF next week 7) my article on coding rights (anti-vigilancia) newsletter was published, is about how to use orbot and orfox. Lots of ppl at cryptorave liked it a lot because they heard all day about tor and appreciate to learn how to use it on mobile. (will tweet it from torproject is in PT and ES) 8) synced with Brad and others on things we can invoice for sponsorR and sponsor4 9) still on prep-work to get the job posts out - hopefully pads for job descriptions will be send out by EOW Georg: 1) we are about to release another alpha (probably the last one before Tor Browser 7.0) to test our latest fixes 2) preparations for SIF 3) thought a bit more about our upcoming work regarding the bridge distribution problem (Sponsor4, obj 2.2) Karsten: 1) We'll likely complete fourth of six Sponsor X deliverables by the end of this week. 2) We're going to have iwakeh as backup operator for CollecTor really soon. 3) We're planning to put out metrics-lib 1.7.0 soon and 2.0.0 in June.

1 0

GSoC17 Student Introduction
by Pushkar Pathak 13 May '17

13 May '17

Hello everyone! I am Pushkar, one of GSoC17 accepted students. I am a third year undergraduate at International Institute of Information Technology, Hyderabad (India). I have been working on 'Ahmia - Hidden Service Search' [0][1] for some time now and will be extending my contribution through GSoC this summer. I am being mentored by Juha Nurmi (numes) and George (asn). Ahmia is a search engine that indexes, searches, and catalogs content published on Tor Hidden Services. Furthermore, it is a medium to share meaningful insights, statistics, and news about the Tor network itself. There are several improvements and upgrades required in Ahmia. *>>Tasks * ● Automate Blacklisting Fetch a list of child abuse media sites and remove these sites from Elasticsearch. Also add MD5 checksums of child abuse websites to banned database for others to check. ● Add Hidden Services page Improve the existing Add page so that adding a website stores the data to SQL Database under '/onionsadded'. From there crawler can crawl these websites once a day. Remove the entries after 1 week so that the list is fresh . ● Data visualization Graphs need to plotted for various statistics in the Statistics page. Some examples include: ○ Linking structure between sites and keyword based labeling for onions in the graph - ○ Popularity of domains according to backlinks and search clicks. I plan to use either Google Charts or D3.js to plot these graphs. ● Replace Polipo with Tor Socks5 proxy in ahmia-crawler As of now Ahmia crawler uses Polipo as an HTTP proxy to direct tor traffic. But since Polipo is now no longer maintained and torsocks can provide better functionality, the crawler code needs to be updated to use torsocks. Modules like socksipy can be used to connect crawler to torsocks. ● Upgrade support from Elastic 2.4.0 to 5.X Ahmia settings should be adjusted accordingly to support Elastic 5.X. It will require a full cluster restart since rolling upgrades are not supported in major version upgrade. Upgrading includes replacing Groovy scripts with Painless. Painless is sandboxed and a Elasticsearch targeted scripting language which replaced Groovy in Elastic 5.0.0. ● Detailed Documentation and update software dependencies A detailed documentation at ahmia.fi[0] as well as on the Github page[1]. ● Advance search options - Advance Search options as mentioned below can be incorporated in search bar to allow better customisable searches. ○ Double quotes(""): Returns pages that contain exactly "term"(case sensitive). ○ AND operator(&&): Logical AND gate i.e. it returns all the pages that contain all queries separated by ‘&&’. ○ OR operator(||): Logical OR gate i.e it returns all the pages that contain queries separated by ‘||’. This is one of the optional tasks I have included. If any of the features mentioned above is not completed in the given timeline, this feature will be dropped and priority will be given to the uncompleted task. >>Timeline Week 1 - Automating blacklisting of onions with child abuse content Week 2 - Tweaking 'Add' page to save the added onion under '/onionsadded' Week 3 - Replace Polipo with Torsocks5 in ahmia-crawler Week 4 - 1st Evaluation Week 5+6 - Data visualization of statistics Week 7 - Upgrade support from Elastic2.4.0 to Elastic5.X Week 8 - 2nd Evaluation Week 9 - Updating dependancies and documentation Week 10 - Adding advanced search options like "",|| and && Week 11 - Catch up and bug fixes I will be mailing biweekly status report to this list. Feel free to contact me if you have any suggestions or doubts. IRC: mdhash I would like to thank Juha and the Tor team for their constant support and guidance. It has been a great experience for me to contribute to TorProject and I look forward to be a core member of the community. Thanks, Pushkar Pathak [0]: https://ahmia.fi [1]: https://github.com/ahmia

2 1