Hi folks,
Some of us have been invited to the upcoming Mozilla All-Hands meeting
in Austin in December:
https://wiki.mozilla.org/All_Hands/Austin
Georg (gk@torproject) is coordinating our plans, to be sure to have
enough of the right people there. We think it would be wise to get a good
cross-section of the Tor world there -- not just a pile of browser devs.
So, if you (a) got an invite, or (b) would like to go as a Tor person,
please let Georg know.
The deadline for answering their invites is Oct 5, so be sure to do
it with enough time before that deadline.
(Technically it is not an invite, but rather a nomination for an invite,
and then on Oct 5 they'll do some internal process to decide actual
invites.)
--Roger
Hi everyone,
Below is the monthly Transifex report for September, 2017:
### Report
Attached are daily, weekly and monthly translation graphs. The Y
axis is "source words".
13.02K source words
3,429 collaborators
154 languages
41 Project resources
6 languages at 100% completion (across all 42 project resources):
Bulgarian (bg)
French (fr)
Norwegian Bokmål (nb)
Spanish (es)
Turkish (tr)
--
Colin Childs
Tor Project
https://www.torproject.org
Twitter: @Phoul
Hi, all!
You can see the logs from this week's network team meeting at
http://meetbot.debian.net/tor-meeting/2017/tor-meeting.2017-10-02-16.59.html
.
Below are the notes we typed into the meeting pad.
=============
Tor network team meeting pad, 2 Oct 2017
Note new meeting location: #tor-meeting on OFTC!
(See https://lists.torproject.org/pipermail/tor-project/2017-September/001459.ht…
for background.)
Last week's notes:
* https://lists.torproject.org/pipermail/tor-project/2017-September/001457.ht…
Discussion topics:
Things to do before Montreal:
* Teams rotation for October:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/TeamRot…
* RSVP for team dinner in montreal! See network-team ML for info.
* Learn some Rust! https://github.com/chelseakomlo/rust_kata
* Sleep; take care of yourselves; try not to show up burned-out. ;)
* RUST HACKATHON STARTS 10:30am on Monday; please arrive by 10:00am!
Nick:
* Last week
* Wrote a bunch of unit tests
* Did some research on async io in Rust
* Hunted a few bugs (including 8185, 23690); helped with some others
* Helped some with upcoming 0.3.2.2-alpha
* Reviewed and merged various tickets
* Looked at test coverage diffs between 0.3.1 and 0.3.2: we have some
problems. Summary: although our coverage rate is up, we have ~500 more
uncovered lines than we had before. To find out where they are, have
a look at https://people.torproject.org/~nickm/volatile/coverage-diff-summary
. For epecific lines, see
https://people.torproject.org/~nickm/volatile/coverage-diff.xz . For
history:
* 0.2.9: 30014 uncovered; 27157 covered: 47.50% covered
* 0.3.0: 29520 uncovered; 30288 covered: 50.64% covered
* 0.3.1: 30629 uncovered; 34805 covered: 53.19% covered
* 0.3.2: 31138 uncovered; 38385 covered: 55.21% covered
* This week
* Meet with isabela for misc planning
* PETS review
* Encourage people to defer, fix, fix, defer, etc.
* Time permitting, look forward to sponsor8 stuff.
* New England Systems Verification Day (Friday)
* Montreal prep TBD
* Help more with 0.3.2.2-alpha stuff
* Prep for 0.3.1.8 as needed?
dgoulet:
* Last week:
- Work on prop224 tickets. We've postponed the IPv6 + Single onion service
discussion until Montreal.
- Spent some time debugging scheduler KIST issues which turned out to be
more deeper issues within tor: #23709, #23710, #23711, #23712
- Worked on ticket from the 032 milestone including review/triage/patches.
Nothing big though, most of it was hs/sched related.
* This week:
- Continue the 032 milestone work. Some prop224 and scheduler stuff needs
to be done for the stable.
- I want to prep for the IPv6 hackfest in Montreal next week with teor.
Many issues in the hidden service subsystem that I want to document
properly before that.
- Need some prep as well for the circuit building expiring logic for
hidden service for which I want/need to sit down with Mike for this in
Montreal (mostly related to #3733).
teor: (offline, leave this week)
Last week:
* finish off a bunch of experimental PrivCount features and bug fixes
* review Shamir secret sharing proposal
* prepare to travel to Montreal (so many bits of paper!)
* I think there's some tor coding I forgot, too
This week:
* week off: rest, casual coding, enjoy Montreal
ahf:
Last week:
Sponsor 4:
- Fixed Zstd regression in #23551
Sponsor 8:
- Roger found an issue with the HS we used to do our download test
with. We fixed this using JavaScript to download the file in the
browser and report back the results to the server.
Release work:
- Meeting with Isis and Nick about what needs to be done before the
release.
- Reviewed: #23691, #23690
This week:
- Roll 0.3.2.2-alpha together with Isis.
- Prepare for Montreal meeting. Leaving on Friday (and will by in the
air almost the entire day).
- Finish Rust toy project.
- Update the Sponsor 8 HS to allow the testers to specify their country
when doing the testing.
- Go over things I haven't gotten around to in the last week:
go over old
bugs, end of month tasks
asn
Last week:
- Reviewed #23662 and #23603 for prop224. Worked on #23493.
- Reviewed #23100.
- Worked on #23672. Now ready for merge..
- Did some more debugging on #21969 ("missing primary guard
descriptors"). This bug is still alive and a more edge-casey than before so
harder to debug. It occured on my service a few days ago, and left it
hanging for 28 hours until I restarted it. We also received more reports of
this on trac. Needs more hardcore debugging + #23671 + #23670.
- Helped tommy with the upcoming blog post calling for prop224 testing.
- Revamped the prop224 wiki page some more:
https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions
This week:
- More prop224 testing/bugfixing/planning.
- Prepare for Montreal meeting.
- More reviews as needed.
catalyst:
last week (2017-W39):
- bug triage
- reviewed some patches (including #23643 -- Type-check struct
members that are passed to confparse)
- more #20532 investigation
- Tor Launcher progress bar meeting
- fixed a leak in tests (#23691)
this week (2017-W40):
- somewhat slowed down by respiratory virus
- Montreal travel prep
- patches for #20532
- more bootstrap progress reporting improvements
komlo:
last week:
- Submitted #22840 (review for this would be great before Montreal)
this week:
- prep for rust hackfest
pastly: last week: more kist bug wrangling with dgoulet
isabela:
Last week:
- Worked on progress bar error msgs and synced with
catalyst+mcs https://docs.google.com/document/d/1jZNFYXzii7A6gCTJtvpO-PAO-WNZa2RuEEXYJwx…
- Got tests for our network speed done in a some of our target
countries, hope to have 100% coverage for october tests (was important
to have measurements for september tho)
This week:
- organize prep work for team roadmap session in montreal
- get devs survey done in form format and send it to
developers: https://storm.torproject.org/shared/_OX6xP4qVbt5BObbZ5EhCSWOh7snk3O_5iqoaes…
isis:
last week:
- preparation for releasing 0.3.2.2-alpha
- more work on the captcha server to fix some bugs found while
writing tests #15697
- started on moat backend #22871
- organising meeting sessions
this week:
- finish #22871
- go over komlo's rust protover patches on #22840
Notes for September 28 2017 meeting:
Karsten:
1) We accidentally left MeetBot running for a couple of days after out
last team meeting in #tor-dev which led to logging the channel all the
time and which prevented the network team from using MeetBot. I can't
rule out that we'll make the same mistake again, nor can other teams.
Should we move all public team meetings to #tor-meeting instead and use
MeetBot there? I'm happy to do it, but it makes even more sense if
most/all teams do it. Conclusion: yes, we're going to try it. Roger will
send mail to tor-project explaining the plan. [Update 2017-09-30: Turns
out we did *not* log half a week of #tor-dev; we just confused MeetBot
in a way that it did not allow the network team to start a new meeting:
http://meetbot.debian.net/tor-dev/2017/tor-dev.2017-09-21-14.29.log.html]
2) We're currently meeting in Berlin to make a team roadmap for the next
12 months. We're probably going to publish it next week and discuss it
with other teams in Montreal.
Alison:
1) Montreal meeting planning continues apace. I'm still soliciting ideas
for the working sessions.
2) Lots of prep for my IMLS grant. I hope to have the application for
participants completed this week.
3) Outreachy applications are rolling in.
4) Helping Isa find volunteers for Sponsor8 activities.
Roger:
1) Mozilla all-hands meeting, let's tell people a plan for who should be
there. Plan: Roger will mail tor-project@.
2) Brad: I'm doing an NSF proposal, due Oct 10. You could influence the
budget allocations.
3) Membership secretary is in place: Damian is our new membership secretary.
4) OONI and Tor bundling: their mobile version doesn't have Tor in it,
because iOS. That's sad, and we should help them fix it. Plan: have a
session in Montreal to nail down whatever short term fixes we can do.
5) I'm doing a poor job as execdir-to-frontdesk conduit. I wonder if we
should try listing the frontdesk on the contact page, with a big warning
next to it. Plan: I will work with Alison to craft an amazing sentence
for the contact page, and we'll swap frontdesk in and execdir out.
6) What did we do with the FOSDEM invite answers? Plan: The comms@ team
is going to discuss how we coordinate talk proposals so we can be
proactive rather than just trying to respond. Also, some proposals need
to draw from many parts of Tor, so we need a shift in how we as an org
approach them.
Georg:
1) Releases should get out pretty soon; the alpha Tor Browser contains
the missing content sandboxing and the first Tor alpha in the 0.3.2 series
2) I started reviewing the Cloudflare CAPTCHA bypass extension
3) Who is doing the CCC talk coordination? <- I'll start a thread on
tor-internal to get the decision process going [GeKo]
Nick:
1) We're moving forward on various projects in the Network team: mostly
trying to fix bugs for the 0.3.2.x release series.
2) Remember: Isis and Ahf are putting out the next 0.3.2.2-alpha release.
3) Did we wind up with a boston PT meeting plan?
Steph:
1) doing pre-work for Montreal speaker training. Talking with Lyndon
again today about press.
2) Outreach apps coming in. Alison, Tommy, and I participated in a
twitter chat this week.
3) doing prep with GR for fundraising campaign, giving feedback on
acknowledgement letters. chatting again today.
4) helping with OONI comms, following launch process. Iran report blog
post coming today
6) mapping out blog content calendar with Tommy, KIST post will go out
early next week (held back bc of lots of content this week). Post on
onion testing coming up
7) joining publiccode.eu call that publicly financed software developed
for the public sector be made Free and Open Source Software
Isabela:
1) working on sponsor8 indicators for Q1
2) coordinating tasks for designer proposal with TB team and UX team -
writing activities and future plans summaries for tommy to use in the
proposal (due oct 4th)
3) working with network and tb team on sponsor4 tor launcher new experience
4) Linda is finishing up website redesign mocks for me to get it printed
for Montreal discussion session / Antonella is also helping out the
team, she is creating components from bootstrap using our style
guideline (colors fonts etc) - this will be super helpful for anyone
that wants to build sites or pages for their projects following our
guidelines
5) Hiro is one of the 'consumers' of Antonella's bootstrap resource -
she is working on newsletter archive and signup/unsubscribe pages - she
is also finishing up our survey tool (which we will use for a brand
survey for onion browser and a needs finding survey for network team)
6) DRL responded to modularization proposal from core tor - we need to
reduce budget from 900k to 700k, and apply as cost extension of current
sponsor8 grant.
Arturo:
1) We launched ooni-run: https://run.ooni.io yay!
2) Just published the Iran report:
https://ooni.torproject.org/post/iran-internet-censorship/
3) We have been discussing how to make the life easier for mobile
developers that want to integrate Tor into their app, using MK as a test
case. See: https://github.com/measurement-kit/measurement-kit/issues/86
4) Deploying OONI API and the new pipeline this week.
Shari:
1) working on end-of-year fundraising
2) anything to bring up about Montreal?
3) mapping Tor - will add this to the Montreal meeting agenda under the
title "Tor on a bus."
# OONI Monthly Report: September 2017
The OONI team completed all deliverables for its current contract with the OTF.
## Stable release of Measurement API
We released the stable version of OONI's new measurement API, which is available here: https://api.ooni.io/.
The OONI API is based on the stable release of our data processing pipeline, which has been re-engineered over the last year to detect censorship events around the world faster and more accurately.
We added the following to the OONI API:
* API endpoints for listing and filtering anomalous measurements
* API endpoints for downloading full reports
We changed the following:
* Reverse sorting in `by_date` view and hide measurements from time travellers
* Better API documentation thanks to redoc based on OpenAPI 2.0
* Improve request validation thanks to connexion base on OpenAPI
* Oonify the UI
* Better testing
## Stable release of the OONI data processing pipeline
We now have a much stronger data processing pipeline (https://github.com/TheTorProject/ooni-pipeline/pull/62).
The scheduling of the pipeline is based on Airflow which is more robust than our prior luigi based solution. By doing this we have been able to reduce the number of silent outages going undetected for weeks.
The new version of the pipeline allows for much quicker feedback and interation, allowing us to run meaningful SQL queries that return result within minutes (not hours). This is also the foundation for building our new OONI measurements API.
We feel confident that the pipeline is going to easily scale to up to 100 GB of network measurement data per day. The current rate is of ~20GB of measurements per day and we are able to process this amount of data within an hour (~10 min for compression, ~10 min for sanitisation, ~30 min for inserting it into the DB), so we have some margin for scaling up to 100GB a day without significant modifcations to our infrastructure.
Currently the data is being processed in 24 hour batches, but based on the stats above, we are aiming to reduce that to 12h, 8h and eventually 1h.
The pipeline has been designed to be modular and allow us to easily ingest network measurement data from other projects too.
Currently the pipeline is analysing the data looking for blockpages and signs of DNS manipulation. We have plans for adding heuristics for detecting also more advanced insights in future versions.
For the future of the pipeline see: https://github.com/TheTorProject/ooni-pipeline/milestone/2
## Implement visualisations for the measurements interface
Elio Qoshi has started working for OONI as a designer. With him we have started to define the overall design and style of all OONI related content.
Some of this work has been integrated into our living style guide available at the following URL: http://openobservatory.github.io/design.
Part of the work also includes data visualisations that are used to visualise OONI data (see, for example, the BarChart component).
## Data visualisations for reports and blogs
Thanks to the work on the design front, we have started creating templates and guidelines for making charts that much such a task easier in the future. See: https://github.com/OpenObservatory/design/issues/2.
We created data visualizations based on our new style guide that are included in the following reports:
1. Cuba: https://ooni.torproject.org/post/cuba-internet-censorship-2017/#findings
2. Iran: https://ooni.torproject.org/post/iran-internet-censorship/#findings
## Published research report on internet censorship in Iran
In collaboration with ASL19, ARTICLE 19, and Small Media, we published a research report on internet censorship in Iran.
The report is available here: https://ooni.torproject.org/post/iran-internet-censorship/
We also published a summary of the report on the Tor blog: https://blog.torproject.org/internet-censorship-iran-findings-2014-2017
## Stable release of Orchestrator
We released the stable version of Probe Orchestration.
This features:
* Support for sending push notifactions to all mobile probes that include a message
* Support for having them run custom tests via the OONI Run custom URI scheme
* Better unittesting that also speak to the database to ensure everything is working as expected
See: https://github.com/TheTorProject/proteus/pull/28
For the time being we have decided to postpone including support for instrumenting probes to run measurements, following possible security concerns, see: https://github.com/TheTorProject/proteus/issues/24. Once we have come up with a more solid specification and a proof of concept implementation of this scheme, we would greatly appreciate feedback and maybe a third party security audit of our proposed approach.
## OONI Run
We released OONI Run, a custom URI scheme for OONI Probe mobile apps that enables users to:
* Engage others with testing the sites and running the OONI Probe tests (included in OONI mobile apps) of their choice
* Embed an OONI Probe widget to monitor the blocking of their site around the world
OONI Run is available here: https://run.ooni.io/
We also published a blog post that explains OONI Run: https://ooni.torproject.org/post/ooni-run/
## Community
We hosted our monthly community meeting on https://slack.openobservatory.org on 29th September 2017 at 14:00 UTC.
We discussed the following:
1. OONI Run: Questions and feedback
2. Announcement of OONI API release: Questions
## Userbase
In September 2017 ooniprobe was run 81,258 times from 1,595 different vantage points across 149 countries around the world.
This information can also be found through our stats here: https://api.ooni.io/stats
~ The OONI team
Since you're all about to encounter a user asking you this question:
Yes, https://dist.torproject.org/ is messed up. And yes, there are
new versions of Tor Browser out today, and they are all trying
to update, and they are now all giving their users confusing errors
like "The update could not be verified" and "The requested URL
/torbrowser/7.0.6/tor-browser-linux64-7.0.6_en-US.tar.xz was not found
on this server."
The short answer is to sit tight and wait, and everything should sort
itself out over the coming 24 hours.
Unless you're one of the few people who can help fix it, in which case,
please help us make everything sort itself out. :)
--Roger
Hi folks,
Last week I was on a panel with a bunch of US Department of Justice
prosecutors who specialize in child exploitation cases. I wrote notes
for all the things I wanted to say, and of course my plan didn't stay
intact once the panel discussions began, but here are the notes for
posterity. Maybe they will be useful next time I (or you!) find ourselves
in this situation.
--Roger
Three points as general Tor intro:
* Tor's history, including funding -- NRL, EFF, State Dept, Darpa, NSF
* Two pieces to the "metadata security" that Tor provides:
the core Tor component that hides your IP address, and Tor Browser
which deals with application-level fingerprints.
* Millions of users use Tor every day -- ordinary people, activists,
censored people, militaries and law enforcement. That variety is part
of what makes it safe to use for all of them. [Story about Dutch cop
anonymity system if we want it.]
Follow-up question: The core Tor part? Why is Tor different from a
standard proxy or VPN?
* Distributed trust -- privacy by design, not privacy by promise.
* Relays are run by community, 100gbit of traffic on average
[Story about anonymizer if we want it]
* Transparency for Tor is key: design docs, specs, source code, but
also global engagement as real human beings. (It's not a contradiction
for privacy people to believe in transparency. Privacy is about choice,
and we feel that choosing to be transparent is the best way to establish
and grow trust with our communities.)
* Ok, so what are hidden services? Most people use Tor to reach websites
and other services safely. Onion services (aka hidden services) are
special addresses inside Tor that flip that around: people can reach
*you* safely.
- better security built-in
- can be faster since not competing with exit traffic
- reduced vulnerability surface area
- mobility
* We measured what fraction of Tor traffic has to do with onion services: 3%.
- Something like 7000 onion service websites up at a given time
https://blog.torproject.org/some-statistics-about-onions
- Compare to 2.5M-or-more users *each day*
(That's not nothing, but it is tiny. If you find somebody trying to
scare you with huge numbers and pictures of icebergs, make sure you
understand their business model before buying their product or believing
their claims.)
* Some examples of interesting onion services?
[Pause while we get distracted by other panelists]
"Securedrop" is a tool for people to communicate securely with journalists
-- the New York Times, the Guardian, the Washington Post, Toronto Globe
and Mail, the AP, etc all run onion sites.
(Compare to the FBI's tipline, where they pay Cloudflare to mitm it.)
Ricochet
Onionshare
* The biggest website that has an onion service? Facebook. In April
of last year they posted that 1 million people accessed Facebook over
Tor in that month. That's .1% of their user base!
* Onion services protect different metadata than https, and it's about
giving the users choice.
onion services features:
- stronger security, built-in:
- encryption
- authentication, so no dependency on the crappy CA model
- authorization, so untrusted people can't even reach the webserver
- can be faster since not competing with exit traffic
- reduced vulnerability surface area
- mobility
Surprising (to this audience) users of onion services:
Facebook mobile
Debian updates
IoT operators
Activist blogger platform example
Govt and law enforcement
-------
Child exploitation sites/users are bad for Tor! They're bad for society
in general, but they're bad for Tor in particular. We don't want them
as users. See also the discussion at the end of
https://lists.torproject.org/pipermail/tor-talk/2015-April/037538.html
What are onion services "most" used for? It depends how you count:
Internet Watch Foundation annual report has hidden services listed as
"<1%" of the problem:
https://www.iwf.org.uk/report/2015-annual-report
Terbium Labs "dark web" report concludes the majority of onion service
content is legal:
https://terbiumlabs.com/darkwebstudy.html
I hear bad people use google drive and dropbox for better bandwidth.
But all that said, I don't want to say there is no problem.
* What are some ways of screwing up your security while using Tor?
Opsec mistakes; metadata fingerprinting; browser exploits; traffic analysis.
NSA/GCHQ quote about Tor: King of low-latency anonymity systems
UN HR report endorsing Tor.
----
Contradictions for the audience to think about:
- If Tor works, you don't hear about it. So it's easy to overlook or
undercount the "good" users.
- Sometimes investigators have to choose between being able to discover
victims vs being able to bust people.
- If there is some approach that is able to compromise bad people, the
same approach can compromise good people.
- Often, the bad guys work harder on their security than the good guys.
- If we make Tor stronger, we make it stronger for all.
- There are many ways to be bad on the Internet, and fewer ways to be safe.
Central to Tor is the topic of power imbalances: those who have power are
less in need of Tor's protections than the most vulnerable populations.
Matt Blaze's great quote about politicians who ask for crypto backdoors:
"You can put a man on the moon, so surely you can put a man on the sun!"
----
Problems with "govt hacking" as a solution to "bad people":
- 1) Secrecy: we as society need to have an informed discussion, and
if governments won't tell us what they do, how can society make
a good decision?
- NSA's goals, the existence of other countries makes this even harder.
- 2) The Feds lose their zero-days, and that hurts everybody. Cf
"Shadow brokers".
- 3) When mass surveillance becomes the cheapest and easiest option for
fighting any crime...
"Well sure, maybe you trust the people in power now... but what if the
people in power change?"
I bet US govt people are especially sensitive to this argument this year.
In many ways this is the same as the Apple encryption discussion, and
the "https everywhere" discussion.