Hi, here are the meetup notes.
The next Tor Relay Operator Meetup will happen on November 19, 19:00 - 20:30 UTC.
Thanks everyone for joining us!
Gus
-*-
## Relay Operator Meetup - 2022-10-22 @ 19 UTC (online)
* Recap of the agenda * Announcements: * End Of Life: relays running tor 0.4.6.x.
Some weeks ago we rejected 300 or 400 relays that are running this end-of-life version. As soon as a Tor version series goes end-of-life, there is a risk there are unfixed security vulnerabilities. The general policy is to try to contact everyone, give them a couple of weeks to update. We got some hundred to upgrade, but also a fair amount didn't.
If they contact us to explain that they upgraded, we remove them from the blocklist. The rejections we do here are only by relay fingerprint, not by IP address, so if they spin up a new relay they auto come back into the network.
There's a tradeoff here because the network is under load and Tor is actively being useful to people in Russia and Iran, so we aren't yet being too aggressive.
One other exception is moria1, Roger's directory authority, which is still running an older version because bug https://gitlab.torproject.org/tpo/core/tor/-/issues/40622 remains open. https://metrics.torproject.org/networksize.html is where you can see the short drop.
* Tor and Snowflake in Iran
In Sept, many protests began in Iran. Iran is active in censorship not just of destination websites and protocols but also of circumvention tools like Tor / Orbot.
People use Tor a lot in Iran over the past month. Vanilla Tor connections are not working, so you need to use a pluggable transport like obfs4 or Snowflake.
Snowflake is a cool new pluggable transport where everybody can be a proxy in an easy way. Just install a browser extension, or install the headless proxy.
Many new Snowflake volunteers in Sept: mid Sept 30k proxies, now 110k proxies!
Snowflake was working well, users were happy, volunteer proxies happy too. ...Until Oct 4, when we saw a huge drop in connections. Just this past week we have finally understood how the block worked. The next Orbot gets around the block. Tor Browser for computers was not effected by this block.
Read more: - Snowflake proxy metrics: https://metrics.torproject.org/collector/recent/snowflakes/ - Tor censorship in Iran: https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/96 - User counts from Iran: https://forum.torproject.net/t/graphs-of-user-counts-from-iran-since-the-ons... - Net4people thread - "Unexplained drop in Snowflake client polls and bandwidth, testers wanted": https://github.com/net4people/bbs/issues/131
* Bridge: obfs4proxy package update https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1021911
It's available on debian backports, so to get it you need to enable backports. It's not yet decided whether we will kick out un-updated bridges eventually.
Survey: how many people here are running bridges, running Debian, and using backports? Several people who use Debian say that no they don't use backports
* Tor Weather rewrite status Tickets: - https://gitlab.torproject.org/tpo/tpa/team/-/issues/40893 - https://gitlab.torproject.org/sarthikg/tor-weather/-/wikis/home
Google Summer of Code project to revive Tor Weather, which is a tool to notify relay operators of downtime or other issues with their relays. The project is over, and the student has done well.
Q: Sometimes we get alerts already from some tor-weather thing. That one is run by another person right? A: Yes, that one is run by third-party volunteers somewhere on the internet.
* State of DoS attack: * Provide a recommended set of iptables/nftables rules to help in case of DoS attacks See: https://gitlab.torproject.org/tpo/community/support/-/issues/40093 * Provide a set of commands that would gather data which could help you understand the attack
There are several different kinds of overload going on.
The first one visible on the overload graph started mid-July and caused a huge spike in overloaded non-exit relays. The performance impact was limited at that point.
The other one that started in mid Sept mostly affected exit relays. It is unfortunately visible in our performance metrics. Tor is slower because of it. :(
One challenge we recognize here is that both users and relay operators lack visibility into what is going on and what we're doing to handle it.
We worry that the attackers are watching our irc discussions and that means if we are more quiet in our work then it's harder for our community to follow along.
Q: Did proof-of-work get implemented to mitigate ddos attacks? A: https://gitlab.torproject.org/tpo/core/tor/-/issues/40634 has a working proof-of-concept of the proof-of-work idea. but much work remains. But also remember that it only aims to resolve one of the several types of attacks we are seeing.
This is a top priority, even though there isn't as much visibility as we'd like, and even though developers are distracted with funded work.
* New Network Health project
There is a new funded project, woo! The idea is:
1, improve health of the Tor network 2, combat malicious relays. 3, relay dynamics of the network
We've been working the past years on removing and rejecting relays, but the next better step is to look to the community and see how we can steer and improve the relay community.
We have our "expectations for relay operators" document, and we want to start from there and continue to do more things like it. - https://gitlab.torproject.org/tpo/community/team/-/wikis/Expectations-for-Re... - https://gitlab.torproject.org/tpo/community/relays/-/issues/18 - Project: https://gitlab.torproject.org/groups/tpo/-/milestones/44
The objective of the community side of this project is to *find solutions*. For example, should we demand a valid ContactInfo on relays? There are genuine disagreements about what to require and what to expect. We need to understand what the community is expecting so we can make good choices.
Another objective: trying to understand the network relay dynamics better. What kind of relays join and leave? What counts as 'normal' there so we have a chance to recognize anomalies?
We lack tools to automatically analyze and understand these dynamics, so we need to re-do the metrics infrastructure to have more useful database structures, so we can do the right queries easily.
Once we have that we can start annotating relays and relay groups. Do we know the operator of that relay? Of that group of relays? Does it seem similar to that other group of relays that we know something about?
Ideally we will support annotations from several different angles, to start recording what we know and what we don't know about the network.
Links: - OrNetRadar 2.0 https://nusenu.github.io/OrNetRadar/ - https://nusenu.github.io/OrNetStats/#exit-fraction-by-aroi-over-time - https://gitlab.torproject.org/tpo/network-health/metrics/relay-search/-/issu...
Looking at network health *tools* for automatically understanding e.g. exit blocking.
Lastly, there are many ways that relays can deceive the network, and we want to understand those better. For example there are known ways that relays can inflate the amount of traffic they get from clients, and there are known designs to make that harder but they are still in the research phase.
We will probably do some of our new changes directly in Arti, the rust Tor rewrite.
The project is 22 months of work and 6 months of monitoring afterward, so we are excited!
* Next Tor Relay Operator Meetup * November 19 at 19:00 - 20:30 UTC * Relay operator meetup at rC3? (no rC3 planned yet)
Plan: let's do next one Nov 19, and then if RC3 happens Leibi will organize a meet-up there too.
* Q&A -- write all of your questions here!
- Any plans for something like TorConf? virtual is okay, at least users can mass donate on the live stream. Also keep a monero wallet too and expect some donations to come :)
Yes! There will be a "State of the Onion" streaming event Nov 9 (core Tor teams) and Nov 16 (community orgs). Stay tuned for the announcement.
- Are we sure those iptables rules don't produce false positives? e.g. onion services?
This is a great question and one that has been worrying me too. Especially since many people are deploying many different rules, it seems likely that some people are over-blocking. All the more reason to centralize on a recommended good set, in that ticket.
- Would it make sense to send abuse reports to the handfull of ISPs that are the source of the DoS? If yes how to generate Logs for the abuse report?
Do any of you have contacts at OVH or Hetzner who could help us from inside the ISPs? That is, do you have a friend who works there? We could use some better community contacts here.
Also, be very careful with your logs and your reporting, because some of those might be (probably are) legit Tor clients and you could be putting them at risk.
- Any plans to lower DoSConnectionMaxConcurrentCount ? (asking because of https://gitlab.torproject.org/tpo/core/tor/-/issues/40636#note_2844146)
(Currently it is at DoSConnectionMaxConcurrentCount=50 in the consensus) What this param does is: if at any point one IP address has 50 connections open to you, you drop all future connections from it for the next hours. Complicated, but I guess the simplistic answer is "no there are no plans at this moment." Let's continue on that ticket!
- Are relay to relay connections mutually authenticated? Can I tell which relay connects to my exit without looking at the source IP? Can I as an exit operator differentiate client from relay connections?
Yes they are mutually authenticated, and Tor knows which are which but it isn't easy to check at the system (e.g. netstat) level. If you're interested in looking at it at the code level, Roger or others can help after this meetup. See how we use nodelist_probably_contains_address() for a start.
- Please work on the moderation queue of gitlab/anonticket.onionize.space - new notes remain "pending for review" for a long time recently
Gus will take a look at it and see how he can help. Thanks for the reminder!
- Outbound flooding via exits is rather effective to kill even large exits, currently this does not look like an attack against exits and is easy to mitigate - single destination IP only. Any idea on how to deal with outbound flooding to random destinations?
We've been pondering doing rate limiting *per circuit* at exits, i.e. if a single circuit makes too many exit requests, or too many requests to a specific destination, we stop that circuit from being able to make more. That defense would work for that particular exit at that moment, but would it result in the same total number of exit streams happening, just spread out over more exits and also more circuit creation pain? This question is why we haven't built that feature yet.
- Does Snowflake always route to Tor after it goes through the initial proxy? Just worried about abuse reports on my home network if not.
Yes, Snowflake routes everything into Tor. You are not an exit node and don't need to worry about that.
- I run a obfs4 bridge and the users by country are russia --> US --> Iran, why is US above Iran?
We aren't sure! One possibility is that plenty of people in the US are using bridges too. But we are also beginning to suspect that our geoip databases are wrong, especially about Iran. That is, those US users might actually be users in countries that the geoip databases don't care about.
- To fix the Iran censorship: so the standalone snowflake does not need an update ?
Correct, for this particular block, we only needed to update the client. But please do keep your standalone proxy up to date in general!
- still no deb packages for snowflake?
Meskio has been working on that package. But I think it is not yet available? There is one for FreeBSD but not Linux?
- why no progress on https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/579 - I'll ping dgoulet on Monday, I guess this fell through the cracks, sorry --GeKo
- What is the state of the new family feature?
It is still in the general plans, but nobody is working on it currently.
- Will arti 2.0 make IPv6-only relays possible?
First answer, arti 2.0 is in the distant future, there are no funders for it and thus no timeframe and no specifics.
Second answer, one of the goals of Arti is to make it easier for us to improve Tor. So in the glorious future once we have a funder for relay-support-in-Arti and a specific plan and then we have done that plan, then yes I hope we will proceed to do all of the other pending Tor fixes we have been needing. But it will be a while.
- Is there a new board member?
No, not yet. We have many good candidates and we are still working through them. Last I heard, we plan to pick more than one.
- If i have a stand-alone machine and want to help Iran is obfs4 or snowflake better? (what are the benefits of snowflake/obfs4 for a dedicated machine)
A: A standalone snowflake is a good pick at this point, since it will reach more users more scalably. Both of them are useful in general, and which is more useful could change from one week to the next depending on blocking by the censor.
- Team Cymru's threat monitor collects lots of information that also cover Tor IPs. Do they collect information on relays?
We talked to them and asked exactly this question, and they said no in a way that made sense to us. They definitely have a mess on their hands in terms of public relations, and also they do seem to have some genuinely dangerous products. But they don't seem to focus on Tor in particular in any way, and also there are some individuals there who value Tor and don't *want* to have their products attack it. So, we should continue being wary, and also remember there are many other threat intelligence databases out there that seem at least as scary.
On Sat, Oct 22, 2022 at 03:29:44PM -0300, gus wrote:
Hello,
We're meeting today, October 22, at 19UTC (in 30 minutes).
See you soon! Gus
On Mon, Oct 10, 2022 at 05:46:09PM -0300, gus wrote:
Dear relay operators,
The next Tor relay operator meetup will happen on Saturday, October 22 at 19.00 UTC.
## Where
BigBlueButton: https://tor.meet.coop/gus-og0-x74-dzn
## Agenda (WIP)
- Announcements
- Tor and Snowflake in Iran
- EOL rejection
- State of DoS attack
- New network health project
- Q&A
- Next Tor Relay Operator Meetup
Meeting pad: https://pad.riseup.net/p/tor-relay-op-meetup-o22-keep
Everyone is free to bring up additional questions or topics at the meeting itself.
## Registration
No need for a registration or anything else, just use the room-link above. We will open the room 10 minutes before so you can test your mic setup.
Please share with your friends, social media and other mailing lists!
cheers, Gus -- The Tor Project Community Team Lead
-- The Tor Project Community Team Lead