Hi,
Thank you for attending the Sysadmin 101 workshop!
You can find the workshop slides here: https://nycbug1.nycbug.org/sysadmin101/
And below the workshop notes.
Gus
# Sysadmin 101 notes - June 4th 2022
~67 people in the workshop
### Resources
Join the relay operator community: - IRC channel: #tor-relays on irc.oftc.net - Matrix channel: #tor-relays:matrix.org - Having issues to get in touch? Check this page: https://support.torproject.org/get-in-touch/irc-help/ - Mailing lists: - Tor-relays: https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays - Tor-announce: https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-announce
- Tor Relay documentation: - Documentation: https://community.torproject.org/relay/ - Support: https://support.torproject.org/relay-operators/ - Training: https://community.torproject.org/training/resources/tor-relay-workshop/ - Expectations: https://gitlab.torproject.org/tpo/community/team/-/wikis/Expectations-for-Re... - Social contract and code of conduct: https://gitweb.torproject.org/community/policies.git/tree/ https://support.torproject.org https://community.torproject.org https://forum.torproject.org
- Other resources - slides: https://nycbug1.nycbug.org/ - survey stats https://gitlab.torproject.org/tpo/community/relays/-/issues/36#note_2810037 - Running a relay isn't for everyone. If you're not comfortable running your own relay, consider running a Snowflake or Donating - Here to one of the many non-profits that run exit relays: https://community.torproject.org/relay/community-resources/relay-association... - NSA "Tor stinks" url from the Guardian https://commons.wikimedia.org/wiki/File:Tor_Stinks.pdf - Metrics https://metrics.torproject.org/
### Q/A
- How many people signed up?
- 100+. With 60-70 attendees in practice.
- Tor log: there have been x users in the last 6 hours... What's the algorithm for what a distinct tor user is? (torix)
- I believe bridges count it by IP address, rounded up to the next multiple of 8. Your bridge also publishes these stats plus more in its "extrainfo" descriptor, which you can find in https://collector.torproject.org/recent/bridge-descriptors/extra-infos/ and maybe also in the stats/ directory in your DataDirectory.
- How much time (per week or month) and how many times, should you plan to invest?
- Depends on what you're doing and how you're doing it.
- "My eyeballs are the first line of defense." Watching the Tor logs, watching the system logs, can help you get more comfortable with how things are going (and what they look like when things are going fine).
- What are the regular monitoring or upkeep activities we should be performing to not "set it and forget it"
- Log in regularly. Check for updates and if your box needs to be rebooted. (Set an alarm or calendar event to log in and check.) If using Debian/Ubuntu, enable UnattendedUpgrades.
- prometheus: https://forum.torproject.net/t/suggestion-a-summary-page-of-relay-bridge-ins...
- george is into "agentless monitoring"
- What are acceptable domains or communications approaches for listing in ContactInfo? E.g. what about a duck address.
- Use any domain that you use for normal communication. Don't use an address that you never check.
- Any contact info that you regularly check.
- DO NOT obfuscate your contact information! Maintainers already burn a lot of time trying to decipher obfuscated contact info!
- (Some people are concerned about spam, and that's why they try to obfuscate the address. But actually, spam isn't so bad these days; or if it is for you, consider using a separate email address for your contact info.)
- Is a relay that allows exits to port 53 but routes those queries to a pihole considered a bad node that is tampering with traffic? - Please no! Don't mess with exit traffic. Redirecting outgoing tcp port 53 connections to somewhere else is going to break things.
- There have been cases where a DNS on a distinct machine increased performance
- What is the best way to figure out if a bridge/IP got burned (i.e. blocked in certain countries)? What should be rotation intervals?
- At the beginning of 2022, we added a new feature where we're measuring reachability of bridges from inside Russia, and annotating relay-search with the results.
- Check metrics.torproject.org, there will be indicators if your bridge is blocked or not
- This "your bridge is blocked in Russia" feature is in-progress: the user experience at the end is not intended to be "you have to watch your metrics page and then go cycle your IP address manually". So don't worry too much about reacting to the relay-search page yet.
- Can you release a Docker image instead? Why create OS-specific packages when you can Docker?
- Bridge docker image: https://community.torproject.org/relay/setup/bridge/docker/
- https://gitlab.torproject.org/tpo/community/relays/-/issues/43
- Can you talk about partial exit nodes? For example, I block all ports except for 8333 (Bitcoin), 9999 (DASH) & 53 (DNS/TCP)
- You need to allow exiting to ports 80 and 443 in order to get the Exit flag, and most clients skip relays that don't have the Exit flag.
- So, yes you can do just those tiny set of ports, but mostly clients will skip over your relay.
- Can we run a relay on a 1 GB RAM Server? Yes, have an exit at Frantech with 1 GB https://metrics.torproject.org/rs.html#details/D00795330D77C75344C54FB880053...
- Thanks, got it. Can you share how much memory footprint does running a relay have? Like 200-300 MBs?
This is traffic dependent, lowest is 512 MB (source) the more data being moved the more RAM needed till you reach other bottlenecks (Such as bandwidth limits/CPU)
- Is it legal/okay to run a relay node on oracle cloud free tier servers?
-I run 2 guards and a bridge on their free ARM machines and have not had problems with them so far.
- Check your provider's TOS.
- Middle nodes rarely cause issues, exits cause abuse, so ask beforehand.
- What is the "Sandbox" option in torrc? Would it be a good idea to run relays with it?
- Sandbox enables certain seccomp style rules to prevent your Tor relay from doing surprising activities (like unexpected syscalls). I would say "if your Tor package turns on Sandbox, then yes, use it", but if your package doesn't use it, it is fine to ignore for now.
- Can you elaborate on deploying host-based and network-based intrusion detection for relays? What are best practices for not leaking sensitive data?
- One approach: do not store or let sensitive data touch your "unsecured" box in the first place.
- NTP or SNTP for a Tor relay?
Since Debian Buster and Bullseye, timesyncd is enabled by default. Systemd-timesyncd is an SNTP client which is less accurate than NTP. I mean, Roger once said that the time accuracy in the Tor network of 1-2s is sufficient. You can also run chronyd and use NTS: https://anweshadas.in/how-to-use-network-time-security-on-your-system/ https://www.whonix.org/wiki/Network_Time_Synchronization (http://www.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion/wi...) https://tails.boum.org/contribute/design/Time_syncing/#index1h1
- Lessons for next time: in the notification / reminder emails, suggest that people show up 5-15 minutes early, so they have time to mess with their audio and make it work.+1+1
- How about using singleboard computers like beagleboard, olimex for setting up bridges ? - Whatever you're comfortable with. They can work. They might make your experience more exciting / more work. If you're excited about gaining that new experience, go for it. - Make sure the hardware you pick can handle the throughput: fast relays don't easily fit on tiny hardware.
- What is the advantage of ed25519 over RSA, and should I change my keys? First and foremost, ed25519 is more break-proof as compared to RSA. Second, it also encrypts keys into a compatible OpenSSH format by default. Third, if you must use RSA use only 4096 bits. Fourth, YES, You should change keys asap. - If you mean your Tor relay identity keys, Tor will do that for you automatically: for the past few years, Tor will generate a new ED key for you automatically. - But if you mean ssh key or GPG key or etc., listen to kushal
- How accurate from "true" UTC does it matter? - A few seconds is the limit, but NTP should get sub 1 second constantly, so should not be a issue
- Instead of S/NTP, can we use a GPS receiver as a time source and synchronize your computer's clock to that?
- is it ok to run Tor on a FreeBSD jail? - Yes, many people do.
- Why shouldn't I list my bridges in MyFamily?
- Gives off information about related nodes, for the anonymizing layer that may be bad
myfamily is *bidirectional*, and your bridge fingerprint is supposed to remain a secret, so if you list your bridge fingerprint in your relay's MyFamily, it's no longer secret.
There are future plans for changing Tor's design, so you *can* list your bridges in your family: https://gitweb.torproject.org/torspec.git/tree/proposals/321-happy-families....
...So in the future this will get more confusing because we will change to telling you to be sure to list your bridge in your MyFamily. :)
- Does the ORPort number matter? In the beginning, the ORPort mattered a little bit, because some users are behind firewalls that only let them reach e.g. ports 80 and 443. That's why you see a bunch of relays using 443 for their ORPort. It matters a bit less these days, I'd say, because more censorship is based on things other than destination port.
- I set my ORPort to 22 and SSHD to 22222. Is this in any way a bad idea? - One upside is picking 22 for your ORPort is: it helps users get around censorship, if they are allowed to use port 22, but they're not allowed to use other ports like 9001. - One downside to picking 22 for your ORPort is: network admins around the internet look for connections to port 22 and freak out when they see them. So, they see a user on their system connecting to your port 22, misunderstand and assume it's ssh, and send a warning mail or block you or something. - DPI can distinguish ssh traffic from tor traffic, so maybe obfuscation through setting your ORPort on a known service port is not a good idea, as the service (ssh) has a distinct profile that is easily profiled and can be inspected for "baseline" behavior.
- Should I use full disk encryption (FDE)? What are the tradeoffs? If you have a serial console, yes. But after a (re)boot, the password must be inserted. The torservers.net people actually recommend *don't* use full disk encryption for exit relays in your own hardware, since that way if law enforcement steals your server it will be quickly clear that there is nothing on it. (Otherwise, they stare at your encrypted disk and assume the worst.) For *non* exit relays, the tradeoff is more blurry. There have been cases where law enforcement shows up to steal guard relays, e.g. because they have some case where they think a user connected to your IP address. But also, protecting your relay identity key from external attack (offline masterkey) is useful. But also, you should be willing to throw away your key and start with a new one if anything suspicious happens.
- Bridges should not have set myfamily, yet via metrics searching for the contact one can find relays and bridges operated with the same contact (therefore a family). Why is that? [should merge this Q with the family Q above]
- Does tor project support IRC block tor? https://support.torproject.org/abuse/tor-ban-irc/
- there's been reports of large scale malicious relays in operation, how are we dealing with that problem? https://blog.torproject.org/malicious-relays-health-tor-network/
- Is it ok to run multiple snowflake instances in a single network (for example, a university network)? What is the recommended maximum of instances per network/IP-block/IP? (And why?)
- Yes, it is ok.
- Almost all of our Snowflake volunteers are behind restricted NATs, which makes them less useful because they can't be matched with Snowflake users who are also behind restricted NAT. So if you have a choice, try to make your Snowflake not behind a restricted NAT.
- If I reinstall my server, should I keep the keys from the earlier install, or should I generate new keys?
- If you reinstall servers, you should back up your key if it is not compromised or sign your "per box" key with your offline master key. Your per box key is your "reputation" if you don't have an offline master key.
Offline masterkey?
You sign your "per box key" with a key that never touches your physical hardware or VPS that runs the tor proxy, so the chances of it being compromised is a lot less than the key that resides on your hardware.
See ⇾ https://support.torproject.org/relay-operators/upgrade-or-move/
- Can you run firefox/chrome snowflake and a middle relay/obfs4 bridge from the same home ip? - Don't mix things, different systems is a good idea in a local LAN - The reason to avoid having both kinds of bridges on one IP address is that if the IP address gets blocked because of *one* kind (e.g. Russia discovers your obfs4 bridge using bridges.torproject.org), then the other kinds (like your Snowflake) end up blocked too.
How useful is a middle relay these days with only 6 Mb/s upstream (because 10+ is required now)? - Not as useful as it was a decade ago. Consider running a bridge or a Snowflake on a connection like that. Thanks for contributing!
Notes: - SECURITY IS IMPORTANT! - DO NOT BE A MALICIOUS NODE OPERATOR! DO NOT PROXY! - AUTOMATE PATCHING!!! (unattended-upgrades) - Diversity is important: different tech stacks (network, OS, hardware) - Automate OS updates - Use selinux and built-in security features. (see "SELinux for mortals") - use ssh key authentication with ed25519 keys, don't use the default ssh port - use ssh Postquantum crypto: KexAlgorithms sntrup761x25519-sha512@openssh.com - disable root user and SSH password authentication (disable it). - Use firewalls and fail2ban - Use hardware keystores (Yubikey, Solokey and Nitrokey) https://kushaldas.in/posts/ssh-authentication-using-fido-u2f-hardware-authen... https://kushaldas.in/posts/using-your-openpgp-key-on-yubikey-for-ssh.html - Use UTC on public facing servers; Makes correlating events easier - If possible: use NTS https://anweshadas.in/how-to-use-network-time-security-on-your-system/ - Optional but recommended: @daily emails and monitoring notifications - Optional: Diversity in DNS servers (DO NOT MESS WITH DNS OR YOU HAVE A HIGH POSSIBILITY OF BEING IDENTIFIED AS A BAD RELAY) - Recommended: If you run multiple tor nodes, check "my family" option in torrc - Communicate with the community, join irc, matrix, mailing list .... (Tor is a COMMUNITY effort!) IRC channels #tor #tor-relays on OFTC. ircs://oftcnet6xg6roj6d7id4y4cu6dchysacqj2ldgea73qzdagufflqxrid.onion#tor-relays - Read change logs - Optional but recommended: keep a changelog of tor versions on your client (there are tradeoffs when it comes to doing your adversary's job of enumeration, though) - Run a snowflake in your browser! It is a low effort, high impact contribution to the network! You can run snowflake from a Firefox extension: https://addons.mozilla.org/en-CA/firefox/addon/torproject-snowflake/
- Other options here: https://snowflake.torproject.org/#extension
Next meetup ONLINE for relay operators:
June 25, 1900 UTC: https://forum.torproject.net/t/tor-relays-event-tor-relay-operator-meetup-ju...
Upcoming Hackweek, in late June: https://hackweek.onionize.space/hackweek/
On Mon, May 23, 2022 at 04:42:28PM -0300, gus wrote:
Join us June 4th at 1900 UTC for new and prospective Tor relay and bridge operators on the basic "sysadmin foo" required to contribute to the network!
## Sysadmin 101 for new relay operators
So you want to contribute to the open-source Tor network by running a relay or maybe a bridge?
The Tor network is the most important tool for evading surveillance and bypassing internet censorship. And Tor relays and bridges are vital to the health and integrity of the Tor network. Millions of users rely on relays and bridges to stay safe, and how you configure and maintain that relay or bridge is critical.
Volunteers aren't a nice enhancement. They are a core feature.
Running a relay or a bridge raises frequent questions:
- Should I run a relay or a bridge?
- Should I run a relay or a bridge from a residential/home internet
connection?
- Which operating system should I run for my Tor node (hint: the one
you are most comfortable with securing and maintaining)
- More generally, what does it take to keep that relay or bridge
operating safely, but both you and Tor users?
This workshop will start with a presentation approaching some of the core issues that arise when running a Tor node. The session will move into an "ask me anything" discussion to approach other common and less common questions.
The 90-minute event will be geared towards current and prospective Tor bridge and relay operators, particularly those relatively new to running public internet services.
Seasoned Linux and BSD Tor operators will be attending the event ready to address the discussion.
## How to join the workshop
The workshop is entirely free, and participants need to fill out this registration form. The event will take place on BigBlueButton, an online video conference platform, on June 4th at 1900 - 2030 UTC.
You can register here: https://nc.torproject.net/apps/forms/cDLPxryHJcP5kMeW
## Facilitators
The workshop will be facilitated by:
- George (@gman999) - Tor *BSD Diversity Project member, Serge bridge
directory authority maintainer, long-time relay operator and a wide variety of other contributions.
- Kushal Das (@kushal) - RPM Tor maintainer and member of the Tor
Community team.
-- The Tor Project Community Team Lead
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays