May 2012 - tor-relays - lists.torproject.org

sustained bandwidth drop through noisetor
by Andy Isaacson 03 May '12

03 May '12

noisetor-01 was pushing 300-400 Mbps of traffic from 2012-02-15 through 2012-04-13. Since mid-April we've seen traffic decrease significantly; over the last week, our daily peak has been 260 Mbps (versus 450 Mbps in March) and our daily trough has been 100 Mbps (versus 300 Mbps in March). The traffic levels dropped off in a smooth fashion over a 10 day period, April 15-25 if I read the graphs correctly. There's no sign that anything is amiss on the node; CPU and memory usage are consistently lower, today, than they were in March -- as expected, given the lower utilization. The logs are silent. I've been postprocessing torstatus's Tor_query_EXPORT.csv regularly, and it appears that the overall exit throughput of the network has gone up slightly over the last few months; some new significant nodes came online during 2012Q1 and may have affected the traffic distribution. However, while noisetor has seen a significant drop, other families of exit nodes have seen their bandwidth go up or stay steady. Has there been a change in the routing algorithm, or any other network changes that might explain this drop? Thanks, -andy

3 3

Re: [tor-relays] [tor-dev] Can we stop sanitizing nicknames in bridge descriptors?
by Karsten Loesing 03 May '12

03 May '12

[Cc'ing tor-relays, because this discussion might be relevant for relay/bridge operators, too. Please keep the discussion on tor-dev. See https://lists.torproject.org/pipermail/tor-dev/2012-May/003489.html for the whole thread.] Hi Sebastian, On 5/2/12 9:35 PM, Sebastian G. <bastik.tor> wrote: > [...] > > Do similar names actually mean that bridges are located where the relay > is? (Apparently you've got the data to see these correlations) A fine question. How do we define "similar" and "located where the relay is?" I can see how a relay "bastik1" and a bridge "bastik2" have similar nicknames, but would we also teach a program that "bastikrelay" and "bastikbridge" are similar? And are two IP address in the same, say, /30 located nearby, or is the same /28 or even /24 okay, too? So, while we have the data to see these correlations, I think that whatever similarity algorithm we come up with, somebody else might come up with something smarter. If we do the analysis you suggest and learn that it's safe to include nicknames, that doesn't say very much. Only because we have the data to confirm how well our attack would works doesn't automatically mean we're in a good position to design the attack. If you want to run this analysis with the 2008 tarball (assuming there won't be general objections within the next two weeks), I'm happy to take your list of likely bridge IP addresses and tell you how accurate your algorithm is. > "We don't need it, so better remove it." I really like that. I think we're really conservative with giving out bridge data, and that's good. At the same time there's a value in giving out information about bridges, so that "remove everything" is not a good answer. For example, I think if we give bridge operators better feedback how their bridge is doing, we'll suddenly have a lot more bridges. Making it easy for bridge operators to use Atlas would be a good step into that direction. The same applies to funders who realize from our statistics how successful the Tor Cloud project is and who then want to fund it more to make it more usable, support more cloud providers, etc. >> And are we giving away anything else with the nicknames? > > Maybe it's location ;) > > As I read "hints on the location" for the first time; I though it would > mean that "TowerBridge" or "BridgeofLondon" would be bad since it could > hint to London. Well, in that case you'd learn that there's a (Tor) bridge in London. But that wouldn't help you very much, would it? > Could it make sense to ask the same question on the tor-relay list? Here > you (the Tor people) have more data again and know who subscribed to > both lists. I for myself assume that relay and bridge operators, which > could object, because it's their naming scheme that could reveal > something, are more likely to be subscribed to tor-relays. Good idea. I added tor-relays to the Cc to let relay/bridge operators know. Let's keep this discussion on tor-dev though. >> And if nobody >> screams, I'll provide the remaining tarballs containing original >> nicknames another two weeks later. > > Probably two weeks later, since unpacking, processing and re-packing > takes some time :) I know the sanitized ones are large when they are > unpacked. Windows needs some time to delete the extracted files. Right. :) I'll probably start sanitizing all bridge descriptors at once in two weeks, starting with the 2008 ones, and provide only the 2008 tarball then. It's going to keep my CPU and disks busy for a while. Thanks for your input! Best, Karsten

1 0

Towards a Tor Node Best Best Practices Document
by Mike Perry 01 May '12

01 May '12

It's occurred to me that we have yet to provide any official recommendations with respect to best practices for operating Tor relays. This post is my attempt to define a reasonable threat model and use it to develop some recommendations. My plan is to post it here first, to subject it to review by the relay operator community. After operators have a chance to comment, the plan is to relocate the document to the Tor Wiki and/or a blog post. As always, to focus our thoughts, we start with the adversary's goals. Adversary Goals There is a significant difference between adversaries that can see inside of router-to-router TLS vs those that cannot. I believe this capability distinction governs the adversary goals in terms of compromising relays as opposed to merely externally observing them. Adversaries that can unwrap router TLS can perform every attack that an actual node can perform, at any location between the user and the node, and/or between the node and other nodes. In particular, adversaries that can see inside router TLS can perform tagging attacks (see https://lists.torproject.org/pipermail/tor-dev/2012-March/003361.html) as well as perform circuit-specific active and passive timing analysis. These attacks can be quite severe. An adversary that is able to obtain Guard identity keys is free to perform a tagging attack anywhere on the Internet. In other words, if the adversary is interested in monitoring a particular user, the adversary need only obtain the identity keys for that user's 3 guard nodes, and from that point on, the adversary will be able to transparently monitor everything that user does by way of using tagging to bias the users paths to connect only to surveilled exit nodes who also have had their identity keys compromised. Based on this distinction, it seems that some simple best practices can increase the costs of an adversary that wishes to compromise tor traffic. Let's now consider how the adversary goes about compromising router TLS. Attack Vectors There are two high-level vectors towards seeing inside node-to-node TLS (which uses ephemeral keys that are rotated daily and authenticated via the node's identity key). Both high-level vectors therefore revolve around node identity key theft. Attack Vector #1: One-Time Key Theft The one-time adversary is interested in performing a grab of keys and then operating transparently upstream afterwords. This adversary will take the form of a coercive request at a datacenter/ISP to extract identity node key material and from then on, operate externally as a transparent upstream MITM, creating fake ephemeral TLS keys authenticated with the stolen identity key. Tor nodes that encounter this adversary will likely see it in the form of unexplained reboots/mysterious downtime, which are inevitable in the lifespan of any Tor node. Attack Vector #2: Advanced Persistent Threat Key Theft If one-time methods fail or are beyond reach, the adversary has to resort to persistent machine compromise to retain access to node key material. The APT attacker can use the same vector as #1 or perhaps an external vector such as daemon compromise, but they then must also plant a backdoor that would do something like trawl through the RAM of a machine, sniff out the keys (perhaps even grabbing the ephemeral TLS keys directly), and transmit them offsite for collection. This is a significantly more expensive position for the adversary to maintain, because it is possible to notice upon a thorough forensic investigation during a perhaps unrelated incident, and it may trigger firewall warnings or other common least privilege defense alarms inadvertently. Unfortunately, it is also a more expensive attack to defend against, because it requires extensive auditing and assurance mechanisms on the part of the relay operator. Defenses It seems clear that the above indicates that at minimum relays should protect against one-time key compromise. Some further thought shows that it is possible to make the APT adversary's task harder as well, albeit with significantly more effort. Let's deal with defending against each vector in turn. Prevent Vector #1 (One-Time Key Theft): Deploy Ephemeral Identity Keys The simplest way to defend against the adversary who attempts to extract relay keys through a reboot is to take advantage of the fact that even node identity keys can be ephemeral, and do not need to persist long term (certainly not past a reboot). This can be achieved with a boot script that wipes your keys (they live in /var/lib/tor/keys) at startup, or by using a ramdisk: http://www.cyberciti.biz/faq/howto-create-linux-ram-disk-filesystem/ Of the two, the ramdisk option is superior, since it will prevent the adversary from easily re-using your old keys after you begin using the new ones. Additionally, ssh server key theft is another one-time vector that can be used to quickly bootstrap into node key theft. For this reason, node admins should always use ssh key auth for tor node administration accounts, since it prevents ssh server key theft from implying continuous server compromise: http://www.gremwell.com/ssh-mitm-public-key-authentication Issues With Ephemeral Identity Keys There are a few issues with deploying ephemeral identity keys. Issues With Ephemeral Identity Keys: Client guard node loss The primary issue with ephemeral identity keys is client Guard node loss. If your relay obtains the Guard flag, you should endeavor to keep it. If you have planned maintenance and controlled reboots, you should copy your identity keys to a safe location prior to reboot so that clients aren't forced to rotate their guards prematurely due to unnecessary rekeying. Issues With Ephemeral Identity Keys: MyFamily The next issue is that identity key rotation makes the use of MyFamily very complicated. For large families, it's nearly impossible to update the MyFamily line of each node instance after every unexpected reboot. We've filed a bug to see if we can find a more convenient way to provide the same feature, but it's not clear that MyFamily is worth maintaining: https://trac.torproject.org/projects/tor/ticket/5565 It is very likely that the security benefit from maintaining MyFamily pales in comparison the gain from deploying ephemeral identity keys, and MyFamily should be abandoned entirely. Issues With Ephemeral Identity Keys: Tor Weather The final issue with ephemeral identity keys is that node monitoring mechanisms such as Tor Weather become difficult to use in the face of rotating keys. We've filed this bug to improve Weather's subscription mechanisms: https://trac.torproject.org/projects/tor/ticket/5564 Preventing Vector #2: Isolation Hardening and Readonly Runtime Once one-time key theft has been dealt with, you can begin to consider how to deal with the Advanced Persistent Threat. The effort required to defend against this adversary is considerable, and it is not expected that all operators will devote the effort to do so. To limit scope, we are not going to deal with the daemon compromise vector; for that see your OS least-privilege mechanisms (such as SElinux, AppArmor, Grsec RBAC, Seatbelt, etc). Instead, we will deal with how you can attempt to protect your identity keys once an adversary already has root access. If you are serious about defending against this adversary, the first thing you will want to do is disable access to the 'ptrace' system call from userland, which allows easy key theft using debugging tools. Note that all current built-in kernel mechanisms to do this still allow root users to use ptrace on arbitrary processes. In order to disable ptrace for root users, you need to load a kernel module to patch the syscall table to remove access to the syscall itself. Two options for this are: https://gist.github.com/1216637 http://people.baicom.com/~agramajo/misc/no-ptrace.c Once access to the ptrace system call is removed, you need to disable module loading to prevent it from being restored. On Linux, this is accomplished via 'sysctl kernel.modules_disabled=1'. You should perform this operation as early in the boot process as possible. One technique that works on Redhat-based systems is to place a shell script in /etc/rc.modules to load the modules you need for operation, insert the ptrace module, and then issue the sysctl to disable further module loading. Redhat-derivatives launch /etc/rc.modules first thing at the top of /etc/rc.sysinit. After that comes ensuring runtime integrity. There are several ways to achieve this, but most are easily subverted by an attacker with direct access to the hardware. The most robust approach seems to be to create a small encrypted loopback filesystem that contains all of the libraries required to run the 'tor' process as well as all of the requisite configuration files. Requisite libraries can be determined via 'ldd /usr/bin/tor'. The encrypted loopback filesystem doesn't need to be more than ~25M in size, but you will also need an auxillary var loopback that needs to be a hundred megs or so. Here are the commands for creating the root loopback filesystem: dd if=/dev/urandom of=./tor-root.img bs=1k count=25k losetup /dev/loop1 ./tor-root.img cryptsetup luksFormat /dev/loop1 cryptsetup luksOpen /dev/loop1 tor-root mkfs.ext4 /dev/mapper/tor-root Once this encrypted loop is created such that it can run your relay's Tor processes, you should take the sha1sum of the file and store it offsite. When you use this loopback, you will mount it readonly, and mount an unencrypted var directory inside of it, and a ramdisk for your keys inside of that: dd if=/dev/urandom of=./tor-var.img bs=1k count=200k losetup /dev/loop2 ./tor-var.img mkfs.ext4 /dev/loop2 mount /dev/mapper/tor-root /mnt/tor-root -o ro mount /dev/loop2 /mnt/tor-root/var mkfs -q /dev/ram1 128 mount /dev/ram1 /mnt/tor-root/var/lib/tor/keys cd /mnt/tor-root/ chroot . start_tor.sh Once you start your tor process(es), you will want to copy your identity key offsite, and then remove it. Tor does not need it to remain on disk after startup, and removing it ensures that an attacker must deploy a kernel exploit to obtain it from memory. While you should not re-use the identity key after unexplained reboots, you may want to retain a copy for planned reboots and tor maintenance. scp /mnt/tor-root/var/lib/tor/keys/secret_id_key offsite_backup:/mnt/usb/tor_key rm /mnt/tor-root/var/lib/tor/keys/secret_id_key Upon suspicious reboots, you can verify the integrity of your tor image by simply calculating the sha1sum (perhaps copying the image offsite first). You do not need to do anything special with the var loopback. These steps should prevent even adversaries who compromise the root account on your system (by rebooting it, for example) from obtaining your identity keys directly, forcing them to resort to kernel exploits and memory gymnastics in order to do so. Don't forget to periodically update the libraries stored on your loopback root using a trusted offsite source, as they won't receive security updates from your distribution. One alternative to make your loopback fs creation, tor startup, and maintenance process simpler is to statically compile your image's tor binary on an offsite, trusted computer. If you do this, you should no longer need to bother with chrooting your tor processes or copying libraries around. However, it still does not save you from the need to recompile that binary whenever there is a security update to the underlying libraries, and it may come at a cost of exploit resistance due to the loss of per-library ASLR. Ok, that's it. What do people think? Personally, I think that if we can require a kernel exploit and/or weird memory gymnastics for key compromise, that would be a *huge* improvement. Do the above recommendations actually accomplish that? If so, should we work on providing scripts to make the loopback filesystem creation process easier, and/or provide loopback images themselves? Even the APT defenses end up not working out, I would sleep a lot better at night if most relays deployed only the defenses to one-time key theft... Thoughts on that? -- Mike Perry

3 6