Greetings Tor relay operators,
First of all, if you operate non-Linux Tor relays then good job! Thanks! This notice isn't for you. Only Linux is affected.
This morning I scanned the tor network for vulnerability to CVE-2016-5696 [1], again so that I could compare the results with the scan from last month. Linux implemented RFC 5961 [2] using a global counter which can be used as a side channel allowing blind TCP injection and other consequences as well.
I used this collector consensus file for my scan: https://collector.torproject.org/recent/relay-descriptors/consensuses/2016-1...
That's 7080 relays. The scan produced:
246 relays had connection timeouts and were not scanned. 60 relays had connection refused. 6795 relays scanned. 2034 relays are vulnerable according to the rationale I used a month ago to categorized the results as vulnerable (if the probe response receives between 90 and 220 Challenge ACKs).
These results should be reproducible. The source code to the scanner is here: https://github.com/david415/scan_tor_rfc5961
Last month's scan showed 4069 vulnerable relays. Each probe does a TCP handshake and then sends 500 in-window packets and counts the resulting Challenge ACKs. (followed by a TCP connection close). However we've updated the buckets of probe results to this:
"r == 0" : "zero challenge ACKs", "0 < r <= 10" : "probably new kernel", "10 < r <= 20" : "maybe new kernel?", "20 < r <= 90" : "inconclusive?", "90 < r < 100" : "probably vulnerable", "(r%100)==0 and 0<r<500" : "almost definitely vulnerable", "(r%100)!=0 and 100<r<450" : "probably vulnerable", "(r%100)!=0 and 450<=r<=490" : "inconclusive?", "490 < r < 500" : "most likely applied sysctl workaround", "r == 500" : "probably applied sysctl workaround", "500 < r" : "extra duplicate packets?",
The researchers who discovered this vulnerability published an excellent paper [3]. Although they incorrectly described an attack on tor's path selection it may be possible that other attacks on the Tor network are possible. I think it would be better to upgrade and reboot your relays rather than risk the Tor network's exposure to attacks that may utilize CVE-2016-5696 in combination with other techniques. Furthermore being vulnerable to CVE-2016-5696 also implies that you have a Linux kernel which is also vulnerable to dirty COW; https://dirtycow.ninja
If you operate a relay you can check to see if your relay's TCP stack has counted many TCP Challenge ACKs. Please let us know if your counters have very high values.
Vulnerable Linux kernels will have snmp stat counters like this::
root:/# netstat -s | grep -i challenge TCPChallengeACK: 113338 TCPSYNChallenge: 1157
High counter values maybe indicative of attacks. Linux 4.7.x has got a per socket rate limiting feature which if it has a high value may also be indicative of attack attempts and probing::
root:/# netstat -s | grep -i challenge TCPChallengeACK: 2614 TCPSYNChallenge: 442 TCPACKSkippedChallenge: 141795
If you are too lazy to upgrade your Linux kernel and reboot then you may apply a convenient workaround:
sysctl net.ipv4.tcp_challenge_ack_limit=9999999999
read more about this workaround here:
https://blogs.akamai.com/2016/08/vulnerability-in-the-linux-kernels-tcp-stac...
Special thanks to Least Authority for supporting my research.
sincerely,
David Stainton
1. https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-5696
2. https://tools.ietf.org/html/rfc5961
3. Off-Path TCP Exploits: Global Rate Limit Considered Dangerous http://www.cs.ucr.edu/~zhiyunq/pub/sec16_TCP_pure_offpath.pdf
Hi.
I added the scan output to the repo, this includes the output csv file and a list of vulnerable relays:
https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_... https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
Upgrade your Linux kernel and reboot your tor relays!
Cheers, David
Hi David, Thanks for the heads up! It turns out that my relay is in the list of affected hosts, however, the kernel I was running (3.16.36-1+deb8u1) is claimed by Debian to be fixed (see: https://security-tracker.debian.org/tracker/CVE-2016-5696).
Since your script determines whether the host is affected or not based on the actual TCP comms (rather than banner grabbing a kernel version or something), I'm not sure what to make of that - it would seem to indicate that either the weighting you've devised doesn't fit Debian hosts, or it could indicate perhaps that the patch Debian maintainers applied to address the issue wasn't sufficient. I won't pretend to be clueful enough about low-level TCP stack programming to be able to tell for sure which is the case, but wanted to mention it in case others see the same thing.
For my part, I've since updated the kernel on my relay to 3.16.36-1+deb8u2, and applied the sysctl work-around as an additional measure. I checked the ACK count using netstat both before and after, and have included those results here:
Before: TCPChallengeACK: 1107 TCPSYNChallenge: 7
After: TCPChallengeACK: 2 TCPSYNChallenge: 2
Thanks!
-- Jason
On Thu, Nov 17, 2016 at 2:30 AM, dawuud dawuud@riseup.net wrote:
Hi.
I added the scan output to the repo, this includes the output csv file and a list of vulnerable relays:
https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_... https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
Upgrade your Linux kernel and reboot your tor relays!
Cheers, David
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
My relays have been patched to the latest available kernels, and aren't in the list of vulnerable relays, however they still show high values for TCPSYNChallenge:
----
$ ansible tor -a 'bash -c "netstat -s | grep -i challenge"' -b --ask-become-pass
lon | SUCCESS | rc=0 >> TCPChallengeACK: 14197 TCPSYNChallenge: 2926
fra | SUCCESS | rc=0 >> TCPChallengeACK: 12907 TCPSYNChallenge: 3461
----
$ ansible tor -a 'bash -c "cat /etc/lsb-release && uname -rv"' -b --ask-become-pass
fra | SUCCESS | rc=0 >> DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016
lon | SUCCESS | rc=0 >> DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016
----
On a Debian 8 updated relay too : # netstat -s | grep -i challenge TCPChallengeACK: 19497 TCPSYNChallenge: 12991
Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
Something else for being sure ?
Le 17/11/2016 à 20:24, Univibe a écrit :
$ ansible tor -a 'bash -c "netstat -s | grep -i challenge"' -b --ask-become-pass
lon | SUCCESS | rc=0 >> TCPChallengeACK: 14197 TCPSYNChallenge: 2926
fra | SUCCESS | rc=0 >> TCPChallengeACK: 12907 TCPSYNChallenge: 3461
On a Raspberry pi... Linux 4.4.26+ #915 Thu Oct 20 17:02:14 BST 2016 armv6l GNU/Linux
$ netstat -s | grep -i challenge TCPChallengeACK: 10
(no TCPSYNChallenge result ??)
Le 17/11/2016 à 20:24, Univibe a écrit :
My relays have been patched to the latest available kernels, and aren't in the list of vulnerable relays, however they still show high values for TCPSYNChallenge:
$ ansible tor -a 'bash -c "netstat -s | grep -i challenge"' -b --ask-become-pass
lon | SUCCESS | rc=0 >> TCPChallengeACK: 14197 TCPSYNChallenge: 2926
fra | SUCCESS | rc=0 >> TCPChallengeACK: 12907 TCPSYNChallenge: 3461
$ ansible tor -a 'bash -c "cat /etc/lsb-release && uname -rv"' -b --ask-become-pass
fra | SUCCESS | rc=0 >> DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016
lon | SUCCESS | rc=0 >> DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi Jason,
Thanks for your observation. I'll try to investigate soon.
Cheers,
David
On Thu, Nov 17, 2016 at 12:02:05PM -0500, Jason Ross wrote:
Hi David, Thanks for the heads up! It turns out that my relay is in the list of affected hosts, however, the kernel I was running (3.16.36-1+deb8u1) is claimed by Debian to be fixed (see: https://security-tracker.debian.org/tracker/CVE-2016-5696).
Since your script determines whether the host is affected or not based on the actual TCP comms (rather than banner grabbing a kernel version or something), I'm not sure what to make of that - it would seem to indicate that either the weighting you've devised doesn't fit Debian hosts, or it could indicate perhaps that the patch Debian maintainers applied to address the issue wasn't sufficient. I won't pretend to be clueful enough about low-level TCP stack programming to be able to tell for sure which is the case, but wanted to mention it in case others see the same thing.
For my part, I've since updated the kernel on my relay to 3.16.36-1+deb8u2, and applied the sysctl work-around as an additional measure. I checked the ACK count using netstat both before and after, and have included those results here:
Before: TCPChallengeACK: 1107 TCPSYNChallenge: 7
After: TCPChallengeACK: 2 TCPSYNChallenge: 2
Thanks!
-- Jason
On Thu, Nov 17, 2016 at 2:30 AM, dawuud dawuud@riseup.net wrote:
Hi.
I added the scan output to the repo, this includes the output csv file and a list of vulnerable relays:
https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_... https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
Upgrade your Linux kernel and reboot your tor relays!
Cheers, David
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi David,
Thanks for your work!
dawuud:
I added the scan output to the repo, this includes the output csv file and a list of vulnerable relays:
https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_... https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
FYI, I produced results with platform strings and fingerprints based on this data [1].
It's pretty interesting that there are not only Linux relays are 'vulnerable' (90 < ChACKs < 220) in David's scan: % cat combined_results.csv | grep -v notvulnerable | grep -v Linux | grep Tor
Tor 0.2.8.9 on NetBSD,3F5440FF003DFF8A12AA308CFD4087FBC157ABE0,78.47.45.36:9001,1.08132791519,500,142,vulnerable Tor 0.2.5.10 on NetBSD,508004552343E5374B6570C76E9239AA23310684,86.62.117.171:63500,1.00646305084,500,103,vulnerable Tor 0.2.8.9 on NetBSD,8806C3E6FA42B07113F3A1553DE70C0A30101201,139.18.25.35:9001,1.02995896339,500,113,vulnerable Tor 0.2.7.6 on FreeBSD,9C5461498004325F87C0685BDA5DA99AC5335314,62.194.144.196:9001,1.06730103493,500,211,vulnerable Tor 0.2.8.9 on FreeBSD,BCFE548EA3FF8A0B3610779C238350124A8ED6DE,207.172.209.83:9001,1.06568193436,500,214,vulnerable Tor 0.2.7.6 on NetBSD,F88C4D522EE7BD8B18B6C6418B8548E6E6BC74E9,195.43.138.226:9001,0.994502782822,500,100,vulnerable
After I've rescanned these relays myself for several times, FreeBSD ones stopped being 'vulnereable' while NetBSD ones somehow still reproduce 'vulnerable' Linux status.
I don't know why does this happen, maybe someone can scan these relays (or maybe all NetBSD ones due to TCP stack specifics) themselves and get different results. Anyway these are just curious false positives.
[1] https://github.com/nogoegst/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
-- Ivan Markin
Hi all,
I'm sorry that there are some false positives. I did previously test against a FreeBSD tor relay and presumed NetBSD would have a similar result.
Thanks for looking closely at this Ivan. It sounds like the scanner needs to be fixed. I'll try to test with a netbsd host soon.
Cheers!
David
On Thu, Nov 17, 2016 at 07:46:00PM +0000, Ivan Markin wrote:
Hi David,
Thanks for your work!
dawuud:
I added the scan output to the repo, this includes the output csv file and a list of vulnerable relays:
https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_... https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
FYI, I produced results with platform strings and fingerprints based on this data [1].
It's pretty interesting that there are not only Linux relays are 'vulnerable' (90 < ChACKs < 220) in David's scan: % cat combined_results.csv | grep -v notvulnerable | grep -v Linux | grep Tor
Tor 0.2.8.9 on NetBSD,3F5440FF003DFF8A12AA308CFD4087FBC157ABE0,78.47.45.36:9001,1.08132791519,500,142,vulnerable Tor 0.2.5.10 on NetBSD,508004552343E5374B6570C76E9239AA23310684,86.62.117.171:63500,1.00646305084,500,103,vulnerable Tor 0.2.8.9 on NetBSD,8806C3E6FA42B07113F3A1553DE70C0A30101201,139.18.25.35:9001,1.02995896339,500,113,vulnerable Tor 0.2.7.6 on FreeBSD,9C5461498004325F87C0685BDA5DA99AC5335314,62.194.144.196:9001,1.06730103493,500,211,vulnerable Tor 0.2.8.9 on FreeBSD,BCFE548EA3FF8A0B3610779C238350124A8ED6DE,207.172.209.83:9001,1.06568193436,500,214,vulnerable Tor 0.2.7.6 on NetBSD,F88C4D522EE7BD8B18B6C6418B8548E6E6BC74E9,195.43.138.226:9001,0.994502782822,500,100,vulnerable
After I've rescanned these relays myself for several times, FreeBSD ones stopped being 'vulnereable' while NetBSD ones somehow still reproduce 'vulnerable' Linux status.
I don't know why does this happen, maybe someone can scan these relays (or maybe all NetBSD ones due to TCP stack specifics) themselves and get different results. Anyway these are just curious false positives.
[1] https://github.com/nogoegst/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
-- Ivan Markin _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hello,
I think the best approach for elliminating the false positives would be to make the scanner perform the timing inference attack as described in the paper.
Unfortunately I don't have enough time to look into this more.
Cheers, David
On Thu, Nov 17, 2016 at 09:22:47PM +0000, dawuud wrote:
Hi all,
I'm sorry that there are some false positives. I did previously test against a FreeBSD tor relay and presumed NetBSD would have a similar result.
Thanks for looking closely at this Ivan. It sounds like the scanner needs to be fixed. I'll try to test with a netbsd host soon.
Cheers!
David
On Thu, Nov 17, 2016 at 07:46:00PM +0000, Ivan Markin wrote:
Hi David,
Thanks for your work!
dawuud:
I added the scan output to the repo, this includes the output csv file and a list of vulnerable relays:
https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_... https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
FYI, I produced results with platform strings and fingerprints based on this data [1].
It's pretty interesting that there are not only Linux relays are 'vulnerable' (90 < ChACKs < 220) in David's scan: % cat combined_results.csv | grep -v notvulnerable | grep -v Linux | grep Tor
Tor 0.2.8.9 on NetBSD,3F5440FF003DFF8A12AA308CFD4087FBC157ABE0,78.47.45.36:9001,1.08132791519,500,142,vulnerable Tor 0.2.5.10 on NetBSD,508004552343E5374B6570C76E9239AA23310684,86.62.117.171:63500,1.00646305084,500,103,vulnerable Tor 0.2.8.9 on NetBSD,8806C3E6FA42B07113F3A1553DE70C0A30101201,139.18.25.35:9001,1.02995896339,500,113,vulnerable Tor 0.2.7.6 on FreeBSD,9C5461498004325F87C0685BDA5DA99AC5335314,62.194.144.196:9001,1.06730103493,500,211,vulnerable Tor 0.2.8.9 on FreeBSD,BCFE548EA3FF8A0B3610779C238350124A8ED6DE,207.172.209.83:9001,1.06568193436,500,214,vulnerable Tor 0.2.7.6 on NetBSD,F88C4D522EE7BD8B18B6C6418B8548E6E6BC74E9,195.43.138.226:9001,0.994502782822,500,100,vulnerable
After I've rescanned these relays myself for several times, FreeBSD ones stopped being 'vulnereable' while NetBSD ones somehow still reproduce 'vulnerable' Linux status.
I don't know why does this happen, maybe someone can scan these relays (or maybe all NetBSD ones due to TCP stack specifics) themselves and get different results. Anyway these are just curious false positives.
[1] https://github.com/nogoegst/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
-- Ivan Markin _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi tor-relays@,
Getting back with more results on this. I've implemented CVE-2016-5696 scanner in Go [1] and scanned the Tor network several times [2]. First results I've got using technique similar to David's (sending 500 RSTs in one burst), second ones are got via another method (send 111 RSTs in burst and then 111 RSTs 1 second later*).
Current statistics: 32% of Linux relays are vulnerable. That is 23% of Tor network.
--
Now some magic! Those 3 NetBSD relays from before still behave like they are vulnerable Linuxes (as they did in David's scanner, and two of mine):
$ cat grill-tor-2016-12-09 | grep -v Linux | grep vulnerable 78.47.45.36:9001,3F5440FF003DFF8A12AA308CFD4087FBC157ABE0,Tor 0.2.8.9 on NetBSD,200,1.847787ms,1.834238ms,vulnerable 86.62.117.171:63500,508004552343E5374B6570C76E9239AA23310684,Tor 0.2.5.10 on NetBSD,200,1.999138ms,1.839057ms,vulnerable 139.18.25.35:9001,8806C3E6FA42B07113F3A1553DE70C0A30101201,Tor 0.2.8.9 on NetBSD,200,3.936046ms,3.777501ms,vulnerable
Yes, nmap -O reports them to be NetBSD hosts.
Actually I don't know what's going on here. Thoughts: * relays are behind vulnerable Linux middleboxes * RFC 5961 got implemented partly in NetBSD and it is actually vulnerable * ???
Okay then. I've brought up NetBSD 7.0.2 VM and scanned it locally. 0 challenge ACKs. Fine. I've put it under vulnerable Linux DNAT and it was 'kinda' vulnerable (some small random amount of ChACKs). Probably I did something wrong here. I headed out and scanned netbsd.org (self-hosted?) and it's vulnerable also.
I've lurked through NetBSD's src code and found some bits of RFC5961. But I was unable to see anything offensive.
If someone have some insight on this dark magic, that would be awesome!
---
Thanks for bringing up the diversity issue in light of this CVE, Alex! Just to make everyone feel sad today:
$ cat grill-tor-2016-12-09 | grep -v offline | grep Linux | wc -l 6435 $ cat grill-tor-2016-12-09 | grep -v offline | grep -v Linux | wc -l 550
Sadly, Linuxes are typical ~2σ of the network. ;( Please run more different (e.g. BSD) relays!
[*] I think it should be more accurate. [1] https://github.com/nogoegst/grill [2] https://gist.github.com/nogoegst/d2de330b794b47158b4cfbed0987b4de
-- Happy life without suffering, Ivan Markin
On 9 Dec. 2016, at 16:31, Ivan Markin twim@riseup.net wrote:
Hi tor-relays@,
Getting back with more results on this. I've implemented CVE-2016-5696 scanner in Go [1] and scanned the Tor network several times [2]. First results I've got using technique similar to David's (sending 500 RSTs in one burst), second ones are got via another method (send 111 RSTs in burst and then 111 RSTs 1 second later*).
Current statistics: 32% of Linux relays are vulnerable. That is 23% of Tor network.
--
Now some magic! Those 3 NetBSD relays from before still behave like they are vulnerable Linuxes (as they did in David's scanner, and two of mine):
$ cat grill-tor-2016-12-09 | grep -v Linux | grep vulnerable 78.47.45.36:9001,3F5440FF003DFF8A12AA308CFD4087FBC157ABE0,Tor 0.2.8.9 on NetBSD,200,1.847787ms,1.834238ms,vulnerable 86.62.117.171:63500,508004552343E5374B6570C76E9239AA23310684,Tor 0.2.5.10 on NetBSD,200,1.999138ms,1.839057ms,vulnerable 139.18.25.35:9001,8806C3E6FA42B07113F3A1553DE70C0A30101201,Tor 0.2.8.9 on NetBSD,200,3.936046ms,3.777501ms,vulnerable
Yes, nmap -O reports them to be NetBSD hosts.
Actually I don't know what's going on here. Thoughts:
- relays are behind vulnerable Linux middleboxes
- RFC 5961 got implemented partly in NetBSD and it is actually vulnerable
- ???
Okay then. I've brought up NetBSD 7.0.2 VM and scanned it locally. 0 challenge ACKs. Fine. I've put it under vulnerable Linux DNAT and it was 'kinda' vulnerable (some small random amount of ChACKs). Probably I did something wrong here. I headed out and scanned netbsd.org (self-hosted?) and it's vulnerable also.
I've lurked through NetBSD's src code and found some bits of RFC5961. But I was unable to see anything offensive.
If someone have some insight on this dark magic, that would be awesome!
Thanks for bringing up the diversity issue in light of this CVE, Alex! Just to make everyone feel sad today:
$ cat grill-tor-2016-12-09 | grep -v offline | grep Linux | wc -l 6435 $ cat grill-tor-2016-12-09 | grep -v offline | grep -v Linux | wc -l 550
Sadly, Linuxes are typical ~2σ of the network. ;( Please run more different (e.g. BSD) relays!
[*] I think it should be more accurate. [1] https://github.com/nogoegst/grill [2] https://gist.github.com/nogoegst/d2de330b794b47158b4cfbed0987b4de
Hi Ivan,
Thanks for doing this work, and the reminder to upgrade (or install a non-Linux OS).
For Tor client path selection, it is typically the vulnerable consensus weight that matters, not the number of relays. (Except in the case of HSDirs, where the hash ring is unweighted.)
Have you looked at the vulnerable consensus weight proportion?
T
teor:
For Tor client path selection, it is typically the vulnerable consensus weight that matters, not the number of relays. (Except in the case of HSDirs, where the hash ring is unweighted.)
Have you looked at the vulnerable consensus weight proportion?
Thanks for the tip! Laziness just took over me then.
It turns out that vulnerable consensus weight fraction is 0.249602 or 25%.
-- Ivan Markin
4 server rebooted, thank you very much.
markus
On 9 Dec 2016, at 06:31, Ivan Markin twim@riseup.net wrote:
Hi tor-relays@,
Getting back with more results on this. I've implemented CVE-2016-5696 scanner in Go [1] and scanned the Tor network several times [2]. First results I've got using technique similar to David's (sending 500 RSTs in one burst), second ones are got via another method (send 111 RSTs in burst and then 111 RSTs 1 second later*).
Current statistics: 32% of Linux relays are vulnerable. That is 23% of Tor network.
--
Now some magic! Those 3 NetBSD relays from before still behave like they are vulnerable Linuxes (as they did in David's scanner, and two of mine):
$ cat grill-tor-2016-12-09 | grep -v Linux | grep vulnerable 78.47.45.36:9001,3F5440FF003DFF8A12AA308CFD4087FBC157ABE0,Tor 0.2.8.9 on NetBSD,200,1.847787ms,1.834238ms,vulnerable 86.62.117.171:63500,508004552343E5374B6570C76E9239AA23310684,Tor 0.2.5.10 on NetBSD,200,1.999138ms,1.839057ms,vulnerable 139.18.25.35:9001,8806C3E6FA42B07113F3A1553DE70C0A30101201,Tor 0.2.8.9 on NetBSD,200,3.936046ms,3.777501ms,vulnerable
Yes, nmap -O reports them to be NetBSD hosts.
Actually I don't know what's going on here. Thoughts:
- relays are behind vulnerable Linux middleboxes
- RFC 5961 got implemented partly in NetBSD and it is actually vulnerable
- ???
Okay then. I've brought up NetBSD 7.0.2 VM and scanned it locally. 0 challenge ACKs. Fine. I've put it under vulnerable Linux DNAT and it was 'kinda' vulnerable (some small random amount of ChACKs). Probably I did something wrong here. I headed out and scanned netbsd.org (self-hosted?) and it's vulnerable also.
I've lurked through NetBSD's src code and found some bits of RFC5961. But I was unable to see anything offensive.
If someone have some insight on this dark magic, that would be awesome!
Thanks for bringing up the diversity issue in light of this CVE, Alex! Just to make everyone feel sad today:
$ cat grill-tor-2016-12-09 | grep -v offline | grep Linux | wc -l 6435 $ cat grill-tor-2016-12-09 | grep -v offline | grep -v Linux | wc -l 550
Sadly, Linuxes are typical ~2σ of the network. ;( Please run more different (e.g. BSD) relays!
[*] I think it should be more accurate. [1] https://github.com/nogoegst/grill [2] https://gist.github.com/nogoegst/d2de330b794b47158b4cfbed0987b4de
-- Happy life without suffering, Ivan Markin _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi Ivan and tor-relay operators,
The Golang rewrite of the scanner is cool!
btw i'm surprised you wrote https://github.com/nogoegst/rough/blob/master/tcp.go instead of using https://github.com/google/gopacket
Maybe you could also implement my Tor guard discovery attack that uses this vulnerability?
I've been asked to write a proof of concept but I don't feel motivated to do so. Also, there are some doubts about weather this guard discovery attack would be feasible on the real Tor network... though we could probably make it work in a test network.
Now that such a small percentage of the Tor network is vulnerable it's probably safe/responsible for me to post my theoretic Tor guard discovery attack, right?
Sincerely,
David
On Fri, Dec 09, 2016 at 05:31:00AM +0000, Ivan Markin wrote:
Hi tor-relays@,
Getting back with more results on this. I've implemented CVE-2016-5696 scanner in Go [1] and scanned the Tor network several times [2]. First results I've got using technique similar to David's (sending 500 RSTs in one burst), second ones are got via another method (send 111 RSTs in burst and then 111 RSTs 1 second later*).
Current statistics: 32% of Linux relays are vulnerable. That is 23% of Tor network.
--
Now some magic! Those 3 NetBSD relays from before still behave like they are vulnerable Linuxes (as they did in David's scanner, and two of mine):
$ cat grill-tor-2016-12-09 | grep -v Linux | grep vulnerable 78.47.45.36:9001,3F5440FF003DFF8A12AA308CFD4087FBC157ABE0,Tor 0.2.8.9 on NetBSD,200,1.847787ms,1.834238ms,vulnerable 86.62.117.171:63500,508004552343E5374B6570C76E9239AA23310684,Tor 0.2.5.10 on NetBSD,200,1.999138ms,1.839057ms,vulnerable 139.18.25.35:9001,8806C3E6FA42B07113F3A1553DE70C0A30101201,Tor 0.2.8.9 on NetBSD,200,3.936046ms,3.777501ms,vulnerable
Yes, nmap -O reports them to be NetBSD hosts.
Actually I don't know what's going on here. Thoughts:
- relays are behind vulnerable Linux middleboxes
- RFC 5961 got implemented partly in NetBSD and it is actually vulnerable
- ???
Okay then. I've brought up NetBSD 7.0.2 VM and scanned it locally. 0 challenge ACKs. Fine. I've put it under vulnerable Linux DNAT and it was 'kinda' vulnerable (some small random amount of ChACKs). Probably I did something wrong here. I headed out and scanned netbsd.org (self-hosted?) and it's vulnerable also.
I've lurked through NetBSD's src code and found some bits of RFC5961. But I was unable to see anything offensive.
If someone have some insight on this dark magic, that would be awesome!
Thanks for bringing up the diversity issue in light of this CVE, Alex! Just to make everyone feel sad today:
$ cat grill-tor-2016-12-09 | grep -v offline | grep Linux | wc -l 6435 $ cat grill-tor-2016-12-09 | grep -v offline | grep -v Linux | wc -l 550
Sadly, Linuxes are typical ~2σ of the network. ;( Please run more different (e.g. BSD) relays!
[*] I think it should be more accurate. [1] https://github.com/nogoegst/grill [2] https://gist.github.com/nogoegst/d2de330b794b47158b4cfbed0987b4de
-- Happy life without suffering, Ivan Markin _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
dawuud:
The Golang rewrite of the scanner is cool!
Thanks!
btw i'm surprised you wrote https://github.com/nogoegst/rough/blob/master/tcp.go instead of using https://github.com/google/gopacket
You shouldn't; rough is just a convenient wrapper on top of TCP-ish stuff from gopacket (it makes TCP hacks simpler).
Maybe you could also implement my Tor guard discovery attack that uses this vulnerability?
Why not. I just don't know what the attack is. Can you point me to it?
I've been asked to write a proof of concept but I don't feel motivated to do so. Also, there are some doubts about weather this guard discovery attack would be feasible on the real Tor network... though we could probably make it work in a test network.
Now that such a small percentage of the Tor network is vulnerable it's probably safe/responsible for me to post my theoretic Tor guard discovery attack, right?
Hmm, I *don't* think that 1/4 of the network is actually small percentage... [I think we should somehow encourage vulnerable relays to update their kernels to lower affected percentage below ~10-15%.] Also, you saying "guard discovery attack based on pure off-path TCP attack" make this *slightly* obvious. So if someone actually got it, it's likely that they're already exploiting it.
-- Ivan Markin
btw i'm surprised you wrote https://github.com/nogoegst/rough/blob/master/tcp.go instead of using https://github.com/google/gopacket
You shouldn't; rough is just a convenient wrapper on top of TCP-ish stuff from gopacket (it makes TCP hacks simpler).
ah right. cool.
Maybe you could also implement my Tor guard discovery attack that uses this vulnerability?
Why not. I just don't know what the attack is. Can you point me to it?
On second thought I guess we better stick to writing scanners because if we start writing exploits then eventually some script kitty will come along and try to attack the Tor network with it; and even though my attack might not work it involves doing various things that utilize resources on the Tor network; so it would be bad for the health of the Tor network.
I've been asked to write a proof of concept but I don't feel motivated to do so. Also, there are some doubts about weather this guard discovery attack would be feasible on the real Tor network... though we could probably make it work in a test network.
Now that such a small percentage of the Tor network is vulnerable it's probably safe/responsible for me to post my theoretic Tor guard discovery attack, right?
Hmm, I *don't* think that 1/4 of the network is actually small percentage... [I think we should somehow encourage vulnerable relays to update their kernels to lower affected percentage below ~10-15%.]
Also, you saying "guard discovery attack based on pure off-path TCP attack" make this *slightly* obvious. So if someone actually got it, it's likely that they're already exploiting it.
It's traffic profile would be obviously identifiable for passive network observers. A nation state actor would have much better/faster results using other well known publicly documented Tor guard discovery attacks. Pretty sure they like to be sneaky when they deanonymize Tor circuits.
I would however be very interested to hear back from tor-relay operators if any of them have found Challenge ACK counter values higher than a million... which would indicate some kind of funny business.
Cheers, David
dawuud:
Maybe you could also implement my Tor guard discovery attack that uses this vulnerability?
Why not. I just don't know what the attack is. Can you point me to it?
On second thought I guess we better stick to writing scanners because if we start writing exploits then eventually some script kitty will come along and try to attack the Tor network with it; and even though my attack might not work it involves doing various things that utilize resources on the Tor network; so it would be bad for the health of the Tor network.
Right, I didn't mean that to be an exploit but just a PoC of this attack vector. But you're right, I haven't thought that it will put load on the network, and doing this is definitely not OK. It's not just some harmless TCP segments, there is much more than this (circuit rebuilding, etc).
It's traffic profile would be obviously identifiable for passive network observers. A nation state actor would have much better/faster results using other well known publicly documented Tor guard discovery attacks. Pretty sure they like to be sneaky when they deanonymize Tor circuits.
I doesn't mean that nobody would like to use it. There are attackers that use botnets to do their nasty business and they don't care much about how visible it is.
I would however be very interested to hear back from tor-relay operators if any of them have found Challenge ACK counter values higher than a million... which would indicate some kind of funny business.
It may not indicate this. Since I was able to scan whole Tor network in just 7 minutes (someone can use more than 127 concurrent scans and scan even faster), it may indicate that there is some aggressive scanning is going on by multiple parties.
-- Ivan Markin
I would however be very interested to hear back from tor-relay operators if any of them have found Challenge ACK counter values higher than a million... which would indicate some kind of funny business.
Thanky you for your work.
I know of 3 relays with ACK above 1 million:
TCPChallengeACK: 1081146 TCPSYNChallenge: 1062995
TCPChallengeACK: 1270948 TCPSYNChallenge: 1254428
TCPChallengeACK: 1189549 TCPSYNChallenge: 1171422
all running under Linux vm20198 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
There seems to be no relation between uptime of the server and challenges apart from rebooting, which resets to 0.
What about relays not on the list at all?
I would assume that not everybody of that 23 percent does know what exactly to do, apart from better running on BSD - could you please give detailed recommendation for beginners - your discussion seems on a high level :-)
Thanks and regards
Paul
pa011:
What about relays not on the list at all?
You mean that are not subscribed for tor-relays@?
btw, it would be awesome to give away t-shirts or something for running diverse relays.
I would assume that not everybody of that 23 percent does know what exactly to do, apart from better running on BSD - could you please give detailed recommendation for beginners - your discussion seems on a high level :-)
Agree, I personally don't see any way to notify these operators about what to do (except clear instructions at blog.tpo or tor-relays@). With pleasure. There is an awesome The Tor BSD Diversity Project. The instructions for BSD beginners can be found here [1].
[1] https://torbsd.github.io/relay-guides.html -- Ivan Markin
Am 10.12.2016 um 21:12 schrieb Ivan Markin:
pa011:
What about relays not on the list at all?
You mean that are not subscribed for tor-relays@?
No, forget that one - was my mistakable in the spreadsheet
btw, it would be awesome to give away t-shirts or something for running diverse relays.
that was a least a promise the year ago (its not any more)- and I believe one should stand to his promises
I would assume that not everybody of that 23 percent does know what exactly to do, apart from better running on BSD - could you please give detailed recommendation for beginners - your discussion seems on a high level :-)
Agree, I personally don't see any way to notify these operators about what to do (except clear instructions at blog.tpo or tor-relays@). With pleasure.
Yes please spread this out - as simple as possible - the list has about 1700 hundret subscribers I, if I am correct - I reckon the important and interested ones are on it
There is an awesome The Tor BSD Diversity Project. The
instructions for BSD beginners can be found here [1].
I used them about a week ago - they were the best I could find - sure they have room for improvement, especially for beginners, but there are several nice people out there -glad to help - some awful question have me done here by me already
Could you give some explanation please on the difference between:
-lots of challenge ACKs -multiple challenge ACKs -one challenge ACK -two challenge ACKs -vulnerable -zero challenge
Thanks Paul
pa011:
Could you give some explanation please on the difference between:
-lots of challenge ACKs
received exactly the same number of chacks as number of sent RSTs (fixed kernel, sysctl workaround, ...)
-one challenge ACK
received just one chack during this connection
-two challenge ACKs
received one chack after first RST burst, another one after second burst
-vulnerable
100chacks/s rate limit was hit twice
-zero challenge
RFC5961 is not supported
-multiple challenge ACKs
anything else, i.e. there are some random number of chacks received but less than number of sent RSTs, probably rate-limited
Current (these) definitions are here [1]. But they are a subject of change, because I'm trying to improve scanning method (separating counters for each of bursts).
[1] https://github.com/nogoegst/grill/blob/master/verdict/verdict.go -- Ivan Markin
On 12/10/2016 09:52 PM, pa011 wrote:
btw, it would be awesome to give away t-shirts or something for running diverse relays.
that was a least a promise the year ago (its not any more)
- and I believe one should stand to his promises
I don't know where this idea is coming from that it's not any more the case. I have not heard of any changes in that respect, the instructions are still the same: https://www.torproject.org/getinvolved/tshirt.html
You need to email tshirts@, include your fingerprints, and then wait weeks or months until the one poor person that is processing all the requests and all the requests from the campaign and also has other hats to get to it.
It's a thank you gift. I don't mean you, pa011, when I say this, but in general I would expect a bit more gratitude and understanding what it means to run a small understaffed organization. :(
Moritz
On 2016-12-11 at 22:50, Moritz Bartl wrote:
You need to email tshirts@, include your fingerprints, and then wait weeks or months until the one poor person that is processing all the requests and all the requests from the campaign and also has other hats to get to it.
It's a thank you gift. I don't mean you, pa011, when I say this, but in general I would expect a bit more gratitude and understanding what it means to run a small understaffed organization. :(
Moritz
Hi Moritz,
I do understand that it's hard to run an organization with too few people, it's my daily life working for staff at my university (I am the only administrator for 16 tablets, 34 laptops, 3 servers and 7 thinclients, and we are not allowed to use centralized tools, I have to administrate all of these devices manually), so there are definitely all my "thank you"s I can give.
But I, for example, didn't even get an answer first, I would even love to get an automated email saying "Sorry if it takes a very long time, we are overwhelmed by the work we currently have to do" (waiting for almost 7 months now by the way). After 2-3 months I got an answer when we talked about that topic via this list in summer (though nothing official besides the mailing list talk). The problem is that only one person handles the shirts and it's ok even if I have to wait another 7 months or so (it's a gift at all, as you already said).
It's just that you/we/they should change something about the handling of the situation. It was (and for others most probably still is) the lack of communication that frustrates eligible relay operators so much.
But to conclude, thanks for all the work towards Tor and everything. Everybody has to give his work and support, so we can stand for free Internet (free as in freedom, not as in free beer). This was not meant to be against you, it was only in response to your mail because you got to this topic :)
Best, Michael
Hi Moritz,
I do understand that it's hard to run an organization with too few people, it's my daily life working for staff at my university (I am the only administrator for 16 tablets, 34 laptops, 3 servers and 7 thinclients, and we are not allowed to use centralized tools, I have to administrate all of these devices manually), so there are definitely all my "thank you"s I can give.
But I, for example, didn't even get an answer first, I would even love to get an automated email saying "Sorry if it takes a very long time, we are overwhelmed by the work we currently have to do" (waiting for almost 7 months now by the way). After 2-3 months I got an answer when we talked about that topic via this list in summer (though nothing official besides the mailing list talk). The problem is that only one person handles the shirts and it's ok even if I have to wait another 7 months or so (it's a gift at all, as you already said).
It's just that you/we/they should change something about the handling of the situation. It was (and for others most probably still is) the lack of communication that frustrates eligible relay operators so much.
But to conclude, thanks for all the work towards Tor and everything. Everybody has to give his work and support, so we can stand for free Internet (free as in freedom, not as in free beer). This was not meant to be against you, it was only in response to your mail because you got to this topic :)
Best, Michael
I agree with what you already expressed Michael.
On top of that I just want to remind on two mails back from June this year -obviously a time when the project was more focused on other issues - even there is and was volunteer help around:
Date: Mon, 13 Jun 2016 22:07:26 +0200
I would offer 2 helping hands and possibly more as well to get this and my own shirt out - please contact me
Paul
Am 08.06.2016 um 18:05 schrieb l3thal.injecti0n@gmail.com:
If tor weather isn't running, and tshirt emails aren't being sent out, is someone doing this manually then? How can I help get the show on the road? Not gonna lie, I was really looking forward to the tshirt email as my relay definitely should have earned one about 2 weeks ago. haha. Actually I just donated $100... maybe now I can get a tshirt?
https://atlas.torproject.org/#details/1F45542A24A61BF9408F1C05E0DCE4E29F2CBA... _______________________________________________
Playing devils advocate here.
People want the T-Shirts. Its free advertisement for Tor. Perhaps it would be great if people could get T-Shirts in exchange for other goods. Like this:
http://i.imgur.com/9Y8p7.gif http://i.imgur.com/9Y8p7.gif
A long time ago this was possible: https://www.torservers.net/wiki/tshirt/index#order https://www.torservers.net/wiki/tshirt/index#order
Perhaps we should do this again and #MAKETORGREATAGAIN
Markus
On 11 Dec 2016, at 22:50, Moritz Bartl moritz@torservers.net wrote: On 12/10/2016 09:52 PM, pa011 wrote:
btw, it would be awesome to give away t-shirts or something for running diverse relays.
that was a least a promise the year ago (its not any more)
- and I believe one should stand to his promises
I don't know where this idea is coming from that it's not any more the case. I have not heard of any changes in that respect, the instructions are still the same: https://www.torproject.org/getinvolved/tshirt.html
You need to email tshirts@, include your fingerprints, and then wait weeks or months until the one poor person that is processing all the requests and all the requests from the campaign and also has other hats to get to it.
It's a thank you gift. I don't mean you, pa011, when I say this, but in general I would expect a bit more gratitude and understanding what it means to run a small understaffed organization. :(
Moritz _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 12 Dec. 2016, at 09:46, niftybunny abuse@to-surf-and-protect.net wrote:
Playing devils advocate here.
People want the T-Shirts. Its free advertisement for Tor. Perhaps it would be great if people could get T-Shirts in exchange for other goods. Like this:
A long time ago this was possible: https://www.torservers.net/wiki/tshirt/index#order
Perhaps we should do this again and #MAKETORGREATAGAIN
Markus
It is possible to donate and get a t-shirt:
https://donate.torproject.org/
We learnt a lot from doing it last year, and we have plans to make it more efficient this year. (And get more people on it.)
We have already gone from having 0 paid people on it, to having 1 paid person on it (and they do many other tasks as well). I think we are getting more to help over the next few months.
This should hopefully help relay operators get t-shirts as well.
Get back to us in early March if it hasn't worked out by then, and I'll make sure I follow it up at the next Tor Meeting.
Tim
On 11 Dec 2016, at 22:50, Moritz Bartl moritz@torservers.net wrote: On 12/10/2016 09:52 PM, pa011 wrote:
btw, it would be awesome to give away t-shirts or something for running diverse relays.
that was a least a promise the year ago (its not any more)
- and I believe one should stand to his promises
I don't know where this idea is coming from that it's not any more the case. I have not heard of any changes in that respect, the instructions are still the same: https://www.torproject.org/getinvolved/tshirt.html
You need to email tshirts@, include your fingerprints, and then wait weeks or months until the one poor person that is processing all the requests and all the requests from the campaign and also has other hats to get to it.
It's a thank you gift. I don't mean you, pa011, when I say this, but in general I would expect a bit more gratitude and understanding what it means to run a small understaffed organization. :(
Moritz _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
T
Hi,
I don't really want to derail this thread, and I'm sure it's been said before, but I'd love to put in a plug for the TorBSD Diversity Project here:
As a whole, the network is pretty homogeneous in terms of its reliance on the Linux kernel so kernel bugs and exploits like this can end up affecting a great portion of the network.
There are some graphs showing (the lack of) network diversity here, which are interesting to look at:
http://torstatus.blutmagie.de/network_detail.php
Just a thought to consider when setting up future relays, for those who already have experience working with other kernels like FreeBSD/OpenBSD/Solaris etc.
Cheers,
Alex
On 17/11/16 07:30, dawuud wrote:
Hi.
I added the scan output to the repo, this includes the output csv file and a list of vulnerable relays:
https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_... https://github.com/david415/scan_tor_rfc5961/blob/master/scan_archive/nov17_...
Upgrade your Linux kernel and reboot your tor relays!
Cheers, David
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Am 19.11.2016 um 16:49 schrieb Alex Haydock:
There are some graphs showing (the lack of) network diversity here, which are interesting to look at:
yes, this chart displays routers' domain country codes
Another view on the blutmagie live database is showing the distribution over autonomous systems.
all routers https://torstatus.blutmagie.de/as-scoring.php?exit=0 exits only https://torstatus.blutmagie.de/as-scoring.php?exit=1
More than 10% of all exit network AS are under french control. The total number of routers is show at the table bottom.
regards Olaf
On 21/11/16 14:43, Olaf Selke wrote:
Am 19.11.2016 um 16:49 schrieb Alex Haydock:
There are some graphs showing (the lack of) network diversity here, which are interesting to look at:
yes, this chart displays routers' domain country codes
Further down the page is a chart displaying routers by OS family, which is the one I was referring to.
Though I do also agree that having a large number of relays running under just a few AS networks is a problem too.
Regards, Alex
tor-relays@lists.torproject.org
-
Alex Haydock -
dawuud -
Ivan Markin -
Jason Ross -
Michael Armbruster -
Moritz Bartl -
niftybunny -
Olaf Selke -
pa011 -
Petrusko -
teor -
Univibe