running Tor relay live with AddressSanitizer
Has anyone tried running a live relay with an image built using GCC 4.8 and -fsanitize=address? AddressSanitizer documentation says it should be no worse than about x4 on CPU and typically about x2, so it looks reasonable to try. I'm seeing peak CPU of about 7-8% of a 2.2GHz AMD core. I'd probably create a static build, with 'libevent' and 'tor' compiled using -fsanitize=address -O1 -fno-omit-frame-pointer -g per the Google documentation and 'libz' compiled normally since 'zlib' is stable and mature. Any thoughts or experience? Thanks
At 18:14 10/24/2013 -0400, starlight.2013q4@binnacle.cx wrote:
Has anyone tried running a live relay with an image built using GCC 4.8 and -fsanitize=address?
Took an initial jab at it by compiling just 'tor' with CFLAGS = -g -O1 -D_FORTIFY_SOURCE=2 -fstack-protector-all -Wstack-protector -fwrapv --param ssp-buffer-size=1 -fPIE -Wall -fno-strict-aliasing -fsanitize=address -fno-omit-frame-pointer where the modified or added flags are are -O1 -fsanitize=address -fno-omit-frame-pointer Fired it up and is looking good so far. CPU consumption quite reasonable at below 10%, though the relay is capped at 350 Kbytes/sec (while spiking to 500+ on a regular basis). CPU hardly seems to have increased at all. Memory consumption is, of course, up substantially with 'tor' swallowing 40% of the available 2G. Patched src/common/util.c:finish_daemon() to direct stdout/stderr to out/err.tor. If AS traps-out a violation the reported details will be captured. Had to add libasan.so.0 -> libasan.so.0.0.0 libasan.so.0.0.0 libstdc++.so.6 -> libstdc++.so.6.0.18 libstdc++.so.6.0.18 to the 'chroot' jail. Surprised to see that AS has C++ elements. Unfortunately AS requires /proc/self for access to address-space map files so mount -o bind /proc /chroot_tor/proc is required, increasing the kernel and local process attack-surface substantially. Salved the pain with echo "1" >/proc/sys/kernel/modules_disabled 'tor' is running on a box dedicated to routing and has no need for loading modules after boot completes. ----- Since it works well I'll probably look at compiling other components with AS in a couple of weeks. Those would primarily be openssl libevent zlib Though a ball-of-yarn dimension exists in that one should probably also look into compiling glibc with AS as well--a nightmare I suspect. Hopefully applying AS to the 'tor' daemon has enhanced trip-wiring of much of the available the attack surface. Looking forward to running with the upcoming 'seccomp' sandbox.
Based on my experience today, I'd say this is an excellent idea for any Tor relay operator. Have been running 'tor' on the router since late November with AddressSanitizer and it has crashed exactly ONE time: Today when someone attempted to apply the OpenSSL "Heartbleed" attack against the relay. I asked the AS devs about removing the need for mounting /proc in the 'chroot' jail and it turns out a simple call can be inserted to close /proc after AS caching of address maps is complete. Haven't created a patch for that yet, but here is the info https://code.google.com/p/address-sanitizer/issues/detail?id=237 My current patch to direct sdtout/stderr to files is attached. Only downside is that AS does consume a lot of memory. After a month it eats 80% of the 2G of RAM on the router. However it seems to level off there and cease growing. At 15:04 10/31/2013 -0500, starlight.2013q4@binnacle.cx wrote:
At 18:14 10/24/2013 -0400, starlight.2013q4@binnacle.cx wrote:
Has anyone tried running a live relay with an image built using GCC 4.8 and -fsanitize=address?
Took an initial jab at it by compiling just 'tor' with
CFLAGS = -g -O1 -D_FORTIFY_SOURCE=2 -fstack-protector-all -Wstack-protector -fwrapv --param ssp-buffer-size=1 -fPIE -Wall -fno-strict-aliasing -fsanitize=address -fno-omit-frame-pointer
where the modified or added flags are are
-O1 -fsanitize=address -fno-omit-frame-pointer
Fired it up and is looking good so far. CPU consumption quite reasonable at below 10%, though the relay is capped at 350 Kbytes/sec (while spiking to 500+ on a regular basis). CPU hardly seems to have increased at all. Memory consumption is, of course, up substantially with 'tor' swallowing 40% of the available 2G.
Patched
src/common/util.c:finish_daemon()
to direct stdout/stderr to out/err.tor. If AS traps-out a violation the reported details will be captured.
Had to add
libasan.so.0 -> libasan.so.0.0.0 libasan.so.0.0.0 libstdc++.so.6 -> libstdc++.so.6.0.18 libstdc++.so.6.0.18
to the 'chroot' jail. Surprised to see that AS has C++ elements.
Unfortunately AS requires
/proc/self
for access to address-space map files so
mount -o bind /proc /chroot_tor/proc
is required, increasing the kernel and local process attack-surface substantially. Salved the pain with
echo "1" >/proc/sys/kernel/modules_disabled
'tor' is running on a box dedicated to routing and has no need for loading modules after boot completes.
-----
Since it works well I'll probably look at compiling other components with AS in a couple of weeks. Those would primarily be
openssl libevent zlib
Though a ball-of-yarn dimension exists in that one should probably also look into compiling
glibc
with AS as well--a nightmare I suspect.
Hopefully applying AS to the 'tor' daemon has enhanced trip-wiring of much of the available the attack surface. Looking forward to running with the upcoming 'seccomp' sandbox.
I updated the patch to 1) have AS close /proc 2) enable core dump files One should add /proc /chroot_tor/proc none noauto,bind 0 0 to /etc/fstab (note the 'noauto'). Then the 'tor' startup script does a mount /chroot_tor/proc ...start tor sleep 10 umount /chroot_tor/proc And it works like a charm. 'tor' starts up with full AddressSanitizer monitoring but with no pesky /proc file system available to potential attackers. Attached are the patch and the /etc/rc.d/init.d/tor startup script.
On Thu, Apr 10, 2014 at 11:53 PM, <starlight.2014q2@binnacle.cx> wrote:
I updated the patch to
1) have AS close /proc
2) enable core dump files
One should add
/proc /chroot_tor/proc none noauto,bind 0 0
to /etc/fstab (note the 'noauto'). Then the 'tor' startup script does a
mount /chroot_tor/proc ...start tor sleep 10 umount /chroot_tor/proc
And it works like a charm. 'tor' starts up with full AddressSanitizer monitoring but with no pesky /proc file system available to potential attackers.
Attached are the patch and the /etc/rc.d/init.d/tor
I'm sold on integrating AddressSanitizer into Tor as a compile-time option. I've got a ticket for doing so #11477 (https://trac.torproject.org/projects/tor/ticket/11477). I've uploaded your patch there, and am looking into how to better integrate it. If you could make sure that the code _I_ have successfully builds Tor with AddressSanitizer when you configure --enable-compiler-hardening, that would rock. (If you like sandboxes, and Linux, you might also like to try the seccomp2 sandbox code, once Tor 0.2.5.4-alpha is out. It's present in Tor 0.2.5.3-alpha, but it's kind of buggy.) Also, see bug #11232 (https://trac.torproject.org/projects/tor/ticket/11232) for the stuff I found running under AddressSanitizer and ubsan already. best wishes, -- Nick
I'm sold on integrating AddressSanitizer into Tor as a compile-time option. I've got a ticket for doing so #11477 (https://trac.torproject.org/projects/tor/ticket/11477). I've uploaded your patch there, and am looking into how to better integrate it. If you could make sure that the code _I_ have successfully builds Tor with AddressSanitizer when you configure --enable-compiler-hardening, that would rock.
Hi Nick, I'm glad others find the idea interesting --just saw your post so sorry for the delayed reply. The patch is half-baked and needs work to make it properly portable and to open the stdxxx files only when appropriate. I'll try to find some time to improve it. I tried having 'tor' umount the /proc file system but that does not work inside a chroot jail. So all the relay process can do is issue the AS call to cache and close the address map and the 'umount' must be handled by the start script. I learned that AS disables core files separately from the Linux setting via an environment variable and that this may be a good idea due to the huge size of memory under AS. Even if the core is written as a sparse file (which I think it is), the kernel seemed to spend a ridiculous amount of time scanning for populated pages the one time I tried it. However I run an old (but minimized and security-patched) kernel that may not be the most efficient w/r/t sparse core file generation.
(If you like sandboxes, and Linux, you might also like to try the seccomp2 sandbox code, once Tor 0.2.5.4-alpha is out. It's present in Tor 0.2.5.3-alpha, but it's kind of buggy.)
secomp2 is excellent, but I'll wait till it's further along. I can take only limited quantities of bleeding- edgedness.
Also, see bug #11232 (https://trac.torproject.org/projects/tor/ticket/11232) for the stuff I found running under AddressSanitizer and ubsan already.
thank you, I'll take a look Sadly I found that AS only catches Heartbleed if the attacker reads past the end of the 17kb heap allocation that goes with the vulnerable code, even when 'openssl' is fully compiled with AS active. No doubt AS is Valgrind on steroids, but it doesn't catch improper memory accesses if the memory touched is "legal".
Finally got around to compiling 'openssl' and 'libevent' with AddressSanitizer. Found and patched one minor bug as a consequence https://trac.torproject.org/projects/tor/ticket/12227 After running for a week the relay lost its guard status due to declining BWAUTH measurements. At the time Comcast seemed to be having network problems so I'm not sure if the slowdown from ASan in 'openssl' or Comcast was the cause. Buffer stats showed 80% of cells getting queued and an average delay of 380 ms the day before the guard flags was axed. Much better on earlier days. One BWAUTH had the relay at 400 KBytes/sec so I'm inclined to think the problem was Comcast. Anyway, switched to running just 'libssl.so' with ASan and running the normal 'libcrypto.so' on the theory that the latter is CPU intensive and at least one call-level below the surface, while the former is the easier to exploit attackable-surface component. Couldn't find any CVE's mentioning 'libcrypto.so' as having an exploitable flaw --ever. Started with new keys and it's back to being a Guard and the buffer stats are much better than even the good days from running 'libcrypto.so' with ASan. At 23:53 4/10/2014 -0400, starlight.2014q2@binnacle.cx wrote:
I updated the patch to
1) have AS close /proc
2) enable core dump files
One should add
/proc /chroot_tor/proc none noauto,bind 0 0
to /etc/fstab (note the 'noauto'). Then the 'tor' startup script does a
mount /chroot_tor/proc ...start tor sleep 10 umount /chroot_tor/proc
And it works like a charm. 'tor' starts up with full AddressSanitizer monitoring but with no pesky /proc file system available to potential attackers.
Attached are the patch and the /etc/rc.d/init.d/tor startup script.
participants (3)
-
Nick Mathewson -
starlight.2013q4＠binnacle.cx -
starlight.2014q2＠binnacle.cx