Force OpenSSL AES-NI usage on a VPS without the AES CPU flag passthrough
Hello, Today I found that it is possible to force OpenSSL enable the use of CPU AES acceleration even if it doesn't detect the "aes" CPU flag. Many VPS hosts configure their hypervisors in a way that does not have the flag passed through into VPSes, even though all their host nodes surely have CPUs with support for AES-NI. In my experience hosts have not been forthcoming in reconfiguring their systems to include that flag passthrough. But it turns out we can force OpenSSL to believe AES is supported, even if the CPU does not report it. This can be done with the "OPENSSL_ia32cap" environment variable. Searching around, all I found was scenarios to use it for disabling AES-NI (for testing), e.g.: https://mjanja.ch/2013/11/disabling-aes-ni-on-linux-openssl/ I believe the syntax used there applies "xor" over the real flag values, e.g. OPENSSL_ia32cap="~0x200000200000000" to disable AES. But what if you need to force-enable it. Turns out the syntax working for that is simply: OPENSSL_ia32cap="+0x200000200000000" Let's take one VPS box with the aforementioned problem. # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 13 model name : QEMU Virtual CPU version 1.5.3 stepping : 3 microcode : 0x1 cpu MHz : 1695.729 cache size : 4096 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 4 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl eagerfpu pni cx16 hypervisor lahf_lm bugs : bogomips : 3391.45 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: As you can see "aes" is not present. Testing OpenSSL speed in the default configuration. # openssl speed -elapsed -evp aes-128-gcm ... type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-128-gcm 32332.17k 40365.80k 42265.77k 42376.19k 43196.42k 43401.22k And now we force AES hardware acceleration usage: # OPENSSL_ia32cap="+0x200000200000000" openssl speed -elapsed -evp aes-128-gcm ... type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-128-gcm 59047.07k 104467.39k 141712.21k 151093.93k 158321.32k 156827.65k Almost 4 times faster! For Tor this leads to higher bandwidth utilization and lower CPU usage (which will actually make your VPS host happier :) But how to apply that to Tor? Well, for some initial testing I just chose to add the following line to /etc/init.d/tor (not using systemd here), right after "#! /bin/bash": export OPENSSL_ia32cap="+0x200000200000000" (but this will get overwritten and removed by the next Tor version upgrade). Seems to work just fine, my CPU usage is half of what it was before, at similar bandwidth levels. So now it has headroom to ramp up further. A word of caution, firstly always test as shown above to verify that it works. I believe if the underlying CPU actually doesn't support AES, all programs trying to use it, including "openssl speed", will crash outright, with an incorrect instruction error, or somesuch. So there is no risk to jeopardize encryption strength or security of Tor. Also even if it works now, it may stop working down the line if the host migrates your VPS to a node with older CPU, one which doesn't support AES. But migrations of customers between do not happen very often, and in fact all CPUs used today in a hosting environment should support AES, as that's been implemented in server (and even desktop) CPUs a very very long time ago. (Apologies for double-post, I realized tor-relays should have been the better place to post). -- With respect, Roman
On 22 Aug 2017, at 16:22, Roman Mamedov <rm@romanrm.net> wrote:
Hello,
Today I found that it is possible to force OpenSSL enable the use of CPU AES acceleration even if it doesn't detect the "aes" CPU flag.
This would be a great addition to tor/doc/TUNING. Does someone want to summarise it and submit a patch to: https://trac.torproject.org (I'd open a ticket, but trac is having temporary performance issues.) Here's the existing TUNING file: https://gitweb.torproject.org/tor.git/tree/doc/TUNING
Many VPS hosts configure their hypervisors in a way that does not have the flag passed through into VPSes, even though all their host nodes surely have CPUs with support for AES-NI. In my experience hosts have not been forthcoming in reconfiguring their systems to include that flag passthrough.
But it turns out we can force OpenSSL to believe AES is supported, even if the CPU does not report it. This can be done with the "OPENSSL_ia32cap" environment variable. Searching around, all I found was scenarios to use it for disabling AES-NI (for testing), e.g.:
https://mjanja.ch/2013/11/disabling-aes-ni-on-linux-openssl/
I believe the syntax used there applies "xor" over the real flag values, e.g. OPENSSL_ia32cap="~0x200000200000000" to disable AES. But what if you need to force-enable it. Turns out the syntax working for that is simply:
OPENSSL_ia32cap="+0x200000200000000"
Let's take one VPS box with the aforementioned problem.
# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 13 model name : QEMU Virtual CPU version 1.5.3 stepping : 3 microcode : 0x1 cpu MHz : 1695.729 cache size : 4096 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 4 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl eagerfpu pni cx16 hypervisor lahf_lm bugs : bogomips : 3391.45 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:
As you can see "aes" is not present.
Testing OpenSSL speed in the default configuration.
# openssl speed -elapsed -evp aes-128-gcm ... type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-128-gcm 32332.17k 40365.80k 42265.77k 42376.19k 43196.42k 43401.22k
And now we force AES hardware acceleration usage:
# OPENSSL_ia32cap="+0x200000200000000" openssl speed -elapsed -evp aes-128-gcm ... type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-128-gcm 59047.07k 104467.39k 141712.21k 151093.93k 158321.32k 156827.65k
Almost 4 times faster! For Tor this leads to higher bandwidth utilization and lower CPU usage (which will actually make your VPS host happier :)
But how to apply that to Tor? Well, for some initial testing I just chose to add the following line to /etc/init.d/tor (not using systemd here), right after "#! /bin/bash":
export OPENSSL_ia32cap="+0x200000200000000"
(but this will get overwritten and removed by the next Tor version upgrade).
Seems to work just fine, my CPU usage is half of what it was before, at similar bandwidth levels. So now it has headroom to ramp up further.
A word of caution, firstly always test as shown above to verify that it works.
I believe if the underlying CPU actually doesn't support AES, all programs trying to use it, including "openssl speed", will crash outright, with an incorrect instruction error, or somesuch.
So there is no risk to jeopardize encryption strength or security of Tor.
Also even if it works now, it may stop working down the line if the host migrates your VPS to a node with older CPU, one which doesn't support AES. But migrations of customers between do not happen very often, and in fact all CPUs used today in a hosting environment should support AES, as that's been implemented in server (and even desktop) CPUs a very very long time ago.
T -- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org ------------------------------------------------------------------------
On Wed, Aug 23, 2017 at 11:14:54AM +1000, teor wrote:
On 22 Aug 2017, at 16:22, Roman Mamedov <rm@romanrm.net> wrote:
Hello,
Today I found that it is possible to force OpenSSL enable the use of CPU AES acceleration even if it doesn't detect the "aes" CPU flag.
This would be a great addition to tor/doc/TUNING.
Does someone want to summarise it and submit a patch to: https://trac.torproject.org
I'd be a bit cautious about documenting this; it's arguably a hypervisor bug that the AESNI instructions are enabled but the AES bit is not set in CPUID. If your VM gets moved to hardware that actually doesn't have the instructions, or if the system has AESNI turned off for a good reason (like a buggy encryption implementation), you're asking for more breakage. According to https://software.intel.com/en-us/forums/intel-isa-extensions/topic/287887 there are control bits in MSR 0x13c for AESNI. I'm not arguing that it's unreasonable to play with this force-on setting, or even to run it on a tor relay, but you've gotta know that when it breaks, you get to keep both pieces. :) -andy -andy
Roman Mamedov:
But what if you need to force-enable it. Turns out the syntax working for that is simply:
OPENSSL_ia32cap="+0x200000200000000"
Do you know what platforms and openssl versions support this? thanks, nusenu -- https://mastodon.social/@nusenu https://twitter.com/nusenu_
nusenu <nusenu-lists@riseup.net> wrote:
Roman Mamedov:
But what if you need to force-enable it. Turns out the syntax working for that is simply:
OPENSSL_ia32cap="+0x200000200000000"
Do you know what platforms and openssl versions support this?
And is the same available for libressl, which is now the default SSL package on a growing number of systems? Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************
participants (5)
-
Andy Isaacson -
nusenu -
Roman Mamedov -
Scott Bennett -
teor