Virtualized pfsense, now AES is having no effect on OpenVPN performance

[eroji]

Posted this a couple places looking for some ideas and I thought I'd try here too. Hopefully there are some pfsense experts lurking.

So I went ahead and virtualized pfsense (in ESXi 6, with NICs passedthrough) and everything as far as I can tell functions flawlessly. However, I am noticing a strange problem with OpenVPN performance post the P-to-V. Here are the specs for servers.

Server 1:

• Lenovo RS140
• E3-1225v3
• 2x 8GB DDR3 ECC UDIMM
• X520-DA2

Server 2:

• ASRock EP2C602-4L/D16
• 2x E5-2670
• 16x 8GB DDR3 ECC RDIMM

I have OpenVPN server set up on both, and Server 2 is set as a client to connect to 1 as site-to-site. Server 1 has a 150/150 link, and Server 2 is 200/20. Prior to virtualizing, I was able to pull stuff to Server 2 side at about link speed of Server 1. After virtualizing, it's now stuck at about ~40Mb. I looked around on pfsense forums, and the only options that could be related to OpenVPN performance is aesni.ko on/off, cryptodev on/off and ip.fastforwarding=0/1. I've tried combinations of all 3, and it's having zero effect on the performance. CPU load during transfers is about 3-4% on Server 1 and about 10% on Server 2. I also confirmed that on CLI level, both servers seems to be seeing AES support from CPU properly, and loading the aesni.ko module is indeed allowing the proper ciphers to be available to cryptodev. I was even able to run commands to measure the encryption performancing using AES-128-CBC. I'm not really understanding what else could be causing this performance cap.

 

[j_h_o]

/sub. I am also seeing this on my Hyper-V-backed instances, and I've done similar investigations as you have, but haven't found a fix.

 

[Keljian]

Try turning off all acceleration, openvpn will automatically use it if it is present

 

[eroji]

/sub. I am also seeing this on my Hyper-V-backed instances, and I've done similar investigations as you have, but haven't found a fix.

It doesn't look good on getting an answer on this. I'm kinda surprised on the activity on pfsense forums lately. Any time I actually have a question, it seems to only get a few views and never any response. I might ultimately end up going back to physical pfsense, which is disappointing. I had originally planned it out to have the servers host pfsense and domain controller VMs, so that they would be on a higher horizontal level in the infrastructure than the rest of the VMs.

Try turning off all acceleration, openvpn will automatically use it if it is present

I already tried this combination, with aesni off (and unloaded) and cryptodev off in OpenVPN configuration. It made no difference.

 

[canta]

Do not know on esxi in detail...

on proxmox ( or linux KVM), you need to pass real cpu to VM since hardware acceleration not supported in Virtual CPU.
I change processor to real processor. AES kicks in on openvpn

you should see a big differences on cpu processing when AES disabled or enabled by assuming your openvpn is using AES encryption.

the other to consider, openvpn is running single thread, so highest processor has better performance on through put....

 

[Keljian]

This may sound silly, but did you reboot after each config change?

 

[eroji]

AFAIK, in ESXi, most features of CPU are exposed to the VMs by default. AES should be one of them. I can run the command 'dmesg | grep -i aes' and it shows the following:

Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
aesni0: <AES-CBC,AES-XTS,AES-GCM> on motherboard

 

[eroji]

This may sound silly, but did you reboot after each config change?

Must've done that a dozen times by now.

 

[sachem87184]

Question. Are you running the free version of ESXi? If not, do you have EVC enabled on the host cluster? If you run clustering you have to enable a high enough level of EVC to expose the CPU instruction sets for aesni.

 

[eroji]

Question. Are you running the free version of ESXi? If not, do you have EVC enabled on the host cluster? If you run clustering you have to enable a high enough level of EVC to expose the CPU instruction sets for aesni.

It's a standalone ESXi. No EVC and no cluster currently. The NICs are passedthrough. Gave them 4 cores and 4GB of RAM.

 

[sachem87184]

Note: Connecting over 300Mb wifi to a setup similar to your server 2 on my system I get around 120Mb-150Mb transfer speeds using AES-256-CBC.

 

[sachem87184]

Do you have the crypto selected in System/Advanced/Misc and also in the OpenVPN server config?

and run "cryptostats" in the pfsense shell. Should look like this

[2.2.6-RELEASE][admin@server.network.local]/root: cryptostats
4765489 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations

dispatch->invoke: avg 872 ns : min 768 ns : max 1047 ns [4 samps]
invoke->done: avg 2286 ns : min 1955 ns : max 2933 ns [4 samps]
done->cb: avg 819 ns : min 768 ns : max 837 ns [4 samps]
cb->finis: avg 907 ns : min 837 ns : max 977 ns [4 samps]

 

[eroji]

Yes, that is enabled, which loads the aesni.ko module. I confirmed this in shell level. This also allows cryptodev to list AES-128-CBC as one of the available ciphers once loaded. I was also able to run test in shell to measure the throughput of openssl engine using AES. So as far as I can tell, on the system level, that is enabled and available. It almost seems as if OpenVPN isn't using it, even though the system has it and it is set to used cryptodev in the configuration.

 

[j_h_o]

Once enabled, you should see a line like:
openvpn[41612]: Initializing OpenSSL support for engine 'cryptodev'
in the OpenVPN log.

 

[eroji]

Do you have the crypto selected in System/Advanced/Misc and also in the OpenVPN server config?

and run "cryptostats" in the pfsense shell. Should look like this

[2.2.6-RELEASE][admin@server.network.local]/root: cryptostats
4765489 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations

dispatch->invoke: avg 872 ns : min 768 ns : max 1047 ns [4 samps]
invoke->done: avg 2286 ns : min 1955 ns : max 2933 ns [4 samps]
done->cb: avg 819 ns : min 768 ns : max 837 ns [4 samps]
cb->finis: avg 907 ns : min 837 ns : max 977 ns [4 samps]

Here are the outputs. Looks like it did something. I am not sure if that is from my tests ran from shell level though.

Server 1:

8309853 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations

Server 2:

7461261 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations

 

[sachem87184]

So it looks like it's running. I'll check my setup and see what I have customized.

 

[eroji]

Once enabled, you should see a line like:
openvpn[41612]: Initializing OpenSSL support for engine 'cryptodev'
in the OpenVPN log.

Is this when you select cryptodev in the OpenVPN configuration or when you turn on AES-NI in Advanced > Misc?

I'm not seeing any new entries in /var/log/openvpn.log, but this is interesting... (on Server 2)

Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)
Feb 29 12:00:11 pfsense2 openvpn[10534]: write UDPv4: No buffer space available (code=55)

BTW, you guys are awesome. Thanks for taking the time to reply to this.

 

[j_h_o]

Is this when you select cryptodev in the OpenVPN configuration or when you turn on AES-NI in Advanced > Misc?

That's what I see after I enable cryptodev in the OpenVPN config, and it shows in my OpenVPN log.

 

[eroji]

So looking at the output of crpytostats with aesni enabled, cryptodev enabled and starting a file transfer, I do see the crpyto op count going up fairly rapidly. If I stop the transfer, the growth slows. Seems almost as if it is working but perhaps E5-2670 is just doesn't quite cut it for AES encryption... I previously had a i3-4330T on Server 2 end. For a 35W CPU, it was giving me ~150Mb transfers from Server 1 side.

Here are the openssl speed benchmark ran on both. Looks like 747MB/s vs 664MB/s approximately. Should be more than sufficient.

Server 1:

openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 2021451 aes-128-cbc's in 0.37s
Doing aes-128-cbc for 3s on 64 size blocks: 1913225 aes-128-cbc's in 0.36s
Doing aes-128-cbc for 3s on 256 size blocks: 1630944 aes-128-cbc's in 0.27s
Doing aes-128-cbc for 3s on 1024 size blocks: 1044184 aes-128-cbc's in 0.23s
Doing aes-128-cbc for 3s on 8192 size blocks: 238356 aes-128-cbc's in 0.03s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      88083.65k   340720.42k  1571846.26k  4719423.63k 62483595.26k

Server 2:

openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 1954721 aes-128-cbc's in 0.34s
Doing aes-128-cbc for 3s on 64 size blocks: 1859211 aes-128-cbc's in 0.27s
Doing aes-128-cbc for 3s on 256 size blocks: 1560920 aes-128-cbc's in 0.27s
Doing aes-128-cbc for 3s on 1024 size blocks: 941278 aes-128-cbc's in 0.18s
Doing aes-128-cbc for 3s on 8192 size blocks: 201619 aes-128-cbc's in 0.05s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      90983.38k   435161.61k  1504359.60k  5364138.70k 35235474.09k

 

[j_h_o]

My transfers across the tunnel are performant now.

I have Hyper-V (2012 R2) running on an E5-2620 and a E3-1230v3 across a 1GbE link. I was seeing around 30MB/s, and I'm hitting 100MB/s now.

I'll go shove in my 10GbE switch sometime, and attach SFP+ DACs, but that won't be a few months before I get back to this DC.