I'm in the process of benchmarking an OLAP database and I've run into some interesting results. I have some theories, but I'd love to hear from someone that actually knows what on earth is going on. So I've put together my benchmark server. This server does nothing but switch back and fourth between Windows Server 2012 R2 and ESXi 6.5 with a Windows Server 2012 R2 guest. The hardware looks like this:
Processor(s): (2) Intel Xeon E5-2670 @ 2.6 GHz
Motherboard: ASRock EP2C602-4L/D16
Memory: 128 GB - (16) Crucial 8 GB ECC Registered DDR3 @ 1600 MHz
Chassis: Supermicro CSE-846E16-R1200B
RAID Controller: LSI MegaRAID Internal SAS 9265-8i
Solid State Storage: (4) Samsung 850 EVO 250 GB on LSI SAS in RAID 0
Solid State Storage: (2) Samsung 850 EVO 250 GB on Intel SATA
Solid State Storage: (1) Samsung 850 EVO 250 GB on LSI SAS
NVMe Storage: (1) Intel P3605 1.6TB AIC
Hard Drive Storage: (12) Fujitsu MBA3300RC 300GB 15000 RPM on LSI SAS in RAID 10
Network Adapter Intel X520-DA2 Dual Port 10 Gbps Network Adapter
I've basically set up my storage above with partitions for Windows and ESXi. So for instance, the P3605 has a GPT partition for Windows (800GB) and a GPT partition for ESXi (800GB). This let's me use the same exact drive and go back and fourth without having to do to much work.
So for my disk configurations, I'm noticing that sequential reads and writes are...okay. As a sample, here's the Samsung 850, single drive configuration:
Again, nothing looks funky here to me. physical performs a little better than virtual pretty much across the board. Makes sense. When I look at 4K performance however...things go off the rails:
At high queue depths, things look pretty even. Again, virtual lags behind physical with few exceptions. But at low queue depths, which is what Essbase cares about, performance is just awful. CDM shows is being over 50% slower while Anvil shows it performing a little better.
I thought, okay, this isn't good, but let's keep checking other configurations. So here's my P3605, again sequential:
The P3605 looks like it does pretty well in sequential performance, just like the Samsung 850 did. There's an obvious exception on the seq read at a QD of 1, which is concerning...but anvil shows it performing fine. So I'll toss that out for now.
4K Performance though...another nosedive:
In Anvil, the perfomance is pretty awful across the board. In CDM, the performance is better at high QD, but again...awful at low QD.
So...all of that to ask a simple question. Why is random I/O so awful, specifically at low queue depths? Is this a driver tuning issue? Are ESXi drivers just tuned for better performance at higher queue depths, or is it something deeper in the way that ESXi does things? My virtual disks are all thick provisioned with eager zeroing. Though my testing shows that it doesn't appear to matter. Is there anything I can do to improve low QD 4K performance? Essbase operates at a QD of 1. All of my benchmarks basically follow the QD1 test results for anything in Essbase that does a lot of random I/O. The sequential results are all good in Essbase...but don't try anything random or performance goes off a cliff.
I would love some actual expert opinions. I might be that in Essbase (I stress might), but I'm dangerous at best in ESXi.
Thanks in advance for your thoughts and opinions (and facts)!
Processor(s): (2) Intel Xeon E5-2670 @ 2.6 GHz
Motherboard: ASRock EP2C602-4L/D16
Memory: 128 GB - (16) Crucial 8 GB ECC Registered DDR3 @ 1600 MHz
Chassis: Supermicro CSE-846E16-R1200B
RAID Controller: LSI MegaRAID Internal SAS 9265-8i
Solid State Storage: (4) Samsung 850 EVO 250 GB on LSI SAS in RAID 0
Solid State Storage: (2) Samsung 850 EVO 250 GB on Intel SATA
Solid State Storage: (1) Samsung 850 EVO 250 GB on LSI SAS
NVMe Storage: (1) Intel P3605 1.6TB AIC
Hard Drive Storage: (12) Fujitsu MBA3300RC 300GB 15000 RPM on LSI SAS in RAID 10
Network Adapter Intel X520-DA2 Dual Port 10 Gbps Network Adapter
I've basically set up my storage above with partitions for Windows and ESXi. So for instance, the P3605 has a GPT partition for Windows (800GB) and a GPT partition for ESXi (800GB). This let's me use the same exact drive and go back and fourth without having to do to much work.
So for my disk configurations, I'm noticing that sequential reads and writes are...okay. As a sample, here's the Samsung 850, single drive configuration:
Again, nothing looks funky here to me. physical performs a little better than virtual pretty much across the board. Makes sense. When I look at 4K performance however...things go off the rails:
At high queue depths, things look pretty even. Again, virtual lags behind physical with few exceptions. But at low queue depths, which is what Essbase cares about, performance is just awful. CDM shows is being over 50% slower while Anvil shows it performing a little better.
I thought, okay, this isn't good, but let's keep checking other configurations. So here's my P3605, again sequential:
The P3605 looks like it does pretty well in sequential performance, just like the Samsung 850 did. There's an obvious exception on the seq read at a QD of 1, which is concerning...but anvil shows it performing fine. So I'll toss that out for now.
4K Performance though...another nosedive:
In Anvil, the perfomance is pretty awful across the board. In CDM, the performance is better at high QD, but again...awful at low QD.
So...all of that to ask a simple question. Why is random I/O so awful, specifically at low queue depths? Is this a driver tuning issue? Are ESXi drivers just tuned for better performance at higher queue depths, or is it something deeper in the way that ESXi does things? My virtual disks are all thick provisioned with eager zeroing. Though my testing shows that it doesn't appear to matter. Is there anything I can do to improve low QD 4K performance? Essbase operates at a QD of 1. All of my benchmarks basically follow the QD1 test results for anything in Essbase that does a lot of random I/O. The sequential results are all good in Essbase...but don't try anything random or performance goes off a cliff.
I would love some actual expert opinions. I might be that in Essbase (I stress might), but I'm dangerous at best in ESXi.
Thanks in advance for your thoughts and opinions (and facts)!
