And my ultimate question... can the Intel S3710s be used in a meaningful way to accelerate the performance of the HDD pool, such that I can have my cake, and eat it too. (i.e. use the capacity of the HDD pool for ESX lab space, while only actively using a small portion of it at any specific time (i.e. within my L2ARC space after warm up).
Carved up the Intel S3710 400GB SSDs in to (2) Partitions - 85% (336GB) and 15% (56GB)
With each the SSDs on separate Controllers, and using RAID1/0 for ZIL = (112GB) 2 drives worth of performance on writes
Then used the 336GB * 4 Partitions as non-mirrored L2ARC = 1.3TB of L2ARC.
This might not be the most optimal sizes/scheme, but it's a start...
Hitachi Pool (RAIDz1*2 with SSD) – ZFS Volume
Sync Disabled (QD4)-------- Sync Always (QD4)----------Sync Always (QD10)
Here we see that pool is getting HDD speeds on non-sync workloads, which we expect.
The interesting finding is, with sync enabled and QD4.
We're getting 80% of the pure Intel Pool speeds on Sync Writes across all IO sizes.
But we're also getting 100% of the pure Hitachi Pool speeds on Reads, which is faster than using the Intels in a pool by themselves.
Testing again with QD10 just shows you can squeeze a little more performance out of super small IOs, but larger blocks are about the same.
My current thoughts...
I'm kind of loving the Intels accelerating the Hitachi Pool at the moment.
I'm getting the best of both worlds, in that I can:
- Use the Hitachi HDD pool for large archive/CIFS based workloads (30TB~)
- And also use the same reservoir of space for iSCSI ESX Sync workloads
- I won't get full pure SSD speeds always and forever. But 75-80% is very livable, if that's what it ends up being (where it counts)
- Especially since the Intels by themselves in RAID1/0 is < 800GB~ Usable for ESX (I need 5TB+ to start with < 1TB active
- Plus i'm getting better than Intel SSD speeds on all reads, so they aren't hurting
Iperf Benchmarks on 4 threads were consistently around 15.5Gbits/second.
Add iSCSI iSER/SRP overhead, and seeing 1.7GBs reads/writes is pretty solid.
I'm also seeing the benefit of the latest and greatest CPUs and memory architecture when it comes to RAW ram performance, and how that pertains to ZFS performance. I've also got my eyes on the Intel 750 PCIe NVMe 400GB card, even if using it in my PCIe 2.0 x8 configuration will gimp it.
I could theoretically sell my (4) Samsung Evo 850s and my (4) Intel S3710s, and just put that PCIe behind my hitachi's and end up better off. (you were right!!! <insert you here>)
But unless i'm incredibly mistaken, and just plain doing it wrong... I kind of feel vindicated, that there is a meaningful way to use the Intels to accelerate the Hitachi's, without impacting the performance of the HDDs, and only helping where it counts.