Server 2012 R2, Storage Spaces and Tiering

[PigLover]

Starting a new thread for open discussion of recent upgrades to Storage Spaces. Is this enough to make it interesting?

With the release of Server 2012 R2 / Windows 8.1 MS has added some interesting new capabilities to Storage Spaces: Storage Tearing and Write-Back Cache.

Storage Tiering allows creation of a pool containing both SSD and HDD storage with promotion of frequently accessed data to the SSD tier with less frequently accesses data stored on the (presumably slower) HDDs. SS does the promotion automatically at a "slab" (block) level through a once-a-day scan of the pool. You can also "pin" files to the SSD layer - either specific files or any files with certain characteristics (MSs canonical example is pinning files with the name "*.vhd" so that virtual disks for VMs land on the SSD tier).

Write-Back Cache is simply the allocation of SSD-based space for faster writes. Given SSs well known problems with write speed on Parity (Raid5/6) volumes this could be very attractive.

 

[PigLover]

My first impressions: not that impressed.

I'm early in my testing, but there are still so many other unresolved issues with Storage Spaces that I just can't be excited. Its disappointing because there are so many other related features in Server 2012 that really could make this a competitive platform.

Among other things:

Parity vDisks are still painful. Slower than spit when writing. Unless you turn on "Power Protected" mode. Which is a REALLY dangerous thing to do unless you are actually "Power Protected" (which means a lot more than just having a UPS). Nothing even close to the performance of competing technology (ZFS or Hardware Raids).

Worse, Parity vDisks really only work well at all if you have an exact multiple of 8 drives. Its a long story and requires a deep understanding of how SS actually works - but its effects are easily demonstrated if you build a Parity space on a 9 or 10 drive pool and watch the drive activity lights as you copy data to it.

Write-Back Cache helps write performance - but only up to the point that Cache allocated on the SSD fills up. Then its back to painful performance again. No comparison at all to ZFS with a well-sized ZIL or Hardware WB cache. And - if you build a large Cache on a larger SSD (say 100GB cache) and write lots of data you can watch the hours-long after effects of draining that cache into the HDDs at a snails pace (yes - hours long...no exaggeration).

Tiering seemed very interesting. At first...then reality sets in. Doing their "promotion" scans via a once-daily batch job just doesn't do much for "real time" tiering. Basically, you are always accelerating yesterday's workload. I guess its fine if you are nice consistent and predictable promotion needs. But if they are nice and consistent and predictable then you don't really need tiering at all...just build and SSD array and store the important files there. Duh! Same story for "pinning". I already store my virtual disk files on an SSD array. Not much real benefit.

Also - just to drive the final nail - Tiering is disabled on a Parity vDisk. Ugh! The the most interesting use case and kill it. If you can only use Tiering on Striped or Mirrored pools then who really needs it. You already get the extra IOPs from striping.

Quoting Maxwell Smart seems about right: "Missed it by that much...".

 

[Scout255]

Didn't realize tiering is disabled in parity mode.... that's very disapointing.

Has the performance of parity improved at all with R2 vs 2012? Or are you still limited to 40-50mb/s continuous writes?

 

[PigLover]

I don't have any logged benches from R1, so I can't say "no improvement", but I can tell you that I can't even get close to "40-50MB/s" with R2. So yes - its still slower than <choose your favorite foul word>.

 

[PigLover]

I don't have any logged benches from R1, so I can't say "no improvement", but I can tell you that I can't even get close to "40-50MB/s" with R2. So yes - its still slower than <choose your favorite foul word>.

I will say - and this was true before - it really doesn't perform too badly for Simple (Raid 0) or Mirror (Raid 1+0) spaces. Could be better, but really not too bad. Unfortunately i just can't justify 1+1 redundancy of large HDDs.

 

[PigLover]

I had some success last night figuring out how to get a parity space configured with SSD journaling and a large(ish) SSD writebackcache. Also figured out how to do parity spaces more effectively on a 12-drive HDD pool (hint: use "-NumberOfColumns 4" to get a parity block rotating within every group of 4 drives, yielding the same redundancy as having 3x 4-drive Raid-5s striped over 12 drives).

Last night was getting 400-500MB/s sequential reads, 200-300MB/s sequential writes (and getting these speeds without resorting to dangerous tricks like enabling "power protected mode"). Still less than half of what I'd expect from ZFS or hardware raid over 12 drives with SSD ZIL or LSI Cachecade but from a practical perspective getting into the "disappointing but tolerable" range. I'm going to do some more benches over the weekend to see if this level of performance, coupled with the advantages of SMB3, might actually make a better overall performance solution when measured at Windows 8 based clients.

Also - the disk configuration I'm testing on is somewhat impaired. Performance with less hardware constraints might be much better. SSDs I'm using are crap (old Vertex-1s that I pulled out of a junk drawer). Now that it is getting interesting I do have some Samsung 840 Pros that I can throw into the party this weekend. And the whole thing is running on a DL180, which means the 12 HDDS are behind an expander that limits them to SATA-II speeds and (worse) a single 4-lane link running @ 300MB/s/link to the HBA which creates quite a bottleneck. HBA is an M1015 and the SSDs are at least direct connected on a forward breakout cable.

This weekend I plan to upgrade the SSDs and do some more rigorous benches. I expect the upgraded SSDs will show better write performance, but the reads will remains bottlenecked by the single-linked expander. I'll also bench performance as seen at a Windows 8 client. Then I'll tear it down and re-load with ZFS and re-bench to get apples-apples on the same hardware (thanks, Gea, for creating the VMware appliance to make that easy!).

 

[dba]

Great thread. I anxiously await your further results!

 

[cesmith9999]

PigLover,

can you spill out the SSD configuration and commands that you used to get those performance numbers?

 

[PigLover]

PigLover,

can you spill out the SSD configuration and commands that you used to get those performance numbers?

Yes. When I do the benchmarks this weekend I will include the exact configuration of each test and the powershell commands used to create that configuration.

I can't actually post the powershell from last night because there was too much trial/error getting to it.

I do find it a pain that you can't use the wizards to create what really should be almost standard configs. Its a double pain here because MS won't actually publish the documentation on 2012 R2s powershell until the RTM date in November. While we are in the preview period its guessing, intuition, on-line help inside powershell (which really isn't all that helpful) and reading Jose's blogs.

 

[PigLover]

The next few posts will have a series of benches. Some tonight, some tomorrow and (unfortunately) some later.

The system all of this will be run on is as follows:

- HP DL180 G6
- 2x L5639
- 96GB DDR3 1066, triple channel
- Boot: 2x Samsung 840 Pro 128GB, raid-1 on MB raid controller
- HDDs: 12x Hitachi 2TB 7200 RPM.
- HDDs Connected to chassis backplane with an unfortunate built-in expander that only support STAT-II speeds and links to the HBA on a single 4-lane 8087 cable.
- SSDs for pool: 4x Samsung 840 Pro 128GB
- HBA: IBM M1015 flashed to IT mode. One port connected to HDD backplane and the other to the 4 pool SSDs via a SATA forward breakout.
- Network: Mellanox ConnectX-3 EN 10Gbe (2x Intel 1Gbe from DL180 connected but disabled)

A few photos, just for fun:

Four new Samsung's all ready to join the pools:

SSDs all bundled up in my (should be patented) "custom Velcro mount":

Add SATA power "vampire" plugs:

Take a Molex power cable from the junk drawer, clip the end and press it home:

Press some of the fuzzy size from a roll of Velcro adhesive tape into the side of the DL180 and mount it all up:
(note the other two Samsung's used for boot)

And add the 10Gbe to finish it off:

I do wish I had the PCIe cage get everything stable, but I'll just have to Ghetto-rig something later...

 

[PigLover]

First two benchmarks are just setting some baselines. Test the speeds on a single SSD connected to the M1015 and a single HDD on the expander. Knowing what each drive can do by itself should help set expectations for the rest of the tests.

Here's the Samsung 840 Pro. Pretty much looks like every other benchmark of these drives. Good and fast.

And here's a single Hitachi 2TB 7200. Again - looks just about like it should. No I'll effects from the borked expander when looking at one drive at a time:

 

[PigLover]

Next is a Simple Space of all four SSDs (simple stripe, raid-0 equiv.). No powershell used to create this one - just picked four drives in the wizard, grouped them into a Storage Pool, created a Simple Space using the entire pool and formatted it NTFS.

100% read IOPs: 88,735.

Simple SSD Pool:

This looks just about like what we would expect. Good performance compared to a single SSD in almost every measure, right between 3x-4x a single SSD. Hardware Raids and well tuned software (like ZFS) usually do a bit better than this but not too bad. So far, so good. But frankly Raid0, simple stripes, isn't all that hard to do right...

 

[PigLover]

Simple Pool of all 12 HDDs. Again - all configured with simple clicks through the wizard. Now PowerShell yet.

100% read IOPs: 10,541.

OK - so here we have our first performance "oops". But I can't really blame Storage Spaces for this one. Based on the Single-SSD/4x SSD results you would have expected to see something in the range of 1,500+ MB/s sequential reads here. But that silly expander backplane, 300MB/s limits for SATA and a single 4-lane SAS link all conspire to slow it down. I'll take for granted that this is due to the Expander.

Random reads are pretty close to single disk performance - which is what you would expect for a single-reader test on Raid0. Might have expected random writes to be closer to 12x single disk speeds - the expander doesn't account for that deficit. Hmmm.

Need to get IOPs numbers. Will retest and add them later...

 

[PigLover]

Continuing to baseline. Nothing too interesting yet. Here's a 2-way mirror pool with all four SSDs. Pretty much a raid-10 of 4 SSD drives.

100% read IOPs: 88,604.

Based on what we saw in the single-disk and Simple Space tests for the SSD this isn't too far off what we would expect. A cacheless hardware raid would show about the same thing - read speeds comparable to the Simple Space / Raid 0 and Write speeds about half the Simple stripe (obviously - because you have to write it twice and you don't have any cache...).

 

[PigLover]

And 2-way Mirror Space of the HDDs:

100% read IOPs: 10,098.

At least the results are consistent. Assuming it is the expander slowing down the HDD array - then these would be about the expected results.

 

[PigLover]

Single Parity on the SSDs. Still constructed just using the wizards.

100% read IOPs: 81,979.

This is where you start to see the problems with SS. Read speeds are OK - a little lower than Mirror, but not too bad. Read IOPs look OK too. But write speeds are starting to collapse. Getting less than 300MB/s seq. writes in Crystal and large-block writes at 500-600MB/s in ATT0 are just miserable for a 4-SSDs Raid5.

 

[PigLover]

Single parity on the 12 HDDs. Also constructed with the wizards.

100% read IOPs: 10,546.

Ouch! Reads down slightly from the mirror pool. Read-IOPs OK. But write speeds have completely collapsed! This is totally unacceptable performance - unusable in many cases. This is actually the best performance I've ever seen on this system for the 12-drive parity built by the wizards - almost twice what I was seeing earlier. Don't know why and don't care. This performance of Parity spaces is the number-1 complaint people voice about SS.

 

[PigLover]

OK. So that's all the baselining. Next few will be attempts to add performance by adding tiering, cache and other options from Server 2012 R2

First off, we'll bench a more realistic array configuration. This time we build a parity array on the HDD-only pool but set the option "NumberOfColumns" to 3. This has the effect of building parity groups of 4 drives each. Since the pool has 12 drives in it this means we end up with 3 parity groups. From a resiliency perspective this is similar to a raid-50. Not quite as robust as dual parity. But this still gives you protection against loss of 3 drives (just not ANY 3 drives - if two of them are in the same group you are screwed!). This configuration also gives the option of growing the array in groups of 4-drives at a time.

You need PowerShell to create this VirtualDisk - you can't specify it correctly in the wizards. So build the pool it the wizard and use this command to create the virtual disk:

New-VirtualDisk -StoragePoolFriendlyName HDDtest -FriendlyName HDD_Parity -UseMaximumSize -ResiliencySettingName Parity -ProvisioningType Fixed -NumberOfColumns 4

And then go back into the wizard to create a filesystem.

100% read IOPs: 10,015.

Se here we see read performance tanked and write performance is still similar to (slightly below) the straight-up 12 drive parity pool. But this is the starting point of a realistic deployment scenario (nobody sane would deploy a 12-drive Raid-5 with 2TB or larger drives).

 

[PigLover]

This time we add two of the SSDs to the pool and build the same virtual disk as before. Adding the SSDs and defining tiers in the pool should do two things to help write speeds:

- The SSDs should be detected as journal drives and writes to the array should use journaling (ala ZFS Zil)
- The SSDs should also be used to create a default size WriteBackCache (1GB).

Have to use PowerShell again to set up the drive tiers and create the disk:

Get-StoragePool -FriendlyName TieredPool | Get-PhysicalDisk | ? MediaType -eq SSD | Set-PhysicalDisk -Usage Journal
New-StorageTier -StoragePoolFriendlyName TieredPool -FriendlyName SSD_Tier -MediaType SSD
New-StorageTier -StoragePoolFriendlyName TieredPool -FriendlyName HDD_Tier -MediaType HDD
New-VirtualDisk -StoragePoolFriendlyName TieredPool -FriendlyName HDD_Parity -UseMaximumSize -ResiliencySettingName Parity -ProvisioningType Fixed -NumberOfColumns 4

100% read IOPs: 11,399.

Reads still unreasonably low, but writes are up significantly. IOPs also slightly up (due, I think, to getting some of the reads out of the cache). Getting better. Note that this write performance level was achieved without changing the dangerous "IsPowerProtected" variable.

 

[PigLover]

Next is same 2 SSDs in the pool but set the WriteCache to a large size. 100GB should do for now.

Build the pool using the wizard with 12 HDD drives & 2 SSDs. Then Powershell to create the VirtualDisk:

Get-StoragePool -FriendlyName TieredPool | Get-PhysicalDisk | ? MediaType -eq SSD | Set-PhysicalDisk -Usage Journal
New-StorageTier -StoragePoolFriendlyName TieredPool -FriendlyName SSD_Tier -MediaType SSD
New-StorageTier -StoragePoolFriendlyName TieredPool -FriendlyName HDD_Tier -MediaType HDD
New-VirtualDisk -StoragePoolFriendlyName TieredPool -FriendlyName HDD_Parity -UseMaximumSize -ResiliencySettingName Parity -ProvisioningType Fixed -NumberOfColumns 4 -WriteCacheSize 100GB

100% read IOPs: 31,938.

Looks amazing! But hold on...logic tells me that the HDD pool didn't suddenly get faster than it was in its base-case. This is clearly a case of benching reads from the cache. So write speeds are decent (at least until we fill up a 100GB SSD-based cache). And reads are OK for recently-written data.

Also - note the odd look of the ATT0 bench. Played with this a bit and watched what was happening. The speed drop-offs appear to start when the cache starts flushing itself to the HDDs partway through the benchmark. Probably OK - but could cause some unstable performance under certain workloads.