Server 2012 R2, Storage Spaces and Tiering

[nickveldrin]

Thanks for all the info in this thread. It helped me along with several other articles get the performance I was looking for out of a storage server. I did end up upgrading to Win 2016 as there is a bit more support and tools (including rebalancing) for Storage Spaces using SSD tiering. Note these are 'Simple' tiers, not parity or mirrored. The SATA drives are on RAID6 at the storage controller level for redundancy.

My config is a SuperMicro Dual socket server (E5's) with 64GB of memory.
We installed 24 x 3TB WD Red Pro drives in the server, along with an NVMe Intel P3700 800GB cache drive.

Performance to start was kind of a joke (see first screenshot with inconsistent speeds).

I found an article explaining to set the default logical sector size to 512 for the pool, which along with upgrading to 2016 seems to have done the trick, the speed differences are completely off the charts. (see second image with consistent writes and reads off the chart). Thanks piglover and others for all the gathered tips and helping me get things healthy and happy.

What was your configuration of your storage pool in columns, raid type, etc? Performance looks very nice!

 

[m00t]

It is a one column simple space.. Not the recommended setup, but as mentioned, Storage spaces sees my two drives as single logical drives, not individual disks. Since I'm doing RAID6 at the hardware level (LSI2108 w/BBU and cache), Storage spaces just sees them as huge drives, and doesn't do any of the resiliency via mirroring or parity. I'm letting my controller card do the resiliency, in other words. Storage spaces is just taking care of the tiering. Also forgot to mention this server is running infiniband, I was seeing transfer speeds from my SQL servers of 22Gbit last night.. still think there is some more network performance to pull out of there as well.

FriendlyName          SerialNumber                     CanPool OperationalStatus HealthStatus Usage           Size
------------          ------------                     ------- ----------------- ------------ -----           ----
NVMe INTEL SSDPEDMD80 0000_0000_0000_0000.             False   OK                Healthy      Auto-Select 744.5 GB
SMC SMC2108           00af6c0da528c14c1e00af3a08800403 False   OK                Healthy      Auto-Select  76.4 TB

 

[cesmith9999]

Where did you set the pool size to 512? where is that link?

Chris

 

[m00t]

Where did you set the pool size to 512? where is that link?

Chris

New-StoragePool -FriendlyName (NewPoolName) -StorageSubSystemFriendlyName "Windows Storage on (Servername)" -LogicalSectorSize 512 -PhysicalDisks $PhysicalDisks

A bit more on why this is important in an SSD/HDD Tiered storage array. Hard drives come with a default sector size of 512, while SSDs are typically 4096. Creating the storage pool via the GUI will automatically use the logical sector size of the largest default in the pool. This creates, apparently, a lot of additional overhead.. wasn't quite expecting as much as it did, but it was a big bottleneck for my particular setup. The writes come in as 4096 and then have to be converted to 512 chunks.. not good.

Here's the article I read on it, more good info there, even though I'm not running VMs, it made a significant write difference: SSD’s on Storage Spaces are killing your VM’s performance

 

[cesmith9999]

that is a cool read. something to think about.

now with my POC lab starting to work... time to do some real testing.

Chris

 

[Diavuno]

who would have though 512 vs 4k would have a huge hit to speed/IO....

I need to check on mine now...

 

[Evan]

who would have though 512 vs 4k would have a huge hit to speed/IO....

I need to check on mine now...

Same on other file systems as well, been a DB issue since ever but this is the first I have heard of it for windows based systems not running DB's.

or is it just a storage spaces thing ?

 

[m00t]

Same on other file systems as well, been a DB issue since ever but this is the first I have heard of it for windows based systems not running DB's.

or is it just a storage spaces thing ?

Definitely.. We've always built our storage for SQL with the align= command.. didn't even think to do it for storage spaces, but it made a difference.

 

[cesmith9999]

aligning should not be an issue if you use GPT partitions. MBR is the partition that caused mis-alignment.

Chris

 

[m00t]

aligning should not be an issue if you use GPT partitions. MBR is the partition that caused mis-alignment.

Chris

Correct, guess it's habit from 10 years ago.

 

[Jetlag]

Interesting note: the disk seems to be short 1 disk worth of capacity. I expected ~18GB but it configured ~16GB (same as the prior test with single parity but 4 columns). Odd. I'll look into that later.

Did you ever discover why this happens? I have 8x5TB disks I got for media storage, and I thought dual parity might be called for, but giving up 3 disks kind of stinks. Maybe I'm not looking in the right places but I'm finding it difficult in even finding info on dual parity in Storage Spaces. Is anyone actually using it?

 

[PigLover]

Unfortunately no. For a whole lot of reasons I've stopped working with Storage Spaces (for the time being at least).

 

[JSchuricht]

It's due to the erasure coding Microsoft uses for double paridy. As I understand it the extra drive is used to make rebuilds faster. It does suck loosing an extra drive, I just went back to hardware RAID6 again from storage spaces.

 

[m00t]

I've found for me, in the notes above and my speed findings, that using RAID at the hardware level to increase space was the way to go for me, however I still used storage spaces in a 'simple array' so that I could leverage my P3700 Intel PCIe SSD as a caching device. It has worked FANTASTIC ever since I set it up, and I even pulled a drive from my array to make sure the hardware did what it was supposed to without causing storage spaces to fail. Worth a shot if you still want a fast front end cache for your array. I would never use storage spaces over a fast hardware RAID controller, though, if I wasn't using a cache device.

 

[Chuntzu]

The cache functions and real-time tiering in server 2016 have come a long way I am very impressed with the speed improvements. For parity and dual parity the speed and stability improvements are dramatic vs server 2012r2. I still use storage spaces in dual parity in server 2012 r2 and works great for me.

Sent from my SM-N920T using Tapatalk

 

[m00t]

Same here, I went to 2016 and it was a vast difference.

 

[nickveldrin]

For those that've tried it in 2016, did you guys do ReFS or stay with NTFS?

 

[m00t]

NTFS in 2016 here. I tested with ReFS and it was slower, but that was when I was testing in 2012 R2, haven't tried in 2016 with ReFS.

 

[Chuntzu]

Refs for me and it's great, a whole lot faster than it was in 2012r2

Sent from my SM-N920T using Tapatalk

 

[gigatexal]

Hmm time to test 2016