Slow mirrored tiered storage spaces

[Morgan Simmons]

Hello

I currently have 4 128Gb Samsung 850 pro's and 4 seagate 1Tb in a tiered mirror storage space. The 4 seagate drives are housed in an Rackable SE3016 connected to a IBM 1015 in IT mode. The 4 samsungs are direct connected to the 1015.

My issue is that I'm not getting even remotely close to the speeds that I'm expecting from a tiered storage space with 2 columns.

-----------------------------------------------------------------------
CrystalDiskMark 4.1.0 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 2) : 437.819 MB/s
Sequential Write (Q= 32,T= 2) : 242.609 MB/s
Random Read 4KiB (Q= 32,T= 2) : 7.785 MB/s [ 1900.6 IOPS]
Random Write 4KiB (Q= 32,T= 2) : 31.533 MB/s [ 7698.5 IOPS]
Sequential Read (T= 1) : 87.875 MB/s
Sequential Write (T= 1) : 64.169 MB/s
Random Read 4KiB (Q= 1,T= 1) : 0.782 MB/s [ 190.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 1.650 MB/s [ 402.8 IOPS]

Test : 1024 MiB [E: 27.8% (581.9/2093.9 GiB)] (x3) [Interval=0 sec]
Date : 2015/08/25 22:57:16
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

Everything shows that the drives and the brand new ssd's are in good shape, and I created the space with the default tiered arrangements (1gb write cache, 2 column, mirrored)

I'm kinda stumped at the moment.

Thanks
Marshall

 

[cesmith9999]

please run get-virtualdisk <vdiskname> | fl

what is your column count and your write back cache size? and are you using all of your SSD? is your provisioning fixed or thin?

Chris

 

[Morgan Simmons]

Here ya go Chris

ObjectId : {1}\\SIMMONS-HYPER-V\root/Microsoft/Windows/Storage/Providers_v2\SPACES_VirtualDisk
.ObjectId="{8de3514f-38e4-11e5-80b4-806e6f6e6963}:VD:{05f014e1-4608-11e5-80c9-c8600
09b6844}{82c98f2b-4623-11e5-80ca-c860009b6844}"
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : 2B8FC9822346E51180CAC860009B6844
Access : Read/Write
AllocatedSize : 2248415379456
DetachedReason : None
FootprintOnPool : 4496830758912
FriendlyName : VirtualDisk
HealthStatus : Healthy
Interleave : 262144
IsDeduplicationEnabled : False
IsEnclosureAware : False
IsManualAttach : False
IsSnapshot : False
LogicalSectorSize : 4096
Name :
NameFormat :
NumberOfAvailableCopies :
NumberOfColumns : 2
NumberOfDataCopies : 2
OperationalStatus : OK
OtherOperationalStatusDescription :
OtherUsageDescription :
ParityLayout : Unknown
PhysicalDiskRedundancy : 1
PhysicalSectorSize : 4096
ProvisioningType : Fixed
RequestNoSinglePointOfFailure : False
ResiliencySettingName : Mirror
Size : 2248415379456
UniqueIdFormat : Vendor Specific
UniqueIdFormatDescription :
Usage : Other
WriteCacheSize : 1073741824
PSComputerName :

 

[Morgan Simmons]

Column count is 2, WBC is 1Gig. Full use of all SSD's and HDD, Fixed Provisioning.

 

[cesmith9999]

Can you rebuild the VDisk with –WriteCacheSize 20GB? and retest your results?

One of the ways that Storage spaces gets faster is with a large column count. the rest of the speed is due to the write back cache.

The Rackable SE3016 has a transfer limit of 1 GB. can you also try to directly connect the drives to your IBM 1015?

Chris

 

[Morgan Simmons]

I had the HDD's direct connected to the 1015 previously, but I didn't have any expansion options. I definitely will change the WBC to 20 gigs and give it a shot.

Doesn't the SSD's directly connected to the 1015 negate the speed of the 3016's 1GB limit? I know the HDD's arent hitting that speed currently by themselves (200 read and write by themselves) I figured that the SSD's would be mirrored and striped and be written to first, then shuffled to the HDD.

I had a similar setup with a work server that I set up, with 4 850's and 4 2tb drives all on a sas2 expander, and was easily getting 800mb read write sequential.

BTW, when I set add the 20GB WBC, is that 20 per ssd, or 20 total across all 4
I'll have to pull everything off the array, and rebuld tomorrow. I'll keep you updated.

 

[DavidRa]

You might also be seeing Storage Spaces attempting to optimise the use of the SSDs. As I understand it, SS detects streaming writes and sends the data direct to the HDD tier while the SSD tier and your 1GB write cache/journal is principally used for smaller random writes (e.g. metadata updates, normal random IO).

 

[Darkytoo]

Samsung SSDs don't get along very will with LSI RAID controllers. I've gonb back and forth with LSI tech support and the gist of the conversation is there a small amount of write cache in the SSD that does not have a battery backup, so LSI does not use it to prevent any chance of data loss. I would try benchmarking the drives directly connected to the controller with no storage spaces and see what happens.

 

[Morgan Simmons]

I will check both of these. thanks!

 

[Lance Joseph]

Did you ever say which filesystem you were using? NTFS? ReFS?
I've found that RAID-10 with tiered storage spaces on either FS performs rather poorly.
ReFS is also considerably slower than NTFS which is why I won't use it in a production scenario.

 

[Morgan Simmons]

Did you ever say which filesystem you were using? NTFS? ReFS?
I've found that RAID-10 with tiered storage spaces on either FS performs rather poorly.
ReFS is also considerably slower than NTFS which is why I won't use it in a production scenario.

I'm using NTFS. REFS and NTFS I thought were the only options.

 

[Chuntzu]

I have been trying to figure out this issue myself, the random perf should be better, sequential is right in line since anything over 64k block size goes directly to disk not ssd. Fujitsu has a good paper on this. But still not sure why random Iowa is not full line rate for the ssds.

 

[Lost-Benji]

Marshall, I have found ReFS to be around same speeds on double-Parity (RAID-6). I don't use SSD tiers though, classic drives and lots of them in each pool.

The biggest impact is RAM and the size of files being pushed to them. For small files with high I/O, Hardware RAID and SAS drives or SSD's otherwise, for bulk data, then SS is fine.

If the server has say 10GbE NIC's then you want a lot more than SS will ever do and the use of HW RAID is a must (or a linux RAID distro, not my cup of tea). If only using a GbE NIC, then SS is fine, just put a heap of RAM in and you're golden.

 

[Chuntzu]

Marshall, I have found ReFS to be around same speeds on double-Parity (RAID-6). I don't use SSD tiers though, classic drives and lots of them in each pool.

The biggest impact is RAM and the size of files being pushed to them. For small files with high I/O, Hardware RAID and SAS drives or SSD's otherwise, for bulk data, then SS is fine.

If the server has say 10GbE NIC's then you want a lot more than SS will ever do and the use of HW RAID is a must (or a linux RAID distro, not my cup of tea). If only using a GbE NIC, then SS is fine, just put a heap of RAM in and you're golden.

I have tested and verified mirrored spaces with 8 nvme drives and hit 23 gigabytes per second reads and 9 gigabytes per second writes. This same setup did 1.95 million read iops. With 48 ssds in mirror I hit 24 gigabytes per second reads and 12 gigabytes per second writes. Just for fun I did all 48 in a simple space and hit 24 gigabytes per second read and writes. This same setup 4k reads at 1.9 million iops and writes at 1.91 million. This while using ntfs, refs in server 2012r2 is slower when hiting small reads and writes. To add on top of this I just got done running some tests on connectx2 cards I had lying around and was able to saturate all 7 of them between two servers at 16 gigabytes per second reads and writes with smb rdma. Small 4k read and write iops were actually limited by the cpu in the single socket system at 800,000 iops with an e5-2670. So the need for a raid card is unnecessary if you are using enough disks and in the right layout. With this said maybe the current layout you are using would benefit from a raid card since your column count is a little low. Also try pinning file types to ssd tier ie vhdx files.

 

[Morgan Simmons]

So currently I have 12 Seagate Constellation.2 spinners and 4 Samsung 850 pro's. Mirrored tiered storage with a column count of 2.
It's honestly fast enough for what I'm doing, but I'm always trying to get as much out of the hardware as I can. I have 48Gigs of ram with dual x5650's so I think that the computer is fast enough.

I've never liked the freenas community (their forums are pretty horrendous), but maybe zfs might be a better solution.

I'm just running a homelab fileserver / plex / all on VM's. And only on a gig network, which I believe I am saturating.

Storage Spaces just doesn't work as I expected it to. I thought everything was going to go to the SSD's first, then be moved to the HD's when they haven't been access in a while.


Thanks everyone for keeping this thread going!

 

[Morgan Simmons]

This all REALLY makes me wonder if
A. I should drop the tiered storage and go for a straight mirrored pool on the constellation drives.
B. What would I do with my 4 SSD's

and finally
C. If WS2016 improves upon the performance, and if it would be worth tearing down the system to upgrade early. I know there are some changes that let you rebalance and re-stripe an array.

GRRR.

 

[Chuntzu]

So currently I have 12 Seagate Constellation.2 spinners and 4 Samsung 850 pro's. Mirrored tiered storage with a column count of 2.
It's honestly fast enough for what I'm doing, but I'm always trying to get as much out of the hardware as I can. I have 48Gigs of ram with dual x5650's so I think that the computer is fast enough.

I've never liked the freenas community (their forums are pretty horrendous), but maybe zfs might be a better solution.

I'm just running a homelab fileserver / plex / all on VM's. And only on a gig network, which I believe I am saturating.

Storage Spaces just doesn't work as I expected it to. I thought everything was going to go to the SSD's first, then be moved to the HD's when they haven't been access in a while.


Thanks everyone for keeping this thread going!

With mirrored and tiered column count of 6 could be reached if you added equal parts ssd to hdd, but even then you maybe better served with seperate ssd and hdd pools. I have only focused on s2d (storage spaces direct) in server 2016 and that will be a game changer, but I haven't focused specifically on single server changes to storage spaces so I can't speak to any changes there. Your use case with mirrored tiered storage spaces is something that I have been meaning to figure out for a while. I have played around with it but I haven't personally tested and confirmed how different arrangements work, but have read about them.

Examples of unexpected performance would be sequential speeds being low, I initially expected sequential speeds to that of the ssd tier if write back cache was enabled, but it turns out at or above (or perhaps just above) 64k block size writes bypass the cache and go directly to the spinning drives, so column count becomes key here. So having 4 ssds and 4 hdds sequential large block writes and possible reads (I haven't confirmed this) come directly from hdd tier only.

Other than that I would expect 4k random writes to hit write back cache. And as long as the files that require it are pinned to ssd tier then reads less than 64k should be at ssd tier line rate. Though I need to test my self to confirm.

 

[Lost-Benji]

I have tested and verified mirrored spaces with 8 nvme drives and hit 23 gigabytes per second reads and 9 gigabytes per second writes. This same setup did 1.95 million read iops. With 48 ssds in mirror I hit 24 gigabytes per second reads and 12 gigabytes per second writes. Just for fun I did all 48 in a simple space and hit 24 gigabytes per second read and writes. This same setup 4k reads at 1.9 million iops and writes at 1.91 million. This while using ntfs, refs in server 2012r2 is slower when hiting small reads and writes. To add on top of this I just got done running some tests on connectx2 cards I had lying around and was able to saturate all 7 of them between two servers at 16 gigabytes per second reads and writes with smb rdma. Small 4k read and write iops were actually limited by the cpu in the single socket system at 800,000 iops with an e5-2670. So the need for a raid card is unnecessary if you are using enough disks and in the right layout. With this said maybe the current layout you are using would benefit from a raid card since your column count is a little low. Also try pinning file types to ssd tier ie vhdx files.

Please don't take this as a shot at yourself as you clearly know plenty about SS, the above are nice benchmarks but really, screams, poor use of money. If you had enough coins for all those NVME drives and SSD's wich both have same limits of space compared to Spinners, then a Hardware RAID card would found its way in there long ago. SS still sucks, it is not suited to production environments and especially where VM's are concerned.

I use it because I have around 50TB hanging off it. It suits me well and the average files I am moving are around 1-10GB which move fine, even when it fill the RAM and then falls back to drive write speeds of 60 odd MB/sec (8x drive, double parity ReFS) for one array and the same style array with just single parity is 15MB/sec faster. The speeds when these were NTFS were only less than 5MB/sec different.

I won't bother with the arguments around how much of a pain in the arse SS is to use when it decides to have a shit in its pants. If I had coins for either a HW RAID card or more drives (need transfer space) then I would go HW or ZFS.

 

[Morgan Simmons]

completely off topic, but I wish that I could use ScaleIO with just one server, instead of having to use 3 seperate machines. It looks like a brilliant piece of software.

 

[cesmith9999]

ScaleIO is cool. however, the fault domain has to be at the server level and not at the disk level. that is why it requires a minimum of 3 servers.

Chris