Typical RAID6 performance?

[ninja6o4]

I have installed in a Supermicro 2U 12-bay CSE-826TQ case:
- Supermicro X10SLL-F mobo
- Intel Xeon E3-1246 v3
- 16GB ECC RAM
- 2x Samsung 840 Pro 256GB (onboard RAID1, boot OS)
- 10x Seagate ST6000NM0024 6TB SATA Enterprise drives, RAID6, 512KB Stripe Size
- Adaptec 6805 + Intel RES2SV240NC Expander
- Windows Server 2012 R2 Std

The array houses Plex media and as a backup destination from CrashPlan for a number of PC clients (~10).
I just upgraded from a 5x 6TB on an LSI 9271 and no expander.
I'm finding that streaming performance is terrible compared to before. On the 5x RAID5, there was no problem for upwards of 12 streams at once. In this configuration, I cannot even handle 3, and when I check the resource monitor, the disk activity is sits at 100% with Disk Queue showing anywhere from 3 to 10+, highest response times of 1500ms at times.

Here is the current configuration of the array:



I ran Anvil to benchmark it, but honestly I'm not sure what is a good reading.
Can anyone tell me what is a fairly typical reading for a RAID6 array?

Here are my results:

 

[j_h_o]

I had a 6805 against an LSI expander and saw similar issues (very poor i/o) on 2012 R2. Latest firmware, latest driver, etc.

I switched back (after a painful migration) to 9750-8i and things went back to normal.

 

[ninja6o4]

I had a 6805 against an LSI expander and saw similar issues (very poor i/o) on 2012 R2. Latest firmware, latest driver, etc.

I switched back (after a painful migration) to 9750-8i and things went back to normal.

Thanks for your input. I ran Anvil in a variety of configurations and could not seem to get any significantly improved results. So I believe your theory of the 2 cards being incompatible are sound.

Here are my results from a 2 disk RAID0:


And here is a result of a 10 disk RAID50:


Just for fun here is a 10 disk RAID0:

 

[j_h_o]

I wrote up something about this at the time, but I never finished gathering all the data:

http://justinho.com/blog/adventures-with-hardware-raid-cards-for-mass-storage

 

[Keljian]

While not a benchmark, in general with a 7 disk raid 6 setup I get 120-140mb sustained writes, and 400-450mb sustained reads. I could probably do better with some tuning but this is plenty fast enough for me

I suspect the poor writes are due to a small stripe size.

 

[j_h_o]

Are you seeing "better" performance with a R5 array in that exact same config?

Because the perf issues I saw weren't array type specific, as far as I could tell. My R1 and JBOD containers were just as terrible.

 

[archangel.dmitry]

I get 440/460 sequential R/W with 1 GB file (CrystalDiskMark; RAID6 4x WD Red; Areca), and 450/490 with 2GB file.

 

[5teve]

I cant actually get anvil working on my win8.1 machine.. but using a HP p822 2gb card connected to 12x 2tb Ultrastar SATA drives.. i get (using Crystaldisk) - the HP can pretty much saturate the disks for sequential write

1gb file (uses cache extensively as you can tell )

-----------------------------------------------------------------------
CrystalDiskMark 4.0.2 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 6317.629 MB/s
Sequential Write (Q= 32,T= 1) : 3247.102 MB/s
Random Read 4KiB (Q= 32,T= 1) : 550.757 MB/s [134462.2 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 526.835 MB/s [128621.8 IOPS]
Sequential Read (T= 1) : 1318.161 MB/s
Sequential Write (T= 1) : 1210.314 MB/s
Random Read 4KiB (Q= 1,T= 1) : 205.638 MB/s [ 50204.6 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 163.197 MB/s [ 39843.0 IOPS]

Test : 1024 MiB [E: 14.8% (2766.3/18629.7 GiB)] (x3)
Date : 2015/11/06 9:37:05
OS : Windows 8.1 [6.3 Build 9600] (x64)

4gb File

CrystalDiskMark 4.0.2 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 1981.473 MB/s
Sequential Write (Q= 32,T= 1) : 1381.234 MB/s
Random Read 4KiB (Q= 32,T= 1) : 20.891 MB/s [ 5100.3 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 7.751 MB/s [ 1892.3 IOPS]
Sequential Read (T= 1) : 1.887 MB/s
Sequential Write (T= 1) : 180.107 MB/s
Random Read 4KiB (Q= 1,T= 1) : 0.302 MB/s [ 73.7 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 84.364 MB/s [ 20596.7 IOPS]

Test : 4096 MiB [E: 14.8% (2766.3/18629.7 GiB)] (x3)
Date : 2015/11/06 9:41:18
OS : Windows 8.1 [6.3 Build 9600] (x64)


Dunno if that helps or hinders

Steve

 

[ninja6o4]

Thanks for all the feedback so far. So my 10x RAID6 has fully initialized on my LSI 9271-8i+expander now. Here are the results:





Not sure if these numbers are typical? They are definitely better than Adaptec, though I seem to be hitting a wall in terms of max seq throughput. Still, I am more concerned with the random i/o numbers and making sure they are in line.

 

[ninja6o4]

@j_h_o thanks to your initial tip, I am back in business.
On the Adaptec 6805 + expander, I could not even managed 3 streams, and had 100% disk activity the entire time.
On the LSI 9271 + expander, I saw up to 9 streams yesterday, and 0.0% disk activity (only the occasional spike 10-30% when starting a new stream.)

One thing I did not get to try was configuring the Adaptec with a Full Initialization, like I did with the LSI. I wonder if maybe the Adaptec was running in a slow/failsafe manner until the RAID were verified?

Anyway, it's up, and everything is as it should be, so I'm not screwing with it anymore. Pulling out the Adaptec today and posting it for sale.