10x 3TB in Raid 6. 33% of the performance of a single drive?

[Triggerhappy]

Setup:
System: Xeon E5-2609 in a CSE846 chassis with EL1 expander backplane.
Raid Card: IBM M5014 with BBU flashed to LSI 9260 with Advanced Feature Key.
OS: Server 2012 R2 Essentials virtualized in Server 2012 Standard. Virtual Disk directly passed through.
Drives: 10x Seagate Constellation ES.2 3TB SATA version in Raid 6 config (24TB usable space)

This is a storage server that feeds the home. Holds all our pictures, videos, documents.

Migrated this from a 8x Raid0 that was running for 3 years without issue in another chassis where the raid0 had manual robocopy to a 2x3tb software raid0 of the files I'd die if we lost (would never lose them, offsite backup and all).

The performance of the raid 0 was about 700MB/s on read and write. Far more than I needed as the largest link I had was a dual gbit teamed link to my main workstation.

When I migrated the chassis, I had the M5014 ready for the upgrade in order to get some piece of mind in reliability on both the critical and non critical files. I now have the raid 6 array (which took a whole week to live migrate).

I knew I would lose some performance on both the read and write side of things, but I figured what I'd drop to was maybe 1/2 or 1/3 the performance of the raid0, which would be quite acceptable and still capable of saturating all my links. What happened was my performance dropped to about 30-50MB/s write and still maxxed read. Running ATTO I got roughly the same numbers as direct file copy.

Single Drive: 110-150MB/s read and write
Full Array: 30-50MB/s Write, 300 - 700MB/s read.

My questions are:
Is this sort of performance loss with Raid 6 expected?
If not, what's the bottleneck? M5014 since it doesn't have enough cache (256MB)? Lack of drives? OS? Configuration of the Raid6? The fact that I live-migrated it from Raid0? CPU?

I don't need IOPS, I don't need super mega speed, but i'd like to maximize available capacity while being able to keep the links saturated on a single continuous file transfer.

Any insight would be appreciated.

 

[azev]

Do you have a write back cache enabled ?
Enabling write-back cache will speed up your write speed, but you want to have BBU for power lost protection.

 

[Triggerhappy]

That seems to have put me into the 500MB/s range. Thanks! (Yes, BBU is installed)

Followup: My understanding of write-back cache was that it'd use the onboard memory on the controller to cache the writes. Since it only has 256MB onboard I assumed this would be more or less useless on any larger files. Why on ATTO does this still give me 500MB/s writes even when doing 2GB tests?

 

[PigLover]

It's a bit more complicated than thinking of the cache as a simple space that will fill up quickly. Its not reasonable to write all about it here - the short version is that every write to a parity raid requires read & write of all drives and without the cache you have to do this for every small write. It is inefficient. The presence of a battery backed or flash based cache allows you to do a number of 'write' actions before anything actually has to be written to disk. This allows disk writes to be done in much more efficient chunks. Even a relatively small cache can have massive impact without 'filling up'.

Bigger cache is still better.

 

[TuxDude]

No modern RAID engine should be touching all of the drives for a write to RAID-5/6 unless it is a very small array (only 3/4 spindles). But the basic idea is correct - you need to do a lot of extra work to write a single block to a parity-raid array. In the case of RAID-6 to write one block, first you must read that block and both associated parity blocks, calculate the new parity by comparing the old data block with the new data, and then write the new data and both parity blocks back out to disk.

The only time writes are acceptably speedy on parity raid is when you are doing a 'full stripe write' (you can google that for lots more information), which is what that cache is allowing you to do. When you have enough data cached to write a full stripe across every disk in your array you don't need to worry about doing any reads first - you already have all of the data in cache that you need to calculate the parity. You can just do the calculation, and issue the writes for all of the data blocks plus the parity block. For large sequential transfers your cache size only needs to be big enough to hold a full stripe in order to get this performance benefit. If you start doing small-block random write IO to the array, then you are going to need a MUCH larger cache if you hope to ever get enough data to fill an entire stripe, and it probably still won't happen often if at all.