To hardware RAID or not to hardware RAID (i.e. software)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

perdrix

Member
Mar 3, 2016
75
7
8
71
The Adaptec ASR-51245 RAID card turned up and I've got the following attached:

SAS 4TB
SAS 4TB
SATA 750GB
SATA 500GB

I'm thinking to create one RAID1 from the two 4TB drives, and another combining the two SATA drives (as 500GB each), and then join then using LVM.

So should I:

a) Set up the drives as just JBOD and use software RAID (Ubuntu 14.04 LTS Server), or

b) Use the card to configure the two RAID1 arrays ?

the hardware is an Intel Atom 525 with 4GB.

I'm leaning towards hardware, but am open to good reasons for not doing so.

Thanks
Dave
 

perdrix

Member
Mar 3, 2016
75
7
8
71
My - it's a while since I heard that expression (normally from my own lips)!

The environment is a NAS/Media Server for home network.

NAS will store digital photos and system backups.

Dave
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
I'm leaning towards hardware, but am open to good reasons for not doing so.
With hardware RAID, a controller failure means you need to scramble to find a compatible controller to access the drives, or deal with futzing sector 0 offsets to find the data if you don't have a compatible controller. Some controllers put the metadata at the beginning of the drive, shifting the user data, while other controllers put it at the end of the disk. You don't want to have to find out which your controller used after the controller dies. And of course, for anything more complex than mirroring, you have to deal with stripe sizes, etc.

With software RAID, if you lose the "first" drive (from the system's point of view) the system may not be able to boot properly. Consider the case of a pair of drives with software mirroring. One of the drives is the one the BIOS will load the boot block from - to the BIOS, the second drive is "just some other drive". So you might need to boot from a recovery media (CD/DVD/USB stick) and then access the remaining drive. Some in-between RAID ("host-assisted RAID", "BIOS RAID", etc.) solutions put enough knowledge into the BIOS that the system will be able to boot from another RAID member if the primary boot drive fails.

With modern systems, the decision doesn't matter nearly as much for smaller configs like yours. If you had enough drives to need multiple controllers, that would lean toward software RAID (multi-controller hardware RAID is a rather thorny situation that fortunately doesn't come up much any more). If you wanted to use ZFS or similar, that would also tip the scales to software. If you had a lot of drive space but not much RAM, that would tend to suggest hardware RAID instead of a solution like ZFS.

Note that unless a controller offers a "dumb" mode (which LSI calls IT mode), the controller may put metadata on the drives anyway, even if you are making the individual drives available to the operating system for software RAID. That's one of the big reasons you see a lot of discussion here about flashing a controller to IT mode.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Ok, well, if this is a media server and general NAS, as well as reliability, you'll also be wanting capacity, especially going forward. Using it to hold your system backups also warrants some redundancy. Creating a couple of mirrors, with what you currently have, and managing them with LVM is about the best plan, but a better route might be to sell on the 750GB & 500GB, or if they are 2.5", use them in an external USB enclosure for quick temporary storage, and buy another 2 x 4TB SAS disks. You could at least use Raid6 now moving on to Raid60 later. As for the hardware vs software raid question, adaptec raid cards are good at what they do. You have a lowly D525 at your disposal, which is a decent enough little CPU for serving media etc, but it's not going to win you any medals performing lots of xor calcs, so offloading that to a hardware solution such as an adaptec card is prolly a good idea. I personally prefer using software raid now whenever I am able to. Modern CPU's, a little memory coupled with modern file systems places it head and shoulders above a hardware based solution now in my opinion, however in saying that, there are times when a hardware solution is the better road to take, and I think, given your system limitations, this would be a good choice this time round :)
 

perdrix

Member
Mar 3, 2016
75
7
8
71
Thanks pricklypunter! I tried it both ways with a good I/O load and the CPU was running 40-50% just running the RAID in software. Doing it in the hardware, CPU load was close to nil. So I'll go with hardware RAID and live with the risk of needing to replace the controller with a compatible one if it dies.

Buy another couple of 4TB SAS drives - maybe when my finances recover from buying the first two!!!

Cheers, Dave
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Buy another couple of 4TB SAS drives - maybe when my finances recover from buying the first two!!!

Cheers, Dave
See, now there's your issue right there. You should have bought 4 to start with, that way in the interests of efficiency, you only need to complain that you're broke once ;):D
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
The only advantage hardware raid has over software raid today is that it requires less iops for writes, as it can do all the paritity io-operations on the controller memory and write data sequentially to the disks.

The paritity calculation itself is so cheap that a modern CPU wouldn't even sweat at 1GB/s.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
The only advantage hardware raid has over software raid today is that it requires less iops for writes, as it can do all the paritity io-operations on the controller memory and write data sequentially to the disks.

The paritity calculation itself is so cheap that a modern CPU wouldn't even sweat at 1GB/s.
Software or hardware RAID both require the same number of IOPS against the physical drives. Using the example of a raid-5, unless you are doing a full-stripe write, the efficient thing to do is read the old data + parity, use that with the new data to compute the new parity (trivially simple for raid-5, some math involved for raid-6 but still not bad), and then write the new data + parity. Thats 4 IOPS for a single write operation, and (unless a full-stripe write) they must all be done regardless of it being hardware or software. Hardware RAID does have the advantage that only the original single write operation has to travel over the PCIe bus to the RAID card - the card can then do the 4 IOPS and the math without using up any additional system bandwidth.

If doing full-stripe writes, again software and hardware end up getting the same benefit - already having all of the data for a stripe available lets them skip doing any reads, calculate the new parity from the data for that stripe, then write all of the data + the parity - so just a single extra write operation for the parity on top of the data writes.
 

mstone

Active Member
Mar 11, 2015
505
118
43
46
Software or hardware RAID both require the same number of IOPS against the physical drives. Using the example of a raid-5, unless you are doing a full-stripe write, the efficient thing to do is read the old data + parity, use that with the new data to compute the new parity (trivially simple for raid-5, some math involved for raid-6 but still not bad), and then write the new data + parity. Thats 4 IOPS for a single write operation, and (unless a full-stripe write) they must all be done regardless of it being hardware or software. Hardware RAID does have the advantage that only the original single write operation has to travel over the PCIe bus to the RAID card - the card can then do the 4 IOPS and the math without using up any additional system bandwidth.
The potential IO reduction comes from NVRAM, which is common in hardware RAID solutions and more rare (though possible) in software RAID configurations. The PCIe bandwidth thing cuts both ways; it's generally possible these days to get more aggregate disk bandwidth from software RAID (by using more channels) than hardware RAID (which has a bottleneck at the hardware RAID controller).
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
Sounds like you're talking about a RAID card using NVRAM as a write cache, which is the only way you could ever end up doing less IOPS. I've never seen a RAID card with actual NVRAM for its cache, as that would be very expensive - instead they all use regular DRAM combined with a battery or some other similar technology to save the contents of the write cache in the event of an unexpected power loss. Flash-backed-cache is popular lately, with a large capacitor for power and some NAND to quickly back up the DRAM cache in an emergency.

If you have a big enough write cache, and if the writes you are doing against the array are not sequential but are very close together, then its possible that the RAID engine will be able to combine some of those into a full-stripe write before they are sent to the spindles and save a few IOPS. But cache is not permanent storage; the RAID engine can't sit around forever waiting/hoping to get the data its needs to write a full stripe - most of the advantage of the cache is that the RAID controller can respond to the OS with a successful write immediately and deal with the read-modify-write cycle of parity-raid on its own time. It makes things look a lot faster but generally doesn't save that many IOPS.
 

mstone

Active Member
Mar 11, 2015
505
118
43
46
Sounds like you're talking about a RAID card using NVRAM as a write cache
Yes, I thought that was clear. :) It really depends on your usage patterns, and whether you're doing sync or async writes. RAID5/6 (with a read and rewrite penalty) isn't the only relevant case, it's also possible to boost IOPS by caching writes to a mirrored volume (RAID1/etc, with no read and rewrite patern) and lowering the latency to persistent storage--this allows more ops if you are doing synchronous writes and can't issue a new operation until you receive confirmation that the previous one completed. There are enough workloads these days that aren't sensitive to synchronous random write latency that avoiding HW RAID is a rational choice for a very large number of people. SSDs have also made it possible to get reasonable performance without the NVRAM in a lot of scenarios. And then on the other extreme are people sensitive enough that the latency penalty of the HW RAID controller compared to direct NVMe SSD is too high.

At any rate performance isn't really a selling point for most hardware RAID, and hasn't been for a long time. At the internal storage level the biggest benefit of integrated HW RAID is that you can generally get away with a somewhat more clueless junior person handling drive replacements. (Look for the orange light, swap that drive out, done.) In some environments that's worth its weight in gold. That's not to say its impossible to get to the same place without HW RAID, but if for a couple of bucks at the time of purchase you can ensure a reasonably simple/sane solution it may be worth it.
 

uOpt

Member
Dec 12, 2014
43
7
8
Boston, MA, USA
I'm strongly in the software raid camp.

Let's leave aside post-RAID like ZFS's which is more complicated but clearly not hardware raid. Let's stick with traditional raid that has no idea what the filesystem is doing, that has the raid hole etc.

My reasons for liking software raid like Linux' md in particular:
  • Can combine drives across controllers
  • Don't have to learn new recovery procedures. Trust me, you screw up every now and then. You need to run a fire drill with a new raid system.
  • What do you do if your controller fails? You need a second one. Which lives on the shelf until it usefulness has expired because it is obsolete.
  • Modern single-disk controller interfaces and the disks support useful command ordering, so that in combination with today's filesystems you are quite safe against losing the entire filesystem even if you lose power in the middle of a write. This is entirely different from the elder days where e.g. Linux ext2fs simply ran with asynchronous metadata by default. WTF. Check up FreeBSD's softupdates and how it builds a dependency graph. In fact due to the raid hole you are probably less safe with a hardware controller no matter what you do. Yes it has a battery but it has no idea what the filesystem is doing where.
  • Obviously there's plenty of CPU power.
  • What if you move to a computer that doesn't take your old controller or one that uses the same on-disk format?

IMHO the hardware raid industry committed suicide when they didn't adapt the common on-disk layout proposal. They might have done that to protect their own interest and lock the customers in. But people like me are not going to create any raid that can't be moved. Just imagine how long it takes to copy over an array made from 5 TB 7200 rpm disks. And that's assuming you have a place to copy to. If you can just move the drives, why not?
 
  • Like
Reactions: leebo_28 and Bert

Bert

Well-Known Member
Mar 31, 2018
823
383
63
45
Great discussion! Can we set ASR-51245 into HBA mode? I read somewhere that even in JBOD mode the Adaptec card makes the disk not reachable by other controllers but when I tested it, I can load the JBOD disks via USB host.
 
Last edited: