what your take on using ssd raid1 for system drive?

[vl1969]

Ok, some details are in order.
I am finally zero in on my final setup for a home vm/file server.
Just have a couple of questions about system drive setup.

1. What your take on using ssd as system drive?

Keep in mjnd this is a home server. Not production or very important stuff. Some down time is acceptable .

2. What your take on using ssd in raid1 config for system drive?

3. Same as #2 but regular hdd?

I plan to run open media vault 2.2.3 as main host. With a snapraid+mergefs.

I found a way to run it on raid1 mdadm raid.
So what should it be?

2 hdd in raid1
1 hdd only

2 ssd in raid1
1 ssd only.

Please give your thoughts on best way to go.

Thanks.


Ps. The system drives will be on mb sata ports the data on pcix controllers.

 

[Patrick]

I have been using USB drives in RAID 1 for FreeNAS. It has been working well in the 5 or so systems I have built like that.

 

[ttabbal]

In the past, I've done boot on mdadm mirrors using HDDs. It works fine. These days, I use ZFS mirrors, even with USB sticks on FreeNAS. I wouldn't hesitate to use SSDs for that as well, but they are more expensive, so unless I need the performance I generally reserve SSD for client machines. System boot drives in a server can be slow, if you're rebooting frequently, you're doing it wrong.

 

[T_Minus]

IMHO if you have the ports SSD is the way to go for RAID1.

With 64GB quality USB $25+ and 80gb S3500 $25-$30 if you have the ports def. a better choice to me. If you're using 16GB USB then ya def. 50% or so cheaper.

 

[ttabbal]

The USBs I'm using right now are about $20 for a pair of 32GB. They are pretty quick and work well, that said, I have gotten blocks fixed on a scrub. So yeah, if you can get a decent deal on good SSDs, do that.

 

[vl1969]

Guys I was talking about SSD not USB. Not even sure how to build raid on usb deives.
I have 2 120gb Intel ssd I can use for the server.
Or I can buy a pair of 500gb hdd or use a single 240 ssd or 500gb hdd.
My system does not work well with usb boot. So it was not even a consideration. I can try, the 32 or 64 gb flash are cheap enough but is it worth it for regular os?
I have a 24 bay enclosure so space is not an issue. But having a system drive outside does have it appeal.

My main issue is reliability, not speed. I plan to set up and forget it server. So reboots what ever they take are not an issue.

 

[EffrafaxOfWug]

My current systems are all dual SSDs in RAID1; as mentioned, purely for the reliability. However, I think there's an unavoidable roadblock looming in that the ludicrously complex UEFI seemingly doesn't support booting from a RAID1 pair because its design leaves a lot to be desired. The only way I could get my RAID1's to boot properly was to select CSM/BIOS boot and make sure UEFI is disabled.

Found this out after spending days on various configs; UEFI boot was still in its infancy at the point I built these systems and there was precious little in the way of documentation on the web, but the catch 22 basically plays out as follows (I could very well be wrong on all this so if I've catastrophically misunderstood how UEFI boot works then please correct me);

• Booting from UEFI requires a GPT partition table along with a small BIOS boot area
• The BIOS boot area needs to be formatted FAT32 and have a special partition ID
• But if you're making an mdadm RAID array (or indeed any other RAID array that needs a special partition), UEFI will not recognise the partition and thus will not boot from it
• Thus your only option is to create two of these FAT32 partitions on each of your boot discs to use for GPT boot
• But since those two identical partitions aren't RAIDed, whenever something changes the GPT boot stub thingy, it will only be updated on one device and GRUB will be only configured to point to one device
• There doesn't seem to be any way (at least in Debian GRUB2) to specify to update two or more GPT boot partitions, so you have to do it manually
• Since disk naming isn't constant (and you can't reliably use udev rules for RAID discs since things like serial numbers will change) you run the risk that if there's a disc failure, the GPT boot stub will become inaccessible. Demonstrated this by detaching one of the SSDs after a successful boot, rebooting, bam, straight into GRUB recovery. Makes RAID rather pointless.
• So I resorted to finding out which partition GRUB thought was the "active" one, and dd-ing that over to the "passive" one as a sort of poor-man's async RAID so that I'd at least have a bootable stub on each SSD that I could invoke manually if I absolutely had to.

After living with the above scenario for a couple of months I gave up babysitting it and went with good ol' BIOS with mdadm RAID1. You need to reconfigure GRUB via dpkg-reconfigure post-install in order to get it to update the MBR boot block on both SSDs whenever update-grub is called, but at least that bit works pretty flawlessly. I'm not likely to be booting off a >2TB SSD partition for some time now so personally I'll be sticking with CSM/BIOS boot until this gets satisfactorily resolved.

If anyone's found a reliable way of RAIDing and doing UEFI boot then I will personally award them seven trillion points and a large bowl of raspberry jelly if they were to outline their method here!

 

[vl1969]

My current systems are all dual SSDs in RAID1; as mentioned, purely for the reliability. However, I think there's an unavoidable roadblock looming in that the ludicrously complex UEFI seemingly doesn't support booting from a RAID1 pair because its design leaves a lot to be desired. The only way I could get my RAID1's to boot properly was to select CSM/BIOS boot and make sure UEFI is disabled.

Found this out after spending days on various configs; UEFI boot was still in its infancy at the point I built these systems and there was precious little in the way of documentation on the web, but the catch 22 basically plays out as follows (I could very well be wrong on all this so if I've catastrophically misunderstood how UEFI boot works then please correct me);

• Booting from UEFI requires a GPT partition table along with a small BIOS boot area
• The BIOS boot area needs to be formatted FAT32 and have a special partition ID
• But if you're making an mdadm RAID array (or indeed any other RAID array that needs a special partition), UEFI will not recognise the partition and thus will not boot from it
• Thus your only option is to create two of these FAT32 partitions on each of your boot discs to use for GPT boot
• But since those two identical partitions aren't RAIDed, whenever something changes the GPT boot stub thingy, it will only be updated on one device and GRUB will be only configured to point to one device
• There doesn't seem to be any way (at least in Debian GRUB2) to specify to update two or more GPT boot partitions, so you have to do it manually
• Since disk naming isn't constant (and you can't reliably use udev rules for RAID discs since things like serial numbers will change) you run the risk that if there's a disc failure, the GPT boot stub will become inaccessible. Demonstrated this by detaching one of the SSDs after a successful boot, rebooting, bam, straight into GRUB recovery. Makes RAID rather pointless.
• So I resorted to finding out which partition GRUB thought was the "active" one, and dd-ing that over to the "passive" one as a sort of poor-man's async RAID so that I'd at least have a bootable stub on each SSD that I could invoke manually if I absolutely had to.

After living with the above scenario for a couple of months I gave up babysitting it and went with good ol' BIOS with mdadm RAID1. You need to reconfigure GRUB via dpkg-reconfigure post-install in order to get it to update the MBR boot block on both SSDs whenever update-grub is called, but at least that bit works pretty flawlessly. I'm not likely to be booting off a >2TB SSD partition for some time now so personally I'll be sticking with CSM/BIOS boot until this gets satisfactorily resolved.

If anyone's found a reliable way of RAIDing and doing UEFI boot then I will personally award them seven trillion points and a large bowl of raspberry jelly if they were to outline their method here!

Ok
in order to do what I want to do I followed this guide <a href="Installing OpenMediaVault on RAID-1 array | Josip Lazić">here</a>


1. I have only tested this setup on my Hyper-V VM Gen 1 in windows 10. not a real hardware setting.
but am planning on doing a real setup ASAP. so let you know.
Gen2 VM do not work. could be UEFI or secureboot issue.

2. I think my hardware is old "Supermicro SC846 with MB=H8DME-2 Dual AMD Opteron Hex Core 2431 @ 2.4Ghz " so only have a bios mode not UEFI could be wrong but will see.


3. yes you need to recover manually but you can do this on a running system.
I am still playing with this setup so not sure what it takes to recover from a failed drive but I do know that as is both drives are bootable as I removed each system drive from VM and it booted in a degraded mode just fine.

also I tried to recover from failed drive and made a mistake. I was able to recover from that mistake and redo it properly on a live system. no shutting down (keep in mind it was a VM but still)

 

[EffrafaxOfWug]

Hmm. That approach is done on a running system and doesn't seem to have any FAT32 partitions but it doesn't seem to try to RAID those either. I'll try it in a VM at some point I think. Might be possible that dpkg-reconfigure grub-pc grub-efi can now be used to point GRUB2 at two GPTs instead of just one but it didn't work for me last time. It might just have been d-i hadn't been designed with it in mind and wouldn't let me go any further in the setup wizard, but it's an approach that seems poorly documented at the very best.

What do you mean you need to recover "manually"?

 

[vl1969]

Hmm. That approach is done on a running system and doesn't seem to have any FAT32 partitions but it doesn't seem to try to RAID those either. I'll try it in a VM at some point I think. Might be possible that dpkg-reconfigure grub-pc grub-efi can now be used to point GRUB2 at two GPTs instead of just one but it didn't work for me last time. It might just have been d-i hadn't been designed with it in mind and wouldn't let me go any further in the setup wizard, but it's an approach that seems poorly documented at the very best.

What do you mean you need to recover "manually"?

I mean, first that I do not know how to write scripts to do this via script or automatic
and second, that I actually have to go to CLI and run all those commands from there.
there is no GUI that I can use to do that.
I am doing this on an OpenMediaVault VM , as a test, before setting up my real hardware.
openmedivault has great WebUI to do a lot of things with data drives. I can setup and even recover raid(s) on data drives and more via WebUI
but system drives are totally off limits. I mean they are off limits if you want to keep running that is
you can surely screw-up the system in UI if you are not careful .

let me tell you something else, maybe in the past OMV was not this good and ready. but today
this is a great option for home file server and VM server.
I am going a bit overboard with setup, but if you keep it simple and straightforward it is perfect.

 

[unwind-protect]

RAID1 on identical SSDs is probably a bad idea.

If they are vulnerable to some specific write pattern that you do they would go down together, or at least the second one would die during the subsequent resync for a replacement. This has been observed in practice.

You are definitely better off using different drives, but myself I am still skeptical.

Depending on which OS you use there might be very little write activity on the boot and root filesystems. You should get paging space, /tmp and /var away from it anyway. That would greatly increase the robustness.

 

[EffrafaxOfWug]

The same could be said of installing on two identical platters as well. Moving swap, /tmp and /var away from an SSD would likewise be wasteful IMHO since they're the bits of the OS install that are likely to benefit from random access. If you think writing data to SSDs decreases their robustness then I think you might be buying the wrong SSDs.

Personally I've never had a problem with identical drives (SSD or otherwise) in RAID arrays, at least not since a dodgy HP firmware on some SAS arrays years back, but at home I always back up the system image from the SSDs onto the main data array just in case. New SSD, bootable linux distro and even if you somehow suffered dual simultaneous drive failure you could re-image and be back up and running in less than an hour. By all means use two different drives, just be aware that the array will only be as fast as its slowest link and you might get inconsistent performance.

Incidentally, from my metrics at least, if you ever start actually eating into swap (my home servers only run with a 1GB worth of swap because I've never had a load that went over my physical memory limit and if I did I'd likely want OoM-killer to nix it before it thrashed swap to death) but it's typically >90% reads; usually very little in the way of writes unless you're running your system under extreme memory pressure... in which case, buy more memory!

 

[T_Minus]

RAID1 on identical SSDs is probably a bad idea.

If they are vulnerable to some specific write pattern that you do they would go down together, or at least the second one would die during the subsequent resync for a replacement. This has been observed in practice.

You are definitely better off using different drives, but myself I am still skeptical.

Depending on which OS you use there might be very little write activity on the boot and root filesystems. You should get paging space, /tmp and /var away from it anyway. That would greatly increase the robustness.

What drives are you 'observing' this with?

This sounds like an edge case or a case of the wrong drive used for the application... I wouldn't want to use 2 different drives in raid1.

 

[vl1969]

no need to fight guys
I am planning to use different drives.
I have an intel 120 GB and SanDisk 120GB
similar specs different make.
so that is settled

PS: I do understand that there is a consensus over IT people to use a mixed drives in raid setup to prevent multiple drive failure.

 

[T_Minus]

no need to fight guys
I am planning to use different drives.
I have an intel 120 GB and SanDisk 120GB
similar specs different make.
so that is settled

PS: I do understand that there is a consensus over IT people to use a mixed drives in raid setup to prevent multiple drive failure.

I don't know anyone who'd suggest using mixed drive in a raid setup, and I'm curious which "IT People" you are talking about. As far as what I've heard, and people I've talked to there is no consensus to mix drives, in fact most suggest using the same. Maybe different batches but there's no rule of mismatching drives that is for sure.

 

[vl1969]

Not miss matched but different batches for hardware raid. For software raid miss matched is ok as long as drives are close in specs.
Software raid is less sensitive to different drives.

 

[unwind-protect]

Didn't the no-SSDs-in-RAID1 thing even get Slashdotted back in the day?

 

[vl1969]

Why not? Why can't we use ssd in raid1?

 

[unwind-protect]

Why not? Why can't we use ssd in raid1?

Because SSDs often die from specific write patterns, and in raid1 all drives execute the same pattern. That would mean they are more likely to die at the same time, which would in practice mean the first one degrades the array and the second one dies when you resync a new disk (never finishing the sync).

These are basic SSD properties.

To make matters worse, SSDs are far less likely to announce their intent ahead of failure in SMART.

 

[PigLover]

@unwind-protect, can you provide some references for the idea that SSD die from specific write patterns?

Sent from my SM-G925V using Tapatalk