Linux Software RAID Guide

[rubylaser]

Nope - it it included in many/most debian/ubuntu based distributions, but is neither available or easily added to Fedora, RHEL, CentOS, etc. where you must use a 3rd-party kernel (or compile your own) to get AUFS support. Not available in Gentoo either without some extra work which is what I'm running on my snapraid box (I use mhddfs for pooling).

And @OBasel is right about it having a low chance of being upstreamed. Especially since OverlayFS was merged into the kernel in 3.18 - if I was building a new snapraid system now I would be looking at OverlayFS.

I didn't realize that. I'm sure there are third party repositories that add kernels that support AUFS. I compile my own kernel and AUFS on Ubuntu right now anyways so that I can use a 4+ kernel and AUFS4. It's not that challenging, but I understand this is a big step for most users. mhddfs doesn't work well for me because of the disconnects that occur from large transfers or rsyncs.

Thanks for the comment, but Overlayfs isn't a solution for a media server. It is one filesystem laid on top of another. And the "lower" filesystem is always read-only. As result, it doesn't meet the needs most people would be using SnapRAID for, a fileserver. The primary use of it are livecd/liveusb distros. If you know if a different way to use Overlayfs that overcomes these shortcomings, I'd love to have a third option for pooling.

 

[Martin Jørgensen]

My log says

This morning i were done sync, then i came back from work and made a reboot

Jun 3 07:41:10 Anthea mdadm[1232]: RebuildFinished event detected on md device /dev/md0
Jun 3 17:31:37 Anthea mdadm[1320]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 17:59:33 Anthea mdadm[1120]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:03:57 Anthea mdadm[1189]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:09:07 Anthea mdadm[1366]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:15:42 Anthea mdadm[1337]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:21:35 Anthea mdadm[1441]: DeviceDisappeared event detected on md device /dev/md0

 

[rubylaser]

Hey thanks for your repley

'cat /proc/mdstat says noting after reboot
unused devices: <none>

Before reboot its was completing done sync.

Lsblk shows md0 under each disk before reboot.

mdadm --assemble --scan dont have anything

Have you verified that your disks have superblock metadata on them?

mdadm -E /dev/sd[de]

 

[TuxDude]

My log says

This morning i were done sync, then i came back from work and made a reboot

Jun 3 07:41:10 Anthea mdadm[1232]: RebuildFinished event detected on md device /dev/md0
Jun 3 17:31:37 Anthea mdadm[1320]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 17:59:33 Anthea mdadm[1120]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:03:57 Anthea mdadm[1189]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:09:07 Anthea mdadm[1366]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:15:42 Anthea mdadm[1337]: DeviceDisappeared event detected on md device /dev/md0
Jun 3 18:21:35 Anthea mdadm[1441]: DeviceDisappeared event detected on md device /dev/md0

That looks odd - I don't know why it would be detecting a device dissapearing, and I also don't know how 6 devices could go missing from a 2-device array.

If you could post the contents of your /etc/mdadm/mdadm.conf file and also the end of the output from dmesg (or if its relatively short from a fresh reboot the entire thing) I might be able to help you more. You can also try running 'mdadm --examine \dev\sdx' (replace x as needed) and see what it tells you about your drives.

 

[TuxDude]

I didn't realize that. I'm sure there are third party repositories that add kernels that support AUFS. I compile my own kernel and AUFS on Ubuntu right now anyways so that I can use a 4+ kernel and AUFS4. It's not that challenging, but I understand this is a big step for most users. mhddfs doesn't work well for me because of the disconnects that occur from large transfers or rsyncs.

I'm not a fan of 3rd party kernels. Not that it would matter for home use, but I've gotten used to sticking with distro-provided packages at work. When you call up RedHat support with a problem they aren't going to stay on the line much longer after they learn you've swapped out their kernel. I don't have any support contracts to worry about at home, and don't use snapraid at work, so its only semi-relevant, but at least for me still a strong preference not to replace core bits of the distro - I'd swap to a distro that supports a required feature for that one server/VM instead.

I haven't had any issues with mhddfs with large files/transfers/rsync, and I've hit it with some good ones. I did have a bunch of problems when I had the mhddfs pool exported as an NFS share, but switching over to samba / CIFS protocol has resolved that for now. Long-term I would like to move away from snapraid+pooling and go to a btrfs raid-6 and then I can go back to NFS as well.

Thanks for the comment, but Overlayfs isn't a solution for a media server. It is one filesystem laid on top of another. And the "lower" filesystem is always read-only. As result, it doesn't meet the needs most people would be using SnapRAID for, a fileserver. The primary use of it are livecd/liveusb distros. If you know if a different way to use Overlayfs that overcomes these shortcomings, I'd love to have a third option for pooling.

I haven't looked at either AUFS or OverlayFS in detail - but I know that both are solutions for layering one filesystem on top of another, and that AUFS is also often used for LiveCD/USB setups. In fact behind Docker (can use AUFS to add up the layers to build a container) live distro's are probably AUFS's next most common use. If OverlayFS can't (yet) handle multiple writable layers then you're right - it wouldn't work for this use case.

It's too bad mhddfs doesn't have a good developer wanting to make it a native filesystem instead of FUSE based. That would fix quite a few issues and make the performance a lot better.

 

[rubylaser]

I'm not a fan of 3rd party kernels. Not that it would matter for home use, but I've gotten used to sticking with distro-provided packages at work. When you call up RedHat support with a problem they aren't going to stay on the line much longer after they learn you've swapped out their kernel. I don't have any support contracts to worry about at home, and don't use snapraid at work, so its only semi-relevant, but at least for me still a strong preference not to replace core bits of the distro - I'd swap to a distro that supports a required feature for that one server/VM instead.

I haven't had any issues with mhddfs with large files/transfers/rsync, and I've hit it with some good ones. I did have a bunch of problems when I had the mhddfs pool exported as an NFS share, but switching over to samba / CIFS protocol has resolved that for now. Long-term I would like to move away from snapraid+pooling and go to a btrfs raid-6 and then I can go back to NFS as well.



I haven't looked at either AUFS or OverlayFS in detail - but I know that both are solutions for layering one filesystem on top of another, and that AUFS is also often used for LiveCD/USB setups. In fact behind Docker (can use AUFS to add up the layers to build a container) live distro's are probably AUFS's next most common use. If OverlayFS can't (yet) handle multiple writable layers then you're right - it wouldn't work for this use case.

It's too bad mhddfs doesn't have a good developer wanting to make it a native filesystem instead of FUSE based. That would fix quite a few issues and make the performance a lot better.

I would agree with all of this. I don't use custom kernels at work, but I love tinkering at home in a VM with snapshots. I've experienced disconnects myself along with a fair number of other commenters on my site with mhddfs. I also really look forward to btrfs RAID6 in the future, once other people have ventured through the problems that I'm sure will arise as bugs are removed from the code and more users are trying it out.

+1 If mhddfs where still being actively developed and it could run as a kernel module instead of via FUSE, it would be the best solution, because it just works.

 

[Martin Jørgensen]

Result from mdadm -E /dev/sd[de]
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sde:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)

Results from mdadm --examine \dev\sdd
mdadm: cannot open devsdd: No such file or directory

Does i have something to do with labeling? i try this parted -s -- /dev/sde set 1 raid on
but i have no partitions so no result.

I dont need to have partitions when i create the raid with mdadm right?

 

[Martin Jørgensen]

My mdadm.conf
ARRAY /dev/md0 metadata=1.2 name=Anthea:0 UUID=57387098:44c8ff8f:c6c89a08:84004837

 

[TuxDude]

Oops, I guess I had backwards \'s in my earlier post - \dev\sdd should have been /dev/sdd. You got the info anyways though, as '-E' is the shortcut for '--examine' - That output is showing that you do not have any MD superblock on the raw disk devices, and that there is an MBR partition table with a single partition defined on the disk. What do you get if you run 'mdadm --examine /dev/sdd1'?

Your mdadm.conf also looks a bit short - you should probably have a SCAN line above that one to tell it which devices to scan looking for arrays.

 

[Martin Jørgensen]

Hey TuxDude i really appreciate your help

mdadm --examine /dev/sdd1 -
mdadm: cannot open /dev/sdd1: No such file or directory


These disk i had in a synology box, but got them formated by gparted from a live disk

 

[Martin Jørgensen]

lsblk shows no partitions on sdd and sde

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111,8G 0 disk
├─sda1 8:1 0 103,8G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 8G 0 part
sdb 8:16 0 1,8T 0 disk
├─sdb1 8:17 0 1,8T 0 part
├─sdb2 8:18 0 1K 0 part
└─sdb5 8:21 0 12G 0 part
sdc 8:32 0 1,8T 0 disk
├─sdc1 8:33 0 1,8T 0 part
└─sdc9 8:41 0 8M 0 part
sdd 8:48 0 3,7T 0 disk
sde 8:64 0 3,7T 0 disk

 

[TuxDude]

Thats very odd. I ran 'mdadm --examine' against a couple of drives in my desktop at work - one is a member of a multi-drive btrfs using the raw drives, the other is partitioned (MBR layout) into a few ext4 filesystems - both have no MD superblock (I don't have an md array to play with at work to show sample results).

On the drive that is used for btrfs, examine results in: "mdadm: No md superblock detected on /dev/sda."

On the drive with the MBR partitions formatted as ext4, examine results in:
/dev/sdb:
MBR Magic : aa55
Partition[0] : 1024000 sectors at 2048 (type 83)
Partition[1] : 12337152 sectors at 1026048 (type 82)
Partition[2] : 104857600 sectors at 13363200 (type 83)
Partition[3] : 381897392 sectors at 118220800 (type 05)

That second result matches up with the results you posted in reply #27 (size and number of partitions differ) - so why would the kernel think that there are no partitions on your drive, yet mdadm says that there is a partition. Something funny going on somewhere. Is there anything interesting in dmesg output?

 

[Martin Jørgensen]

I have now created 1 partition on each drive, and start making md0
sdd 8:48 0 3,7T 0 disk
└─sdd1 8:49 0 3,7T 0 part
└─md0 9:0 0 3,7T 0 raid1
sde 8:64 0 3,7T 0 disk
└─sde1 8:65 0 3,7T 0 part
└─md0 9:0 0 3,7T 0 raid1

Now i will let it settle and try reboot agian

 

[EffrafaxOfWug]

My go-to doc for that is: the Linux Software RAID Guide on the Ubuntu forums. Pretty good reference for someone getting started.

One thing that it missed that temporarily bit me in the arse a couple of months back was the process of removing a (still functioning) disc and replacing it (and no, couldn't fail/remove the device beforehand since on a non-SGPIO backplane you can't neccesarily see which device has been kicked out of the array and in debian at least there doesn't seem to be a way to configure udev rules as such to label devices on a consistent basis such as by SCSI port); most cheat sheets will tell you to go:

mdadm --manage /dev/md123 --remove /dev/sdx1

...but since the device has been hotplugged out you get an error because /dev/sdx1 no longer exists. What you actually have to do to remove non-existent discs from an array so that you can add a new one (i.e. without having to stop the array and reassemble with the new drive(s)) is use the following:

mdadm --manage /dev/md123 --remove detached

This will remove any device in the array that is no longer present and then you'll be able to add the new disc:

mdadm --manage /dev/md123 --add /dev/sdy1

Something had obviously changed in the hotplug code in between the last time I did the same thing on debian that causes discs to be removed completely. In case anyone wonders why I was pulling out drives from an active array, this was a simple capacity expansion (i.e. replacing an x TB drive with a 2x TB drive one by one [actually two at a time] and awaiting a resync then growing the array).

 

[TuxDude]

FYI, if you are swapping drives out with larger ones and you have a free slot or two in your server, there is a much faster and safer way to do it. I left easier out of that list because at least last time I did it the user-space mdadm tool still doesn't support it but all the kernel bits are there. The general idea is that instead of removing a drive and adding a new larger one then rebuilding, which can be slow depending on RAID type and leaves your array in a degraded state until it is complete, you add the new larger drive first as a hot-spare and mark one of the smaller drives as wanting to be replaced. When there is a spare available and a drive is marked "want_replacement" the kernel will just read from the old drive and write to the new one - the speed will be as fast as the drives can handle sequential read/writes and the array never has to be in a degraded state. But unlike just a straight image from the old to the new drive this method is part of the RAID system, so if for example you get a read error from the source drive during the copy it will then use the redundancy in the array to rebuild that data for the new drive.

Back when I was doing it, the procedure was to use mdadm to add a new drive as a spare to the array, and then "echo 'want_replacement' > /sys/block/mdx/md/dev-sdy/state" (replacing x and y with appropriate values). That was quite a while back though, so its entirely possible that mdadm has a more friendly interface to do it now.

 

[Martin Jørgensen]

Finaly i got it to work.
Had to remove mdadm.conf then after reboot my md0 where back.
Now with encrypted partition raid 1

Have a nice weekend

 

[EffrafaxOfWug]

echo 'want_replacement' > /sys/block/mdx/md/dev-sdy/state

Nice hack - from looking at the changelogs, support for this was added into mdadm v3.3 via the --replace --with syntax. I'll give this a whirl when I upgrade the array that does have free slots...

mdadm /dev/md0 --replace /dev/sdx1 --with /dev/sdy1

 

[TuxDude]

Nice hack - from looking at the changelogs, support for this was added into mdadm v3.3 via the --replace --with syntax. I'll give this a whirl when I upgrade the array that does have free slots...

mdadm /dev/md0 --replace /dev/sdx1 --with /dev/sdy1

One thing to keep in mind if doing this (assuming it still works the same way using mdadm interface), is that the drive being replaced is marked as failed at the end of the process. Of course if you are just migrating data around the drive isn't actually failed and is safe to re-use it for something else. It also leaves the md superblock in place on the drive, so if you reboot without doing anything else I've seen issues where it wants to try keeping that drive in the array due to the matching UUID - I recommend immediately erasing the md superblock from the drive after performing that procedure.