mdadm RAID6 problem

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
D

DoubleA

New Member
Sep 7, 2023
4
0
1
Hi, I was trying to upgrade my 8 disk RAID6 array to a 11 disk RAID6 array but now they are totally gone. MDADM shows only the 3 newly added disk as raid0 instead of raid6 array with 11 disk :
Code: 
mdadm --detail /dev/md0                
/dev/md0:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 3
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 3

              Name : IBM-X3630:0
              UUID : 82129e2c:1e087cda:fb81b27b:38696198
            Events : 83946

    Number   Major   Minor   RaidDevice

       -       8      144        -        /dev/sdj
       -       8      160        -        /dev/sdk
       -       8      128        -        /dev/sdi
fdisk shows the 3 newly added disk as :
The primary GPT table is corrupt, but the backup appears OK, so that will be used.
Click to expand...
All my disk listed by fdisk as below.
Code: 
fdisk -l
Disk /dev/nvme0n1: 238.49 GiB, 256060514304 bytes, 500118192 sectors
Disk model: ADATA SX8200PNP                     
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1D264D68-9B97-45B0-A949-CE99C45009CF

Device             Start       End   Sectors   Size Type
/dev/nvme0n1p1      2048    999423    997376   487M EFI System
/dev/nvme0n1p2    999424 196311039 195311616  93.1G Linux filesystem
/dev/nvme0n1p3 196311040 500117503 303806464 144.9G Linux filesystem


Disk /dev/sdl: 3.77 GiB, 4026531840 bytes, 7864320 sectors
Disk model: Flash-Disk  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x071118a3

Device     Boot  Start     End Sectors  Size Id Type
/dev/sdl1  *        63  471102  471040  230M  c W95 FAT32 (LBA)
/dev/sdl2       471103 7864319 7393217  3.5G  6 FAT16


Disk /dev/sdb: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 6D03BC37-C1B2-4066-A629-58626E1616B2

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdh: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 10B93A08-19DD-47D1-882E-92B99F5ACD4B

Device     Start        End    Sectors  Size Type
/dev/sdh1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdg: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 78EBC43B-DD79-431D-A011-D71493AB3ACA

Device     Start        End    Sectors  Size Type
/dev/sdg1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdd: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 749FF30B-A483-4BEB-AD95-C024F6F17EBC

Device     Start        End    Sectors  Size Type
/dev/sdd1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdc: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 86152B74-F75A-4772-8DDA-63BFFF5A3207

Device     Start        End    Sectors  Size Type
/dev/sdc1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sde: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: B0EA6919-E351-4F3D-9BB8-53C86C07C64E

Device     Start        End    Sectors  Size Type
/dev/sde1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sda: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E50E451A-AF50-44F9-B8F3-46D8EF41B7FD

Device     Start        End    Sectors  Size Type
/dev/sda1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdf: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


[COLOR=rgb(209, 72, 65)]The primary GPT table is corrupt, but the backup appears OK, so that will be used.[/COLOR]
Disk /dev/sdk: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 5A2E46FD-636C-4A42-858E-F7604D35504F

Device     Start        End    Sectors  Size Type
/dev/sdk1   2048 5860533134 5860531087  2.7T Linux filesystem


[COLOR=rgb(209, 72, 65)]The primary GPT table is corrupt, but the backup appears OK, so that will be used.[/COLOR]
Disk /dev/sdi: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F08A6608-7861-466D-9DDD-8A0FDDD040D2

Device     Start        End    Sectors  Size Type
/dev/sdi1   2048 5860533134 5860531087  2.7T Linux filesystem


[COLOR=rgb(209, 72, 65)]The primary GPT table is corrupt, but the backup appears OK, so that will be used.[/COLOR]
Disk /dev/sdj: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA HDWD130
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 35BFA05D-BCD5-48CB-9F07-F061AEDECBDA

Device     Start        End    Sectors  Size Type
/dev/sdj1   2048 5860533134 5860531087  2.7T Linux filesystem
Please help!
 
M

MBastian

Active Member
Jul 17, 2016
205
59
28
Düsseldorf, Germany
  • Did you already try to expand your existing RAID6?
  • Where the added disks already used in another Raid(0) array?
  • What is in your /etc/mdadm.conf?
  • Did you check if your old array is now assembled under another md device number?
  • Does `dmesg` show some clues?
  • What does a ` mdadm --examine --scan` show?
 
Last edited:
D

DoubleA

New Member
Sep 7, 2023
4
0
1
MBastian said:
  • Did you already try to expand your existing RAID6?
  • Where the added disks already used in another Raid(0) array?
  • What is in your /etc/mdadm.conf?
  • Did you check if your old array is now assembled under another md device number?
  • Does `dmesg` show some clues?
  • What does a ` mdadm --examine --scan` show?
Click to expand...
- Yes, I have 'add' 3 disk to existing 8 disk and 'grow' the array to 11 disk with the below command :
Code: 
mdadm --add /dev/md0 /dev/sdi1
mdadm --add /dev/md0 /dev/sdj1
mdadm --add /dev/md0 /dev/sdk1
mdadm --grow --backup-file=/root/grow_md0_backup_file --raid-devices=11 /dev/md0
- No, they are 3 brand new disk. Just partition using 'gdisk' before 'add' to the array.
- Below the my /etc/mdadm.conf :
Code: 
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0  metadata=1.2 UUID=82129e2c:1e087cda:fb81b27b:38696198 name=IBM-X3630:0

# This configuration was auto-generated on Mon, 01 Feb 2021 05:03:01 +0800 by mkconf
- No, there is only /dev/md0. no other md
- the `dmesg` is very long. I have no clue as to what should I be looking for?
- Below is my ` mdadm --examine --scan` result
Code: 
sudo  mdadm --examine --scan
ARRAY /dev/md/0  metadata=1.2 UUID=82129e2c:1e087cda:fb81b27b:38696198 name=IBM-X3630:0
   spares=3
ARRAY /dev/md/0  metadata=1.2 UUID=82129e2c:1e087cda:fb81b27b:38696198 name=IBM-X3630:0
 
D

DoubleA

New Member
Sep 7, 2023
4
0
1
unwind-protect said:
`cat /proc/mdstat` ?
Click to expand...
Below is the output :
Code: 
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : inactive sdi[9](S) sdj[10](S) sdk[11](S)
      8790403464 blocks super 1.2
       
unused devices: <none>
Found online about the 'examine' option so i ran it on the 1st drive and got some encouraging result on the Raid level and devices :
Code: 
sudo mdadm --examine /dev/sda
/dev/sda:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)

sudo mdadm --examine /dev/sda1  
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 82129e2c:1e087cda:fb81b27b:38696198
           Name : IBM-X3630:0
  Creation Time : Thu Sep 26 02:59:46 2019
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
     Array Size : 26371196928 (25149.53 GiB 27004.11 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=0 sectors
          State : clean
    Device UUID : 74c171c4:eec784f8:e67a3630:480d4859

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Sep  7 21:43:22 2023
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : 2fcac8e5 - correct
         Events : 90401

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
But the newly added 3 drives (sdi, sdj, sdk) show 'Raid Devices : 8' and missing partition just like fdisk reported (see #1 post) :
Code: 
sudo mdadm --examine /dev/sdi1
mdadm: cannot open /dev/sdi1: No such file or directory

sudo mdadm --examine /dev/sdi
/dev/sdi:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 82129e2c:1e087cda:fb81b27b:38696198
           Name : IBM-X3630:0
  Creation Time : Thu Sep 26 02:59:46 2019
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 5860268976 (2794.39 GiB 3000.46 GB)
     Array Size : 17580797952 (16766.36 GiB 18002.74 GB)
  Used Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=2992 sectors
          State : clean
    Device UUID : 44414aee:19b51b4e:3c68e53d:5a700649

    Update Time : Thu Sep  7 00:02:37 2023
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : 6592afc9 - expected 6592afc8
         Events : 83945

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Below is the output of Gdisk on sda (the 1st drive on the original 8 raid6 which probably shows an intact partition table) and on sdi (1 of the newly 'add' disk which shows a damaged partition table after 'grow')
Code: 
sudo gdisk /dev/sda 
GPT fdisk (gdisk) version 1.0.5

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.


sudo gdisk /dev/sdi
GPT fdisk (gdisk) version 1.0.5

Caution! After loading partitions, the CRC doesn't check out!
Warning! Main partition table CRC mismatch! Loaded backup partition table
instead of main partition table!

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: OK
Main partition table: ERROR
Backup partition table: OK

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
 
Last edited:
N

nexox

Well-Known Member
May 3, 2023
695
283
63
It does look like you have gotten into an odd state, first of all, are you positive you used the partition and not the base device with the mdadm --add command? If you left off the 1 it could explain part of this (in future arrays you can simplify things by just not using a partition under md raid volumes unless you actually need to split the device up.)

From what I can see from the --examine output, you have two different arrays with the same UUID, no idea how that happens, but it seems the array composed of the new drives is hiding the original. Perhaps you got some variety of corruption on the new disks due to using a device that was still running with an active partition.

My only idea is to unplug the three new drives and reboot, hopefully the old array comes up as it was before.
 
D

DoubleA

New Member
Sep 7, 2023
4
0
1
nexox said:
It does look like you have gotten into an odd state, first of all, are you positive you used the partition and not the base device with the mdadm --add command? If you left off the 1 it could explain part of this (in future arrays you can simplify things by just not using a partition under md raid volumes unless you actually need to split the device up.)

From what I can see from the --examine output, you have two different arrays with the same UUID, no idea how that happens, but it seems the array composed of the new drives is hiding the original. Perhaps you got some variety of corruption on the new disks due to using a device that was still running with an active partition.

My only idea is to unplug the three new drives and reboot, hopefully the old array comes up as it was before.
Click to expand...
Thanks for your reply, the original 8 drives show up when the 3 newly added drives was unplug but they shows up as 'raid0' instead of 'raid6'. I was hoping that this should be an easy fix as I did not reach the step to expand the partition above the md to 11 drives and stop after the 'grow' process which took more than 20 hours. So can I just 'fix' the partition table of the 3 newly drives? Or reverse the 'grow' process?
Code: 
sudo mdadm --detail /dev/md0 
/dev/md0:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 8
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 8

              Name : IBM-X3630:0
              UUID : 82129e2c:1e087cda:fb81b27b:38696198
            Events : 90401

    Number   Major   Minor   RaidDevice

       -       8        1        -        /dev/sda1
       -       8       80        -        /dev/sdf1
       -       8      113        -        /dev/sdh1
       -       8       97        -        /dev/sdg1
       -       8       65        -        /dev/sde1
       -       8       49        -        /dev/sdd1
       -       8       33        -        /dev/sdc1
       -       8       17        -        /dev/sdb1
 
N

nexox

Well-Known Member
May 3, 2023
695
283
63
I don't really know what you should do from here. I guess you could try recreating the partition tables on the three new drives and hoping that it will reassemble, but you may want to prepare yourself for the possibility that the entire array is hosed and that data recovery may be a difficult exercise.
 
S

sfx2000

New Member
Sep 23, 2023
1
0
1
If this helps...

I usually put LVM between mdadm and the file system...

That way one can add an additional md group of disks, add that the the LVM volume group, and grow the filesystem from there, assuming ext4 or xfs
 
You must log in or register to reply here.
Top Bottom