Hello,
I have a big fat server with 22 * 15.36TB Micron 9300 Pro SSDs connected directly to the motherboard via PCIe lines. I have setup a RAID6 configuration without any write caching, getting an array of about 307,2 TB. On top of the array I have a big fat MySQL instance and what I have observed is that out of nowhere, without any server restart or error, the array started to do some checks where basically it froze all the writes. And at some point it ended up having absolutely no IO requests issued, just mdadm using one core for 100%. The state of array was in active-checking, stuck at 99.9%, with all the blocks actually being checked. The array was build with a chunk size of 256KB, mdadm is on version v4.1 - 2018-10-01 (Ubuntu Server 20.04 LTS) and the array was later formatted as ext4. Also the whole array init took 4 days.
Is there any way to debug mdadm or are there some known limits regarding configurations like mine? Or any special issue with rewriting same chunk over and over again (MySQL write pattern ends up issuing bursts of write and flush commands for same space when doing a lot of transactions)?
I have a big fat server with 22 * 15.36TB Micron 9300 Pro SSDs connected directly to the motherboard via PCIe lines. I have setup a RAID6 configuration without any write caching, getting an array of about 307,2 TB. On top of the array I have a big fat MySQL instance and what I have observed is that out of nowhere, without any server restart or error, the array started to do some checks where basically it froze all the writes. And at some point it ended up having absolutely no IO requests issued, just mdadm using one core for 100%. The state of array was in active-checking, stuck at 99.9%, with all the blocks actually being checked. The array was build with a chunk size of 256KB, mdadm is on version v4.1 - 2018-10-01 (Ubuntu Server 20.04 LTS) and the array was later formatted as ext4. Also the whole array init took 4 days.
Is there any way to debug mdadm or are there some known limits regarding configurations like mine? Or any special issue with rewriting same chunk over and over again (MySQL write pattern ends up issuing bursts of write and flush commands for same space when doing a lot of transactions)?