SAS3 Raid ZFS issues

[michaels2408]

Please bear with me as there is quite a bit to explain. -I have a Supermicro superserver with X11DPU-Z+ board, 512gb DDR4 ECC 2133 ram, 2 x Xeon Gold 6248 CPU's, 12 SAS3 7.2K 12gbs Hard drives on the server back plane tied to a Supermicro AOC3008 SAS3 card in IT mode. These are on CPU 1's PCIE lanes with a AOC 4 port 10g network adapter. On CPU 2's PCIE lanes I have two more AOC3008 cards in IT mode and an older Nvidia 1060 GTX card. The last two mentioned AOC3008 SAS cards, one is used to pass thru 4 10k SAS3 Hard drives for a ZFS RAID array used in a Proxmox Backup server which runs in a VM on the server. The other SAS3008 card has 4 10k SAS3 drives which are the Proxmox ZFS Data array that stores the VM's. Both of these are in external 4 bay non-supermicro drive cages with SAS3/SATA3 back plane using the SATA connectors and a 5v molex 4 pin for power.

VM Data VM Backup Pass thru
------------- --------------
External Cage 1 External Cage 2
SAS3 - 1 SAS3 - 1
SAS3 - 2 SAS3 - 2
SAS3 - 3 SAS3 - 3
SAS3 - 4 SAS3 - 4
------------ ------------
external external
mini sas mini sas
HD SFF-8644 HD SFF-8644
to 4 Port SATA to 4 port SATA
7 Pin DATA 7 Pin DATA
------------- ----------------
AOM-SAS3-8I8E-LP AOM-SAS3-8I8E-LP
2-Port External to 2-Port External to
Internal Adapter Internal Adapter
------------- -----------------
SFF-8643 male to SFF-8643 male to
SFF-8643 male SFF-8643 male
------------- ----------------
AOC3008 AOC3008

Here is the issue. When I use the HD SFF-8644 to 4 port SATA 7 pin DATA on the VM Data array, there is a constant issue of decay, read errors the result in zfs pool degredation. It is not associated with any one hard drive, and while trouble shooting by swapping componants the errors move from one disk to the other causing different disks to come offline and degrade the pool. The VM Backup pass thru zfs pool on external cage 2 NEVER experiences any issues, no matter how much load its under. I have swapped PCI slots, power connectors, external cages, SFF-8644 cables, SFF-8643 cables, and the problem always comes back to the VM data array. The SAS3 AOC3008 cards have the internal miniSAS connectors so to use an external cable I have a Supermicro AOM-SAS3-8I8E-LP 2-Port External to Internal Adapter which I even tried swapping the ports. Problem always stays with the Hard drives in the VM Data external Cage 1. So I thought: its the disks. Well I tried 4 different SAS3 10K drives and had the same results. Then I tried using a Supermicro SFF-8643 to 4 SATA 7 pin Data breakout internal cable routhing it through an open expansion bay to the cage and all the problems went away. As long as I use the internal cable I have no issues. The second I route an external cable the problem returns. Even though the external cage 2 is the exact same type of setup and doesnt have any issues. Is this a Signal Issue? Any ideas, or thoughts would be appreciated. I really want to utilize the external cable on this build for both external cages.

 

[mattventura]

Signal integrity issues due to length is a likely scenario. When you use the internal cable directly passed to the outside, you're skipping the length of the internal cable you would use to connect the 3008 to the AOM.

Also, how are you powering the external cages?

Do you have pictures of the setup?

 

[Stephan]

lsiutil 1.72 e.g. from Arch AUR should support your cards. lsiutil -> select controller number -> 20 diagnostics -> 12 display phy counters, see what the card thinks about the connections. Ask AI if there are other options (there are dozens). Like in 20, number 14, SATA SMART Read Test. What cage is this? Sounds like it is dying.

 

[stronthoop]

This sounds like a faulty backplane of External cage 1. Try this:
- Run the array without the external cages. Get these cables to rule out the disks: Mini SAS HD SFF-8644 to 4 SATA Breakout Cable 1m
- It is well known in the industry that the Supermicro cards dont reflash easily or properly to IT mode, this is heavily protected by Supermicro. This could also be a main reason. Get DELL 355i, or cheap lsi cards that are 100% IT mode. Best is not to get a flashed one but actually get a type that is only HBA
- open up external bay2 and check the partnumber of the backplane. Search part no of backplane on ebay. Buy this new backplane for external bay 1 second hand and swap it.

its most likely caused by lack of correct signaling, either by cables, backplane or your aoc cards. My biggest hunches are backplane1, but also the aoc cards its a 50/50.

GL and let us know.
kr,
Stront