AM5 "Servers", Mellanox Connectx-7 and PCIe Gen5. The experiment continues!

[nasbdh9]

Regardless of the CPU model, as long as the path from the CPU to the second slot does not include PCIe redriver, no AM5 motherboard on the market can stably run PCIe 5.0 or even 4.0.

The X870E Taichi series is currently the only motherboard where both PCIe slots are connected via PCIe redriver (PS7101 with mux function).

 

[TheMachine]

The Asus Pro Art (and pretty much all Asus x870e board) have 4 Phison PS7101 PCIe redrivers on the back of the board. Following trace, it looks like it is for the second pcie slot (and anyway, 4 of those carry x8 lanes). So, it's pretty much not a redriver issue.
Given that other pcie 5 card works without issue on this slot, I suspect it is more of an fundamental hardware incompatiblity with the mellanox cx7 which might require a much cleaner signal. Or just the pcie port link training of the firmware doing stupid thing. Who knows

 

[nasbdh9]

I suspect it is more of an fundamental hardware incompatiblity with the mellanox cx7 which might require a much cleaner signal

Excessive PCIe redriver signal gain can also cause the CX7 to revert to the lowest PCIe speed or fail to recognize devices. I discovered this while fiddling with the PCIe redriver parameters in the BIOS of my X870E Taichi Lite.

 

[nasbdh9]

CX7 will rebalance overshoot PCIe signals, Last FOM value greater than 100 is considered PASS.

 

[TheMachine]

Well, pcie redriver settings are not exposed in the Pro Art x870e bios, so we're at loss here.
When locking gen 5 however, Last FOM are almost all >100. It is an odd issue.

PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node            : 0, 0, 0
Link Speed Active (Enabled)        : 32G-Gen 5 (32G-Gen 5)
Link Width Active (Enabled)        : 8X (16X)

Serdes Tuning Transmitter Info (PCIe)
-------------------------------------
Serdes TX parameters               : fir_pre3 ,fir_pre2 ,fir_pre1 ,fir_main ,fir_post1 ,drv_amp
Lane 0                             : 0        ,0        ,11       ,52       ,0         ,63  
Lane 1                             : 0        ,0        ,11       ,52       ,0         ,63  
Lane 2                             : 0        ,0        ,8        ,55       ,0         ,63  
Lane 3                             : 0        ,0        ,8        ,55       ,0         ,63  
Lane 4                             : 0        ,0        ,8        ,55       ,0         ,63  
Lane 5                             : 0        ,0        ,11       ,52       ,0         ,63  
Lane 6                             : 0        ,0        ,11       ,52       ,0         ,63  
Lane 7                             : 0        ,0        ,11       ,52       ,0         ,63  

EYE Opening Info (PCIe)
-----------------------
FOM Mode                           : SLRG_FOM_MODE_EYEC
Lane                               :      0,     1,     2,     3,     4,     5,     6,     7
Initial FOM                        :    113,   120,   118,   115,   116,   106,   125,   118
Last FOM                           :    103,   103,   120,   105,    98,   108,   109,   116

Management PCIe Performance Counters Info
-----------------------------------------
RX Errors                          : 0
TX Errors                          : 1
CRC Error dllp                     : 0
CRC Error tlp                      : 0
Effective ber                      : 15E-255

EDIT :
Also tried newest firmware. Well, now, it just disappears from the pcie port if it have trouble to lock to gen 5. Gen 4 is still perfectly usable though.

EDIT 2 :
I might have a failed motherboard / or the motherboard is not adapted to this.
After carefully inspecting trace the following layout is :
- 8 lanes straight to slot 1
- 8 lanes going to redriver / demux for switching between slot 1 and slot 2
- Of the 8 lanes going to slot 2, there is a mux/demux for 4 lanes going to secondary M.2 slot.
The mellanox card have no trouble locking to gen 5 using only the 4 lanes going through the redriver so it pretty much clean them for the cause, but fail with the 4 lanes going into the mux/demux which is a passive mux/demux. I have not yet tried the secondary M.2. If it is not working with a pcie gen 5 SSD, guess it is time for RMA / motherboard change. If it works however, the signal degradation of the x4 lanes might be just too much for the cx7.
I'll try more troubleshooting.

 

[TheMachine]

Finally found the cause which is on insight a quite stupid issue from my side.
The x870e pro Art have QSlim release connector on both pcie 5 x16 slot

The mellanox card doesn't have the expected hook to fully engage the QSlim mechanism and as such the mechanism push slighty the card out of contact for the last few pcie lane... Using a riser extension (https://www.amazon.fr/Giyofom-Connecteur-Adaptateur-Convertisseur-Serveur/dp/B0F9YS4KZW) allow the card to be fully socketed and working perfectly fine. I suppose because the superior slot have all x16 physical lane the contact is sufficient to not move the card slightly. I did not expect this mechanism to be so stupid but oh well. I don't know if it can be socketed without such extension either, looking at the card and the slot mechanism.
Anyway, I'll probably ask ASUS support how to correctly socket those card with their system, but the current solution is good enough and also provide additional power for the slot.

 

[RolloZ170]

The x870e pro Art have QSlim release connector on both pcie 5 x16 slot

makes me think about ASUS LAN Guard issues,
all special ASUS stuff makes trouble.

 

[TheMachine]

makes me think about ASUS LAN Guard issues,
all special ASUS stuff makes trouble.

Yeah, they make good hardware, but sometimes there is some... Oversight. Might also just be me and my inability to slot correctly, but seing the mechanism just make me wonder if it is even possible... I might ressort to McGyvering a solution