AM5 "Servers", Mellanox Connectx-7 and PCIe Gen5. The experiment continues!

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
J

jpmomo

Well-Known Member
Aug 12, 2018
596
258
63
While waiting on a real pci gen5 server/cpu, I continue to experiment with any currently available options.

cpu: amd ryzen 7950x

mb: asus prime x6703-pro wifi

ram: corsair dominator platinum rgb (always needed for a server :) ) ddr5 5200MHz 4 x 32

nic: mellanox/nvidia connectx-7 2x 200G pci gen5 x16

OS: esxi 8, ubuntu 22.04

NOTES:
1. the mb supports sriov in the bios which is important as the main use case is to try and utilize the cx-7 nic using multiple VMs as traffic generators.
2. this mb also supports an extra pci 4.0 x4 slot for another nic (used for vmware connectivity). esxi 8.x apparently doesn't support the onboard realtec nic.
3. I can use the integrated graphics from the cpu as I will need the only pci gen5 slot for the nic card.

Everything works without many issues except for the main purpose, the system won't even recognize the cx-7 nic!

For trouble-shooting:
swap cx-7 nic with intel e810 pci gen4 dual port 100G nic and everything is ok.
looked into bios and "monkeyed" with some seemingly related knobs, no luv with the cx-7
tried with both ubuntu and vmware no luv
I have another system that is pci gen4 and more of a server/workstation mb (sm x12-spa-tf). This system recognized the cx-7 in ubuntu and all looks good. the only issue with that system is the pci gen4 bottleneck.

I have a gigabyte mc13-le0 on order which might help. I am hoping that someone might have some suggestions on how to at least get this nic card to be recognized with this mb.

Thanks for anyone's help!
 
  • Like
Reactions: T_Minus and MichalPL
R

RolloZ170

Well-Known Member
Apr 24, 2016
10,064
3,228
113
germany
while the 5.0 NIC works in 4.0 slot of x12-spa
have you tested the 5.0 NIC in 4.0 slot of asus prime x670E-pro wifi ????????????
 
J

jpmomo

Well-Known Member
Aug 12, 2018
596
258
63
RolloZ170 said:
while the 5.0 NIC works in 4.0 slot of x12-spa
have you tested the 5.0 NIC in 4.0 slot of asus prime x670E-pro wifi ????????????
Click to expand...
Yes. I forgot to mention that the 5.0 NIC works in the 4.0 (x4) slot of the asus prime.

thanks
 
J

jpmomo

Well-Known Member
Aug 12, 2018
596
258
63
upgraded to beta bios (with new agesa!) and at least the 5.0 nic is recognized:

01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
Subsystem: Mellanox Technologies MT2910 Family [ConnectX-7]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 158
IOMMU group: 13
Region 0: Memory at fcf2000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn+
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported


Getting closer!
 
J

jpmomo

Well-Known Member
Aug 12, 2018
596
258
63
I am looking for the LnkSta: Speed 32GT/s but only getting 2.5GT/s

Notes from new bios:


Version 0614 Beta Version
2022/09/26 9.13 MBytes
PRIME X670E-PRO WIFI BIOS 0614
"1. AGESA version ComboAM5PI 1.0.0.2
2. Improve system stability and GPU compatibility for GeForce RTX 40 series

(still a lot of "00" in the version # :) )
 
R

RolloZ170

Well-Known Member
Apr 24, 2016
10,064
3,228
113
germany
jpmomo said:
I am looking for the LnkSta: Speed 32GT/s but only getting 2.5GT/s
Click to expand...
sometime other user recognized same with GPU and the reason was:
"The hardware changes the PCIE speed dynamically at runtime based on GPU load to save power."
 
J

jpmomo

Well-Known Member
Aug 12, 2018
596
258
63
makes sense as I have heard something similar before. The beta bios was giving me some issues when trying to set pci link speed to gen 5. I had to reload bios a couple of times but ultimately got the pci link to train at 32GT/s. Now I have to see if I can get some consistency out of this setup :).

01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
Subsystem: Mellanox Technologies MT2910 Family [ConnectX-7]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 124
IOMMU group: 12
Region 0: Memory at 862000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 32GT/s (ok), Width x16 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn+
LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
 
  • Like
  • Love
Reactions: MichalPL, mach3.2, llowrey and 1 other person
M

MichalPL

Active Member
Feb 10, 2019
189
30
28
Are you able to transfer somehow more than 200GBit/s from the system ? is 7950x fast enough ?

I will try to replace some time my super old old 2x Xeon E5 2687W v2 server ;) who is able to deliver ~6.5GB/s peak using ZFS.
and good question is if 7950X/7900X or 13900kf is fast enough.
 
J

jpmomo

Well-Known Member
Aug 12, 2018
596
258
63
MichalPL said:
Are you able to transfer somehow more than 200GBit/s from the system ? is 7950x fast enough ?

I will try to replace some time my super old old 2x Xeon E5 2687W v2 server ;) who is able to deliver ~6.5GB/s peak using ZFS.
and good question is if 7950X/7900X or 13900kf is fast enough.
Click to expand...
Yes, but I am expecting more and am not sure where the bottleneck is yet.

I am running some proprietary sw and have the 2 ports connected with a dac (qsfp56) cable.

The theoretical limit would be appox 400Gbps when running with bidirectional flows.

The pci slot should not be the bottleneck as it is showing pci gen5 and x16 lanes.

On my other systems that are pci gen4, I need to use 2 x16 slots (each with their own 200GE nic).

at these rates, at least with the testing app that I am using, you have to consider more than just how "fast" the cpus are. The amds utilize a chiplet architecture. even though they have larger L3 cache than the current intels (ice lake xeons), that cache is divided into smaller pools.

The intels (8370c) are achieving close to the 400Gbps that I am expecting on a gen4 system with 2 x16 slots. single cpu, 2 x 200GE nics pcie gen4

The amd 7950x is only getting appox 220Gbps when running on a gen5 system with 1 x16 slot using a single cx-7 nic that has 2x 200GE ports gen5.

I have seen similar results when comparing the amd 7763 on a gen4 system until the cpu config was optimized and only on a linux setup (not vmware).
 
M

mirrormax

Active Member
Apr 10, 2020
231
89
28
since you have access to the board can you confirm pcie slot1 supports 4x4x4x4x bifurication? its not in the tech specs for the new boards but its in the manuals and asus faq.

asus might call it PCIE RAID mode. while asrock manuals documents it as just 4x4x4x4x
 
T

TheMachine

New Member
Sep 6, 2025
6
0
1
Hey, sorry to necro-post this thread !
How the AM5 board treated you with the connectx-7 card ?
I am currently trying to run an MCX75310AAS-NEAT on a ASUS X870E Pro Art motherboard on the second pcie slot (normally 5.0 x8) but the pcie speed always lock it to 1.1 speed :

LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)

What workaround did you tried for the 32GT/s speed ? It is working now ?
Thanks !
 
T

TheMachine

New Member
Sep 6, 2025
6
0
1
Is that on the second pcie port ?
I have debugged some more : the mellanox card lock to gen 5 easily on slot 1, sometimes on slot 2 but not always. FOM seems to be similar to ours, but the pre/post emphasis data is a bit odd.
The port is working fine, I tried with a pcie 5 carrier card with an gen 5 SSD that appears correctly.
It seems that the stability of the slot is limit for a mellanox cx7, an since the bios doesn't show redriver / emphasis options, I think it is kinda a dead end. At least, gen 4 works perfectly

Edit : btw, what is your CPU ? I have a x3d chip, that might also play given there was quite a lot of error with those in early days
 
Last edited:
You must log in or register to reply here.
Top Bottom