CX516A-GCAT -> CX556A-ECAT?

[NablaSquaredG]

Hi,

I scored a brand new, factory sealed CX516A-GCAT (dual port 50GBe) for very little money.

I would like to crossflash it to CX556A-ECAT (2x 100G Ethernet / Infiniband)

Has anyone ever tried this before? Any pitfalls I need to be aware of?

 

[nasbdh9]

EN to VPI?
can not...

 

[NablaSquaredG]

EN to VPI?
can not...

Why not?

 

[nasbdh9]

Why not?

EN and VPI are different chips

 

[NablaSquaredG]

EN and VPI are different chips

BTW, this is not true.

I was able to flash GCAT (PCIe 3.0 Ethernet) to EDAT (PCIe 4.0 VPI)

So yeah, don't listen to that. It works. At least for my. YMMV

 

[Rand__]

Have you evaluated that the bandwith of the card is actually PCIe 4 level?

 

[NablaSquaredG]

It negotiates to 4.0, which is definitely a good sign

4e:00.0 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
        Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 275
        IOMMU group: 67
        Region 0: Memory at 9534000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at cf300000 [disabled] [size=1M]
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 16GT/s (ok), Width x16 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

 

[Rand__]

But doesnt necessarily mean its able to really use it...

 

[jpmomo]

That seems too good to be true but now you peaked my interest. I have a need for testing dual port 100GE at full duplex on a server that has an x16 pci gen4 slot. I normally use the 516a-cdat which is gen4. the much cheaper 516a-ccat is similar but only gen3. The link status that you are showing above is at least what should be reported. But as Rand mentioned, you would need to test this to see if it actually performs at gen4 rates.

PM me the details on exactly how you did this and I can test with some custom testing sw that would see if it is actually performing at gen4 rates.

 

[NablaSquaredG]

PM me the details on exactly how you did this and I can test with some custom testing sw that would see if it is actually performing at gen4 rates

Just the regular Mellanox crossflash process. Use the EDAT firmware and flash it to the card.

But given that I also have a CX556A-EDAM engineering sample (so "officially" PCIe 4.0) with date code 2017-03-29

Given that ConnectX-5 were launched in 2016, there may be revisions which are not PCIe4.0 capable. But it could also be that all revs are PCIe4.0 compatible.

 

[jpmomo]

Just the regular Mellanox crossflash process. Use the EDAT firmware and flash it to the card.

But given that I also have a CX556A-EDAM engineering sample (so "officially" PCIe 4.0) with date code 2017-03-29

Given that ConnectX-5 were launched in 2016, there may be revisions which are not PCIe4.0 capable. But it could also be that all revs are PCIe4.0 compatible.

I think I owe you a beer or 2! I was able to take mcx556a-ecat and burn the -edat fw. the ecat is pcie 3.0 and was only running at those rates (I have some proprietary sw that measures performance). the edat is pcie 4.0 and usually about twice the price these days! Both cards looked similar just from a cursory look. They were both manufactured in 2017. So I am not sure how far we can take this but it may have me reconsider some of the 516a-ccat nics instead of the more expensive 516a-cdat.

one caveat is that you may need to add the allow_psid_change like shown below to the flint command:

 

[jpmomo]

have you tried to cross flash any of the hpe branded mellanox cards to the generic mellanox images?

 

[mattaw]

I have just crossflashed two Mellanox badged Connectx-5 MCX556A-ECAT dual 100 IB/Gbe card -> Connectx-5 Ex MCX556A-EDAT on Windows 10 22H2 & Linux Debian 12, note the flint -allow_psid_change option does not show up in the help, but works fine.

1. Download and flash with normal tools, e.g., mst flint mlxconfig:

PS C:\Windows\system32> mst status
MST devices:
------------

  mt4119_pciconf0
PS C:\Windows\system32> flint -d mt4119_pciconf0 -allow_psid_change -i .\fw-ConnectX5-rel-16_35_3006-MCX556A-EDA_Ax_Bx-UEFI-14.29.15-FlexBoot-3.6.902.bin burn

2. Reboot w. hard poweroff (poweroff probably not needed)

3. Reset all the onboard settings to default, noting the new device ID! (reset probably not needed)

mlxconfig.exe -d mt4121_pciconf0 reset

4. Afterwards I set the ports back to Ethernet vs. Infiniband w. new device ID:

PS C:\Windows\system32> mlxconfig.exe -d mt4121_pciconf0 set LINK_TYPE_P1=2

PS C:\Windows\system32> mlxconfig.exe -d mt4121_pciconf0 set LINK_TYPE_P2=2

5. Done!

Note, in the screenshot below, the adapter has a 40 Gbps QSFP28 module in it at the moment, not a 100 Gbps. The card is also in the x4 chipset slot of a Ryzen X570 board, which is PCIe 4.0, and the main reason I wanted to cross flash it, as x4 PCIe 3 -> x4 PCIe 4 is 3940 GB/s -> 7880 GB/s! Soo much faster, ×2 in fact:



In case it is relevant this is the version of flint I used:

PS C:\Windows\system32> flint --version
flint(oem), mft 4.16.1-9, built on Jan 26 2021, 21:35:17. Git SHA Hash: 4a09d5e247df764062bcb1dc13046be456525e05

 

[jpmomo]

the adapter properties only shows it working at 40Gbps.
it show the bus is 16GT but only x4

 

[mattaw]

the adapter properties only shows it working at 40Gbps.
it show the bus is 16GT but only x4

Doh! Thanks for pointing that out, yes it has a 40 Gbps module in it at the moment, and is in the x4 PCIe 4.0 X570 chipset slot as the rest are occupied. The main reason I crossflashed it was to double the speed of those four lanes. Also it might not work with a longer slot, as I had to solder back on some of the PCIe decoupling capacitors for the PCIe lanes on this one. (I purchased two damaged for a song years ago.)

 

[NerdAshes]

I was able to flash GCAT (PCIe 3.0 Ethernet) to EDAT (PCIe 4.0 VPI)

I'm looking at a couple of cheap Mellanox MCX516A-GCAT ... It's been a year since this was re-flashed. How has it worked out?
Thanks!

 

[mattaw]

For me it has been about 3 months, they are both working just fine!

BTW I replaced the thermal paste under their feeble heatsink with some Laird 7000 PCM as these run HOT, (Honeywell PCM PTM7950 is the other one). It lowered their temps by probably 10 degC too. Details here: Laird Tpcm 7250 is as good as Honeywell PTM7950 as thermal paste / interface for PC (note the 2-3 degree difference listed on my blog post is the temps before PCM on idle, and after the PCM under load, so much better than at first glance). PCM is super-useful for this scenario as it performs great with weak clamping force.

 

[thulle]

@mattaw Just to confirm this, would you be able to run iperf or similar? Since PCIe3 x4 would cap at 4 * ~1GB/s * 8bits/Byte ~= 32Gbit/s, and the transceiver is able to do 40Gbit/s, any number between 32 & 40 would definitively settle it.

 

[mattaw]

I'll poke it tomorrow and see how far I get with the existing optics. Unfortunately when I pulled fiber through the house I used mmf, had I ever realized that 100gbe was affordable I would have used smf, sigh. I should just try the 100gbe optics anyway, maybe they will work over the short run?

 

[NerdAshes]

I ended up going with the CX516A-CDATs because the CX516A-GCATs I found on Ebay actually looked different (The end of the card had different circuitry) and I didn't want to risk it. I'm not sure if it is because they are Chinese made in 2022, and the other cards I've seen are older, and made in either India or Israel. I also wanted the cards faster (free 2 day air) and a warranty from a trusted store. I'm am however still temped to get the Ebay cards and see if I can save $100 a card with a simple flash. That and I need the low profile brackets and I'm not sure my CDATs will show up with them (should if new, like their listing says).

Here are the best values I've found - if anyone wants them..
Dell Ex CDAT - New -$379 (What I ended up getting)
Cisco CX516A - Used - $251 (Offer they sent to me)
GCAT - New - $279 (Both brackets, tempting price)

Matt thanks for the link to your blog and the reply. I'll definitely be doing your paste mod!