Mellanox flashing ConnectX-5 to ConnectX-5 Ex ? (PCIe 3.0 to PCIe 4.0 ?)

[MichalPL]

I am wondering if is it possible for "flash" Mellanox MCX516A-CCAT to MCX516A-CDAT ?

Are ConnectX-5 and ConnectX-5 Ex based on the same chip and electronic board and PCIe 4.0 support is only the software thing?

or better to not try it - and there is a big (or even 100%) risk having small green brick then

Best,
Michal

 

[necr]

I am wondering if is it possible for "flash" Mellanox MCX516A-CCAT to MCX516A-CDAT ?

They're really close, both dual-port, should be possible. Do you have Mellanox or OEM cards? OEMs, like HP come with secure-fw, meaning crossflash is nearly impossible without an i2c flash access.

Are ConnectX-5 and ConnectX-5 Ex based on the same chip and electronic board and PCIe 4.0 support is only the software thing?

ConnectX-5 card is VPI (both IB and Eth), and it's really far from Ex cards (Eth only). Please dump the firmware config here ((mst)flint -d <your card> q; (mst)flint -d <your card> dc;
Does your system support PCIe 4.0 in that particular slot (lspci -s <addr> -vvv)

or better to not try it - and there is a big (or even 100%) risk having small green brick then

My CX6 cross-flash has been successful, there are also successful reports of CX5 cross-flashes here.

 

[MichalPL]

Today arrived at the office to test a few things

Do you have Mellanox or OEM cards? OEMs, like HP come with secure-fw, meaning crossflash is nearly impossible without an i2c flash access.

This one is probably pure Mellanox (I bought just 1, my first ConnectX-5 to test it, now it's fully working under Windows10 but it's PCIe 3.0 x16 like cheaper ConnectX4 100GbE) also it' is "too fast" to use it in something else than server (nice one, to fully utilize it if PCIE 4.0 will be working).

But today I flashed also 455A-FCAT to ECAT (40GbE to 100GbE) from DELL and it's also easy only difference is at Windows 10 drivers fail to start it, but you can still flash it easily (Windows command):
flint -d mt4115_pciconf0 -i c:\M\fw-ConnectX4-rel-12_28_2006-MCX455A-ECA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin -allow_psid_change burn
and to switch Ethernet permanently:
mlxconfig -d mt4115_pciconf0 set LINK_TYPE_P1=ETH

but I am still worry about flash PCIe 3.0 to 4.0, my logic says something inside should be different to support double the speed. hmm...
CCAT to CCHT no problem but CDAT hmm.... especially when they are changing name from EN to EX

https://www.mellanox.com/files/doc-2020/pb-connectx-5-en-card.pdf

ConnectX-5 card is VPI (both IB and Eth), and it's really far from Ex cards (Eth only).

Fortunately I need Eth only but still Eth and IB is nothing to PCIe 3.0 and PCIe 4.0 I think - but not sure :/

Please dump the firmware config here ((mst)flint -d <your card> q; (mst)flint -d <your card> dc;

good idea

Does your system support PCIe 4.0 in that particular slot (lspci -s <addr> -vvv)

No - not yet but I will organize it when working

My 2 today test bench are (old workstations):
TR 1950x - I can connect 4 PCIe 3.0 x16 cards
Xeon 1650 v2 - I can connect 2 PCIe 3.0 x16 cards and one x4.


Problem with PCIe 4.0 and 2 ports is, how to saturate it (200GbE) from SSD's - So waiting for TR 6000 or Xeon 12gen

Fastest testbench I can organize for while (until it goes as a workstation) is 12900kf with DDR5 memory, If using PCIe 5.0 slot for Mellanox, still 12 PCIe 4.0 lanes left so about 18 GB/s from NVME is possible, but 2x100GbE is still faster - about 22GB/s. 5950x is too slow in single thread (I believe) and limited bandwidth thru chipset (~8GB/s).

My CX6 cross-flash has been successful, there are also successful reports of CX5 cross-flashes here.

Can you recognize on the pictures if they looks similar or same to EX PCIe 4.0 cards ?






*sorry for the pictures quality, I forgot phone from the car and use old samsung gear wristwatch to take a photos

 

[MichalPL]

hmm..

They're really close, both dual-port, should be possible.

found photos on ebay and one CDAT (PCIe 4.0) have same PCB number like my "CCAT" (PCIe 3.0)

Pardon Our Interruption...

and here CCAT is same like my, CDAT have upgraded PCB number, but looks also same... hmmm

Mellanox CONNECTX-5 100GbE VPI Adapter Card ED 1x MCX516A-CDAT 1x MCX516A-CCAT | eBay

Find many great new & used options and get the best deals for Mellanox CONNECTX-5 100GbE VPI Adapter Card ED 1x MCX516A-CDAT 1x MCX516A-CCAT at the best online prices at eBay! Free shipping for many products!

www.ebay.com

maybe just try it if not working I will have a green brick ?

 

[necr]

maybe just try it if not working I will have a green brick ?

You could try this - there are 2 partitions on the internal flash, so it shouldn't turn into a brick in any case. There are also flash_recovery pins that you can short beforehands to ensure that you have access to the card in recovery mode.

Still, to me personally it's unnecessary risk to do this without a PCIe 4.0 system to test on.

 

[MichalPL]

Challenge accepted

 

[MichalPL]

flashing 12900K waiting for me

 

[MichalPL]

Ok 100% working! PCIe 4.0 16GT/s. 100GbE on both ports - connected temporary to i9 12gen (but next server should be PCIe 4.0 or 5.0 too).

Name of the card (Part Number) is still "CCAT" but working as PCIe 4.0 - and model name is "Ex", 2x 100GbE working too

 

[MichalPL]

Still, to me personally it's unnecessary risk to do this without a PCIe 4.0 system to test on.

Thank your for help
Yes until PCIe 4.0 no sense to flash it (now we know it's possible), the cheapest way to test PCIe 4.0 is Ryzen3 3100x, but it's good just for testing - we were considering it as a test bench but it's too slow for other things.

 

[YenForYang]

Ok 100% working! PCIe 4.0 16GT/s. 100GbE on both ports - connected temporary to i9 12gen (but next server should be PCIe 4.0 or 5.0 too).

Name of the card (Part Number) is still "CCAT" but working as PCIe 4.0 - and model name is "Ex", 2x 100GbE working too

Sorry to resurrect this old thread, but for clarity, what was the exact command you used to flash CCAT to CDAT? Just want to make sure I do things right as I'm looking to do the same for ACAT to ADAT. My assumption is that the process is the same and that it should work.

 

[Drew442_Drew]

also to resurrect a resurrected thread... Any luck cross flashing Huawei cards? these sell significantly cheaper than Dell, HP, Mellanox branded cards. Before I buy some it would be good to know if anyone has had success or if current cross flash methodology won't care what brand we're working with.

 

[blunden]

Did anyone try cross-flashing a MCX512A-ACAT or MCX512A-ACUT to MCX512A-ADAT? Would be nice with a somewhat reasonably priced 10/25 Gbps NIC with PCI-E 4.0.

One PCB image I saw had a 1 instead of a 0 in the PCB number though so perhaps they are too different.

 

[jmsq]

I can confirm MCX516A-GCAT can be flashed to MCX516A-CDAT using similar methods to previous-gen CX4 cross-flashing. This example was done with mstflint on Arch Linux:

sudo mstflint -d 0X:00.0 --allow_psid_change -i fw-ConnectX5-rel-16_35_4506-MCX516A-CDA_Ax_Bx-UEFI-14.29.15-FlexBoot-3.6.902.bin burn

There doesn't appear to be a hardware boundary between CX5 and CX5-Ex at least in this example. The main hardware boundary you'll want to be aware of with this in general is CX4 and CX4-Lx are distinct chips and you shouldn't try to go between those (and possibly CX3 vs. CX3 Pro.) I don't have any experience with CX6 and newer.

 

[necr]

I don't have any experience with CX6 and newer.

On ConnectX-6, if the firmware has a "secure-fw" flag, the sure way to reflash it is to use a MTUSB-1. On ConnectX-7 it's SOL right now, we need some ex-Mellanox with some insider knowledge to proceed. On ConnectX-8...hmm get me a CX8 and I'll tell you lol, probably the same as on CX7.

 

[blunden]

There doesn't appear to be a hardware boundary between CX5 and CX5-Ex at least in this example. The main hardware boundary you'll want to be aware of with this in general is CX4 and CX4-Lx are distinct chips and you shouldn't try to go between those (and possibly CX3 vs. CX3 Pro.) I don't have any experience with CX6 and newer.

What could still be a limitation is trace length and PCB quality, which could limit the ability to do PCI-E 4.0 in practice. Would be great if people reported whether PCI-E 4.0 works reliably for them after the firmware flash and what motherboard it has been tested on.

 

[jmsq]

What could still be a limitation is trace length and PCB quality, which could limit the ability to do PCI-E 4.0 in practice. Would be great if people reported whether PCI-E 4.0 works reliably for them after the firmware flash and what motherboard it has been tested on.

I got the CX5 cards explicitly for use in pcie 4.0 x8 scenarios to reduce my bandwidth bottleneck. In both a z690 aero-D and a z590 master it largely works as expected and I regularly see 60-80Gbps in iperf3. lspci also reports proper functioning of the pcie 4.0 x8 links.

 

[blunden]

I got the CX5 cards explicitly for use in pcie 4.0 x8 scenarios to reduce my bandwidth bottleneck. In both a z690 aero-D and a z590 master it largely works as expected and I regularly see 60-80Gbps in iperf3. lspci also reports proper functioning of the pcie 4.0 x8 links.

I see. Did the Ex and non-Ex cards have the same PCB number in your case?

 

[jmsq]

I see. Did the Ex and non-Ex cards have the same PCB number in your case?

I don’t have an actual Ex card to compare against due to price. Both of my cards are crossflashed from the 516A-GCAT to 516A-CDAT and the rest of my devices are making do with 414A-GCAT crossflashed to 416A-CCAT. For my use case it worked exactly as intended which was to provide extra bandwidth in my pcie 4.0 systems that were topping out at 52Gbps with the crossflashed 414A.

If you’re wondering what cross flashing gains on the 414A with such limited bandwidth and a physical x8 slot, it is able to link with switches that don’t support 40/50GbE like the QNAP QSW-M7308R-4X.

 

[jpmomo]

On ConnectX-6, if the firmware has a "secure-fw" flag, the sure way to reflash it is to use a MTUSB-1. On ConnectX-7 it's SOL right now, we need some ex-Mellanox with some insider knowledge to proceed. On ConnectX-8...hmm get me a CX8 and I'll tell you lol, probably the same as on CX7.

I will get you the cx8 as soon as you sort out the cx7

 

[foureight84]

Is it possible to bond two of these QSFP28 to one QSFP56 and connect to a ConnectX-6 or 7 to try and get more than a 100Gbe connection?