Mellanox flashing ConnectX-5 to ConnectX-5 Ex ? (PCIe 3.0 to PCIe 4.0 ?)

[MichalPL]

I am wondering if is it possible for "flash" Mellanox MCX516A-CCAT to MCX516A-CDAT ?

Are ConnectX-5 and ConnectX-5 Ex based on the same chip and electronic board and PCIe 4.0 support is only the software thing?

or better to not try it - and there is a big (or even 100%) risk having small green brick then

Best,
Michal

 

[necr]

I am wondering if is it possible for "flash" Mellanox MCX516A-CCAT to MCX516A-CDAT ?

They're really close, both dual-port, should be possible. Do you have Mellanox or OEM cards? OEMs, like HP come with secure-fw, meaning crossflash is nearly impossible without an i2c flash access.

Are ConnectX-5 and ConnectX-5 Ex based on the same chip and electronic board and PCIe 4.0 support is only the software thing?

ConnectX-5 card is VPI (both IB and Eth), and it's really far from Ex cards (Eth only). Please dump the firmware config here ((mst)flint -d <your card> q; (mst)flint -d <your card> dc;
Does your system support PCIe 4.0 in that particular slot (lspci -s <addr> -vvv)

or better to not try it - and there is a big (or even 100%) risk having small green brick then

My CX6 cross-flash has been successful, there are also successful reports of CX5 cross-flashes here.

 

[MichalPL]

Today arrived at the office to test a few things

Do you have Mellanox or OEM cards? OEMs, like HP come with secure-fw, meaning crossflash is nearly impossible without an i2c flash access.

This one is probably pure Mellanox (I bought just 1, my first ConnectX-5 to test it, now it's fully working under Windows10 but it's PCIe 3.0 x16 like cheaper ConnectX4 100GbE) also it' is "too fast" to use it in something else than server (nice one, to fully utilize it if PCIE 4.0 will be working).

But today I flashed also 455A-FCAT to ECAT (40GbE to 100GbE) from DELL and it's also easy only difference is at Windows 10 drivers fail to start it, but you can still flash it easily (Windows command):
flint -d mt4115_pciconf0 -i c:\M\fw-ConnectX4-rel-12_28_2006-MCX455A-ECA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin -allow_psid_change burn
and to switch Ethernet permanently:
mlxconfig -d mt4115_pciconf0 set LINK_TYPE_P1=ETH

but I am still worry about flash PCIe 3.0 to 4.0, my logic says something inside should be different to support double the speed. hmm...
CCAT to CCHT no problem but CDAT hmm.... especially when they are changing name from EN to EX

https://www.mellanox.com/files/doc-2020/pb-connectx-5-en-card.pdf

ConnectX-5 card is VPI (both IB and Eth), and it's really far from Ex cards (Eth only).

Fortunately I need Eth only but still Eth and IB is nothing to PCIe 3.0 and PCIe 4.0 I think - but not sure :/

Please dump the firmware config here ((mst)flint -d <your card> q; (mst)flint -d <your card> dc;

good idea

Does your system support PCIe 4.0 in that particular slot (lspci -s <addr> -vvv)

No - not yet but I will organize it when working

My 2 today test bench are (old workstations):
TR 1950x - I can connect 4 PCIe 3.0 x16 cards
Xeon 1650 v2 - I can connect 2 PCIe 3.0 x16 cards and one x4.


Problem with PCIe 4.0 and 2 ports is, how to saturate it (200GbE) from SSD's - So waiting for TR 6000 or Xeon 12gen

Fastest testbench I can organize for while (until it goes as a workstation) is 12900kf with DDR5 memory, If using PCIe 5.0 slot for Mellanox, still 12 PCIe 4.0 lanes left so about 18 GB/s from NVME is possible, but 2x100GbE is still faster - about 22GB/s. 5950x is too slow in single thread (I believe) and limited bandwidth thru chipset (~8GB/s).

My CX6 cross-flash has been successful, there are also successful reports of CX5 cross-flashes here.

Can you recognize on the pictures if they looks similar or same to EX PCIe 4.0 cards ?






*sorry for the pictures quality, I forgot phone from the car and use old samsung gear wristwatch to take a photos

 

[MichalPL]

hmm..

They're really close, both dual-port, should be possible.

found photos on ebay and one CDAT (PCIe 4.0) have same PCB number like my "CCAT" (PCIe 3.0)

Mellanox Cx516a Connectx-5 EX En Dual Port 25 / 50 / 100gbe Qsfp28 Adapter for sale online | eBay

Find many great new & used options and get the best deals for Mellanox Cx516a Connectx-5 EX En Dual Port 25 / 50 / 100gbe Qsfp28 Adapter at the best online prices at eBay! Free shipping for many products!

www.ebay.com

and here CCAT is same like my, CDAT have upgraded PCB number, but looks also same... hmmm

Mellanox CONNECTX-5 100GbE VPI Adapter Card ED 1x MCX516A-CDAT 1x MCX516A-CCAT | eBay

Find many great new & used options and get the best deals for Mellanox CONNECTX-5 100GbE VPI Adapter Card ED 1x MCX516A-CDAT 1x MCX516A-CCAT at the best online prices at eBay! Free shipping for many products!

www.ebay.com

maybe just try it if not working I will have a green brick ?

 

[necr]

maybe just try it if not working I will have a green brick ?

You could try this - there are 2 partitions on the internal flash, so it shouldn't turn into a brick in any case. There are also flash_recovery pins that you can short beforehands to ensure that you have access to the card in recovery mode.

Still, to me personally it's unnecessary risk to do this without a PCIe 4.0 system to test on.

 

[MichalPL]

Challenge accepted

 

[MichalPL]

flashing 12900K waiting for me

 

[MichalPL]

Ok 100% working! PCIe 4.0 16GT/s. 100GbE on both ports - connected temporary to i9 12gen (but next server should be PCIe 4.0 or 5.0 too).

Name of the card (Part Number) is still "CCAT" but working as PCIe 4.0 - and model name is "Ex", 2x 100GbE working too

 

[MichalPL]

Still, to me personally it's unnecessary risk to do this without a PCIe 4.0 system to test on.

Thank your for help
Yes until PCIe 4.0 no sense to flash it (now we know it's possible), the cheapest way to test PCIe 4.0 is Ryzen3 3100x, but it's good just for testing - we were considering it as a test bench but it's too slow for other things.