Help getting 56Gb on ConnectX-3 Pros

[CyrilB]

From Mellanox/NVIDIA developer:
Back to Back 56GbE connection without INI modification

Thanks, but this was discussed before:

• We are looking for a way to accomplish this under Windows where ethtool does not exist.
• The corresponding mlxlink is not compatible with ConnectX-3 adapters ("-E- Device is not supported").

 

[CyrilB]

No. Flint does not deal with .mlx files. The firmware dump file is still in bin format. Even if user names the file .mlx

I understand your objection (but cannot confirm nor approve because I have not gotten around to trying it).

Do you think mlxburn can apply just the .ini file, then?
I wonder if I can extract the .ini using "flint dc", edit the .ini and "mlxburn -d -config burn"?

 

[CyrilB]

No. Flint does not deal with .mlx files. The firmware dump file is still in bin format. Even if user names the file .mlx

You were correct; this did not help.
While "flint -i some_bin_file.mlx query full" and "flint -i some_bin_file.mlx verify" run through correctly, "mlxburn -fw some_bin_file.mlx ..." will not succeed.

In the mean time I searched for a different mechanism to incorporate .ini changes into my .bin file and came across this:

GitHub - BeTeP-STH/mft-scripts: Mellanox firmware files and MFT related scripts

Mellanox firmware files and MFT related scripts. Contribute to BeTeP-STH/mft-scripts development by creating an account on GitHub.

github.com

I believe you're the author, BeTeP. Would you suggest using it for adding lines to the .ini ?
(I mean by extracting the .ini using flint dc to extract the current .ini, editing it and applying the changes.)

 

[CyrilB]

I managed to update fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin (a Mellanox/NVidia download) with a modified version of MCX354A-FCB_A2-A5.ini. I used fs2_update_ini.py to accomplish this.

GitHub - BeTeP-STH/mft-scripts: Mellanox firmware files and MFT related scripts

Mellanox firmware files and MFT related scripts. Contribute to BeTeP-STH/mft-scripts development by creating an account on GitHub.

github.com

Then I used flint verify and flint burn.
Following the burn and a reboot, I could verify that the .INI file additions made it to the configuration by extracting it using flint dc.

-------------------------------------------------

Interestingly the following two lines were already present in the config before my addition:

port1_802_3ap_56kr4_ability = true
port2_802_3ap_56kr4_ability = true

So I only added:

port1_802_3ap_56kr4_enable = true
port2_802_3ap_56kr4_enable = true

-------------------------------------------------

However, unfortunately, these changes had no effect.
Perhaps this is due to the firmware version - the article suggesting this tweak is from 2014 and mentioned the firmware version.2.31.5050:

56 GbE over a ConnectX-3 VPI adapter

Hello, I have a pair of ConnectX-3 VPI adapters (MCX354A-FBCT) in Ethernet mode and am attempting to test out 56 GbE connectivity by directly connecting the adapters to one another via an FDR cable (MC2207128). However, the cards only establish link at 40 GbE. In fact, ethtool at least does not...

forums.developer.nvidia.com

-------------------------------------------------

I am not willing to downgrade the firmware on my adapters, so I am giving up for now.
If anybody has an idea as to what to try, please ask/advise.

 

[CyrilB]

Good find. Unfortunately, a lot of the newer firmware don't come with custom firmware files. According to ConnectX-3 custom firmware files, last version was 2.40.5030. Let me know if you go this route and which firmware you end up using. Thanks.

See my previous post: I succeeded in updating the configuration, but unfortunately, it had no effect.

 

[CyrilB]

If you already have the 40GBE firmware on the nic 56GBE should work "out of the box".
And about 56GBE not showing up in ethtool: 56GBE is a (end of life) mellanox proprietary implementation, other devices/software don't know it

... and possibly newer firmware versions don't support it either.

 

[Bjorn Smith]

... and possibly newer firmware versions don't support it either.

Have you considered testing in linux - just to see if the cards can in fact negotiate 56Gb/s?

If that works - somehow it should also be possible on windows - but its most likely easier to test if its even possible on Linux.

 

[CyrilB]

Have you considered testing in linux - just to see if the cards can in fact negotiate 56Gb/s?

If that works - somehow it should also be possible on windows - but its most likely easier to test if its even possible on Linux.

Thanks for the idea. I'm a Windows guy with very little Linux experience, but never say never...
If the issue won't let me sleep at night, I might try ;-)

 

[KevinH]

@CyrilB did you manage to get 56Gb/s link in Windows? I have it work in linux though.

 

[CyrilB]

@CyrilB did you manage to get 56Gb/s link in Windows? I have it work in linux though.

No, unfortunately not. I ended up adding a second adapter.
Which firmware version did you use?

 

[kevindd992002]

I'm doing something similar. I'm trying to modify the latest 2.42.5000 with a modified ini (to enable sriov) but it is not working with @BeTeP's tool. I used an older mlx file with the mlxburn tool and that worked in enabling sriov but I don't really want to use an older firmware.

Any idea where to get a newer mlx file?

 

[Oliver Mack]

My two CX3 Pros connect with 56GbE without doing anything, all that matters is the right cables


ethtool enp5s0
Settings for enp5s0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
56000baseCR4/Full
56000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
56000baseCR4/Full
56000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 40000baseCR4/Full
56000baseCR4/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 56000Mb/s
Duplex: Full
Auto-negotiation: on
Port: None
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000014 (20)
link ifdown
Link detected: yes