Flashing CX4131A-BCAT (40Gbe) to GCAT (50Gbe) what should I look out for?

[TonyA]

So, I've been doing a lot of reading over the last couple of weeks and while there's pretty extensive stuff on the ConnectX-3 cards, the ConnectX-4 stuff is a bit spotty.

I have two genuine CX-4131A-BCAT cards and one CX-4131A-GCAT card, obviously the GCAT is a 50Gbe card and reports itself as this using ethtool.

I have a couple of QSFP28 transceivers and some OM4 fibre cabling so am now ready to try things out.

I'm running Fedora so have installed mstflint from the repos and mlxburn from the Nvidia site.

The concern I have is that from what I've read the way to flash a 'genuine' firmware from the same card family was changed from ConnectX-3 to ConnectX-4 cards, I've read that I may now need to use a --allow-psid-change flag for example and also I may need to short out two pins, however my concern is that as I haven't been able to find a comprehensive ConnectX-4 upgrade post, I may well be missing other things as well.

I've google searched the site and while there are numerous posts that talk about the ConnectX-4 cards, because it's often in the context of (paraphrased) '....you do this for ConnectX-3, however I think this might have been changed for ConnectX-4....' it means I'm never going to be certain I've not missed reading something critical hidden away in a post I missed and I'd really rather not brick my cards if I can avoid it.

The BCAT cards report as firmware 14.12.1240 on 2.11.2015 with a PSID of MT_2430110027

the GCAT I can't read right now as Secure boot is still enabled on that box and it's running a job till Saturday when I can take it offline.

So, any help or pointers you can give would be much appreciated, this has been a great learning experience thus far, but I realise I'm probably just only slightly past the start of this journey.

 

[i386]

I've crossflashed a hpe 100gbe nic to vanilla mellanox firmware. For that I had to use the allow psid change parameter. I don't think that should be a problem.

If your card has qsfp28 connectors I would try to flash it with a 100gbe firmware (even when the cars has a pcie x8 host interface ). From my understanding mellanox only build one ic/asic per generation and then "configured" it by firmware what speeds it supports.
But like always make a backup of the existing firmware (and config/information about mac address in case you have to use the recovery mode) before trying to crossflash it to something different.

I think with the cx-5 family mellanox introduced signed firmware that could be a pita when trying to crossflash adapters, but I don't have any cards and therefore experience with the newer generations yet.

 

[TonyA]

Thanks, I realise this is training 101, but can I ask what command structure you used?

I'm terrified I'll make some stupid parsing error and brick the card.

Also, is it better to take the transceivers out before flashing a card or doesn't it matter.

Since I posted the initial post I'd actually been playing with the cards and the cable I had, I think I have certainly one and possibly two problems......

The first problem was quite easy to spot as the card in the box I can't take offline kept on going up and down and I wasn't sure why...so of course after messing around for 20 minutes unplugging and replugging cables I thought "...why don't I look at dmesg and see what that says..." (always the obvious things last eh?).

Anyway, I was a little surprised at the message in dmesg, it's not what I expected to see.

mlx5_core 0000:b5:00.0 enp267s0np0 Link Down
mlx5_core 0000:b5:00.0 enp267s0np0 Port Module Event: module 0, Cable plugged
mlx5_core 0000:b5:00.0 enp267s0np0 Link Up
mlx5_core 0000:b5:00.0 enp267s0np0 port_module:252 (pid 0): Port module event(error): WTF is this sheet, I'm not an Air Fryer!!!!

I think that was it, my memory may be a bit creative. So I'll need to sort out some extra cooling I suspect. Also, when the link is up, I can't ping the other card, obviously that may be the temp issues or it might be my old cable is fracked.

But at least I'm making some progress as I can see a 40000 link in ethtool between the boxes for short periods.

 

[TonyA]

Has the -allow-psid-change flag been removed from later versions of mstflint?

My version, 4.23.0, running on Fedora and installed from the Fed repos doesn't recognise the option.

I updated the firmware on my genuine GCAT version so I could get a 'sense' of the steps needed to do this and it seems to have worked OK, however, without the -allow-psid-change option in mstflint I'm somewhat stuck at being able to flash the GCAT firmware onto the BCAT cards.

I did read in another thread about replacing the .ini part of the bin file to 'create' a hybrid file with BCAT identifiers and GCAT firmware, although the thread was from 5 years back so don't want to go down a route that's likely superseded now.

Lastly, what's the best option for fitting a small fan to the Mellanox cards? The GCAT is definitely running toasty and as a result keeps disconnecting itself, as well as filling up dmesg with warnings.

 

[i386]

Has the -allow-psid-change flag been removed from later versions of mstflint?

It still seems to be there:
-allow_psid_change

mstflint/flint/cmd_line_parser.cpp at master · Mellanox/mstflint

Mstflint - an open source version of MFT (Mellanox Firmware Tools) - Mellanox/mstflint

github.com

what's the best option

A server/workstation with decent airflow

 

[TonyA]

See, the problem with reading forum postings is that people post hyphens instead of underlines........

Thanks, flashing the BCAT now.

The Workstation in question is a Dell T7920 so 'should' have decent airflow and I've deliberately mounted the card away from any others, but its still running toasty, so think I'll have to find a specific solution.

 

[i386]

Thanks, flashing the BCAT now.

The famous last words

How did it go?

 

[TonyA]

The famous last words

How did it go?

Flash seems to have gone OK and ethtool now showing it as a 50Gbe capable device.

I've been able to do some card to card pings and even a very quick copy, but the heat issue is causing cutout so will definitely need some kind of customised cooling before I can do a proper test 'under load' and see the results.

While I understand your earlier point about a decent workstation with proper airflow, the Dell T7920 isn't some cheap home generated piece of tot, it's a top level pretty pricey workstation, so the fact that the cards are both generating enough heat in that environment lends itself to a problem with heat generation by the cards, rather than not enough airflow in the box. For example I've been running a couple of x540-T2's doing a circa 200TB data copy from one T7920 to another and while the x540's run warm, they don't run hot to the point of cutting out which the Mellanox's are doing before they are even under copying load, which I honestly have to say is quite disappointing.

I'll get there and will have a look if later Mellanox generations ConnectX-5 etc... suffer from the same heat related issues.

 

[i386]

While I understand your earlier point about a decent workstation with proper airflow, the Dell T7920 isn't some cheap home generated piece of tot, it's a top level pretty pricey workstation, so the fact that the cards are both generating enough heat in that environment lends itself to a problem with heat generation by the cards, rather than not enough airflow in the box. For example I've been running a couple of x540-T2's doing a circa 200TB data copy from one T7920 to another and while the x540's run warm, they don't run hot to the point of cutting out which the Mellanox's are doing before they are even under copying load, which I honestly have to say is quite disappointing.

I'll get there and will have a look if later Mellanox generations ConnectX-5 etc... suffer from the same heat related issues.

Can you run mget_temp -d and report the temperature back?

 

[TonyA]

Can you run mget_temp -d and report the temperature back?

Two cards, one BCAT reflashed and one GCAT, both running 14.32.1010 firmware, in two different T7920s, but both not 'next to' any other cards.

62 and 71 respectively (I assume this is Centigrade).

dmesg on both machines is full of......

[37649.408823] mlx5_core 0000:xx:xx.x: port_module:252:(pid 0): Port module event[error]: module 0, Cable error, High Temperature
[37649.408843] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[37713.462854] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[37719.407007] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0
[37936.401439] mlx5_core 0000:xx:xx.x: port_module:252:(pid 0): Port module event[error]: module 0, Cable error, High Temperature
[37936.401458] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[38000.455826] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[38006.399622] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0
[38412.388834] mlx5_core 0000:xx:xx.x: port_module:252:(pid 0): Port module event[error]: module 0, Cable error, High Temperature
[38412.388853] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[38476.443889] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[38482.386925] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0


and


[77006.141418] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[77070.205991] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[77076.140582] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0
[77223.138798] mlx5_core 0000:xx:xx.x: port_module:252:(pid 0): Port module event[error]: module 0, Cable error, High Temperature
[77223.138817] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[77287.203728] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[77293.137874] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0
[77433.136202] mlx5_core 0000:xx:xx.x: port_module:252:(pid 0): Port module event[error]: module 0, Cable error, High Temperature
[77433.136220] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[77497.201466] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[77503.135331] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0
[77622.133884] mlx5_core 0000:xx:xx.x: port_module:252:(pid 0): Port module event[error]: module 0, Cable error, High Temperature
[77622.133902] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[77686.199485] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[77692.133067] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0
[77811.131611] mlx5_core 0000:xx:xx.x: port_module:252:(pid 0): Port module event[error]: module 0, Cable error, High Temperature
[77811.131629] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 0 0
[77875.197427] mlx5_core 0000:xx:xx.x: Port module event: module 0, Cable plugged
[77881.130803] mlx5_core 0000:xx:xx.x: temp_warn:168:(pid 0): High temperature on sensors with bit set 1 0

So, as you can see, this isn't as if it's one 'outlier' card.

I have one other BCAT card that I'll try to find time to switch out with the existing BCAT, flash update that one to GCAT and see if it reports the same errors.

 

[i386]

62 and 71 respectively (I assume this is Centigrade).

that's even cooler than mine

I think the temperature warning are from dom on transceivers...

 

[TonyA]

that's even cooler than mine

I think the temperature warning are from dom on transceivers...

Don't forget, these have no load on them at all, I'd expect them to get warmer if they were doing 50Gbe copies.

I'll have to do some deep dive to see if the transceivers are producing the dmesg messages, the easiest way would be to pull them and see what happens to the card temps, but I'm just about to go offline for the evening so will try to spend more time on it manana.......

 

[klui]

Either your modules are bad or you installed your card where there's inadequate airflow. I have no problems keeping a link when the controller's temperature is over 70C. What's the output of ethtool -m <device>?

Open up your case and set a fan blowing on the card and see if it improves. You should be able to adjust your workstation's fan profile to a more aggressive one. Airflow requirements are documented at Specifications.

 

[TonyA]

Either your modules are bad or you installed your card where there's inadequate airflow. I have no problems keeping a link when the controller's temperature is over 70C. What's the output of ethtool -m <device>?

Open up your case and set a fan blowing on the card and see if it improves. You should be able to adjust your workstation's fan profile to a more aggressive one. Airflow requirements are documented at Specifications.

I'm not sure that you read my posts fully as I state the machines I'm using and how I've positioned the cards.

It is of course possible that my transceivers are causing the problem, but as I don't think there's a way to probe temps on a transceiver (unless you know differently) than the cards seem to be running hot.

I'm trying to determine what other analysis data may be available to help me identify the issue, in the first instance I've ordered some new OM4 cables as it's possible the old cable I'm using is causing the problem (unlikely, but I'm now in exclude anything that might be causing the problem mode). The other option is to try to find a cheap DAC cable and see what that does, so 'll be scouring E-Bay later.

sudo ethtool device
[sudo] password for chumpchange:
Settings for device:
    Supported ports: [  ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: None     RS     BASER
    Advertised link modes:  10000baseKR/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: Other
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
    Link detected: no (Overheat)

 

[klui]

I don't think there's a way to probe temps on a transceiver

LOL. That's what ethtool -m does.

 

[TonyA]

LOL. That's what ethtool -m does.

I'm not quite sure of your need to post a (my paraphrase) 'you idiot' LoL comment?

I'd posted in the forum a few days back that I had no experience in opticals or transceivers and literally started learning about them only a couple of weeks back.

I noticed though that you didn't address the fact that you didn't seem to have read my earlier posts properly, as your question regarding the case and suggesting having a fan blowing on the card was somewhat redundant.

Thanks for your constructive help though.

 

[TonyA]

I've now updated the 2nd BCAT card and have found using ethtool -m that the temperature limit on the transceivers I'm using is 58c, so that would seem to be the issue. I have no idea if that's a low temperature limit on transceivers or it's the norm. It does sort of confirm though that I'll have to come up with some kind of active cooling on the cards to get the overall and therefore transceiver temps down.

There's some interesting posts around about people mounting fans on Mellanox cards if you google, I'll also need to look at what, if any, extra fan power points there are in the case.

 

[blunden]

There's some interesting posts around about people mounting fans on Mellanox cards if you google, I'll also need to look at what, if any, extra fan power points there are in the case.

If you don't have any additional fan headers available, you could always just use a cheap Y connector to run two fans off a single header. Unless the fans you are using draw unusually amounts of power, it should be perfectly within spec of the motherboard to do so.

 

[TonyA]

If you don't have any additional fan headers available, you could always just use a cheap Y connector to run two fans off a single header. Unless the fans you are using draw unusually amounts of power, it should be perfectly within spec of the motherboard to do so.

Thanks, I'm pretty sure they're be some headers inside the case and on the mobo, it's just a matter of having to down the box and have a look inside with a good light source.

I do know that on the underside where the drives are mounted in these units there are extra fan headers for mounting up to 8 drive flexbays, but I'm pretty sure those fan headers are 'managed' inside the BIOS, so I may need to look at what tweaks may be possible using something like i8kutils which I haven't played with for a while.

I'm also having a look at if it's possible to turn off the Temperature Monitor warning using ethtool -m, from reading the specsheet for the transceivers they should be good up to 70c, however the "Module Temperature High Alarm Threshold" is set for 58c and the "Module Temperature High Warning" is set for on, so I think that's where the cutout is getting instigated from. If I can turn "Module Temperature High Warning" to off, or set "Module Temperature High Alarm Threshold" to say 68c I may be able to get these stable enough to do some short term testing, but I do think I'd still prefer to get some active cooling on the cards in the long term just for peace of mind.

 

[TonyA]

Having now had time to do some further research and having had a ponder about it, I think the solution is to track down something like one of these......





However, in my instance as I'm using a Dell box I'd want to replace the fan with a PWM version, correctly wired as Dell appear to use non standard wiring. And then using either i8kutils or pwmcontrol, which are both Linux utilities, ensure that there's automatic control of the fan speed dependent on the card/optic temperatures.

I have Heath Robinson rigged up a temporary fan in one of the boxes just to see what the difference is, it's not attached to the Mellanox, but is just blowing air across it front to back, that's brought the sensor temperature to about 41c from about 60c, so it helps, but I'm pretty sure that a more customised solution would get the temps down even further.

mlx5_0000:xx:xx.x-virtual-0
Adapter: Virtual device
temp1:        +41.0°C