EMC Mellanox SX6005 and I²C for "external management"

[naptastic]

Can anyone offer guidance, or "The Easy Way" here?

I bought a pair of EMC Mellanox SX6005's before discovering the "beware these switches" thread. It's not the 6012, so I don't have to do the whole manual installation of MLNX-OS. But absolutely zero switch management is available in-band. I can't even ask for a firmware version. (Subnet management works fine AFAICT.)

So I bought an i2cdriver (i2cdriver.com) and I'm not 100% sure but I think that may also have been the wrong move. It speaks I²C, but it's not an actual bus controller (I think?) so connecting it to the switch... well, the voltages are good at least. I have no idea what to do next.

Linux has a Mellanox I²C platform driver, though I'm not sure exactly what it drives; anyway, it's not compiled, so that's on today's to-do list.

For what it is, this switch is fantastic. I really just want two things: fan control, and FDR10 support.

I replaced the fans with Noctuas and blocked the PWM pin so they run at 100%. They're not awful, but it would be nice if they could spin down.

As for FDR10, I have a set of cheap 40gigE transceivers. I know they can work, because I've connected two MCX354-A cards back-to-back, and they happily spoke 40gigE or FDR10 Infiniband to each other. This switch should be able to use those transceivers, shouldn't it?

 

[naptastic]

(Can I please attach an image? Thanks)

ALSO! If anybody knows for sure which pin is SDA and which is SCL that would be very helpful! Finding ground was trivial.

 

[naptastic]

Holy freaking crap, what is the upload limit here?! Am I supposed to just upload thumbnails and link to my own server or something?

 

[necr]

the program that talks to switch to change firmware is flint or mstflint (open-source counterpart). It should be able to talk to the switch both in-band and out-of-band. For out-of-band, I've used MTUSB-1 as it's a supported device. Fan control is an advanced topic, the examples from other threads were mainly about managed switches (AFAIR).

Can you please get the output of:
sudo ibswitches
sudo mstflint -d lid-<the lid number from the command above, for example 3> q
sudo mstflint -d lid-<number> dc

 

[naptastic]

Can you please get the output of:
sudo ibswitches
sudo mstflint -d lid-<the lid number from the command above, for example 3> q
sudo mstflint -d lid-<number> dc

Certainly:

[root]@[carbon][12:03:01][~]# ibswitches
Switch : 0x506b4b0300527dc0 ports 12 "SwitchX - Mellanox Technologies" base port 0 lid 6 lmc 0

[root]@[carbon][12:03:23][~]# mstflint -d lid-0x0006 q
FATAL - Can't find device id.
-E- Cannot open Device: lid-0x0006. MFE_UNSUPPORTED_DEVICE

[root]@[carbon][12:03:27][~]# mstflint -d lid-0x0006 dc
FATAL - Can't find device id.
-E- Cannot open Device: lid-0x0006. MFE_UNSUPPORTED_DEVICE

The same happens with `mst ib add --use-ibdr` both with and without a local device name (e.g., mlx5_0) and /dev/mst/SW* names.

 

[necr]

what about mstflint -d lid-6 q?
which version of mstflint do you have?
are you running OpenSM?

 

[naptastic]

I am running opensm. I customized one partition but otherwise it's running defaults. Using '-d lid-6 q' gives the same error:

FATAL - Can't find device id.
-E- Cannot open Device: lid-6. MFE_UNSUPPORTED_DEVICE

 

[necr]

mst could be conflicting with mstflint, it has a closed-source alternative of mstflint. Can you try to do the same command with flint (ibswitches; flint -d lid-x q; )? "Can't find device id" looks like a error within your system - I could suggest stopping the current server, booting from a stock Debian/Ubuntu, installing opensm and mstflint, and then issuing the commands above.

Or, order an MTUSB-1 and wait for delivery.

Edit: goddamn smileys

 

[naptastic]

It's going to take me some time, since I've got no way to boot metal from an .iso right now (you're allowed to say WTF) but you're right, both are definitely installed and from different places:

[root]@[carbon][08:28:06][~]# dpkg-query -S `which mstflint`
mstflint: /usr/bin/mstflint
[root]@[carbon][08:28:15][~]# dpkg-query -S `which flint`
mft: /usr/bin/flint

Are you sure that it matters, though? /usr/bin/flint is just a wrapper around /usr/bin/mstflint, and mstflint only links against distribution-provided libraries.

 

[naptastic]

Just an update: By setting force_link_speed to 7 and force_link_speed_ext to 30 in opensm.conf, I got the switch to force FDR10 on all ports. (Is there something more graceful than power cycling the switch for this?) Getting FDR10 did not allow my 40gbE optics to work though, so I'm not optimistic on that front. Maybe a firmware update would improve compatibility but these units were made in 2018, so they might already be on the latest version. Idk.