Beware of EMC switches sold as Mellanox SX6XXX on eBay

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
A

andvalb

Member
Feb 15, 2021
27
25
13
Ulyanovsk, Russian Federation
Stephan said:
Trying to wrap up the project... stuck with a few questions. I run two converted EMC SX6012 now with 3.6.8012. Only software modification is some fan taming added to /etc/rc.d/rc.local.
  1. What is the purpose of the "Host ID"? I set this to the 12 hex digits of mgmt0 ethernet MAC.
  2. Web interface is not showing information from within the power supplies, but only this:View attachment 22021
    I checked what power supplies are in there, they are CLP0212FPXX5Z01A open frame PSUs. Then I checked the pinout of HDR3 (see https://www.sager.com/_resources/pdfs/product/CLP0212.pdf) and it seems to me there is no I2C capability? So what the web interface shows is correct, there is just no information like input voltage etc. Is this perhaps only for bigger models with beefier hot-plug PSUs?
  3. CPU cooling of PPC could perform better at lower airflow. Original heatsink has ~57 mm distance between center of holes, height around 10mm. Is somebody running the switch with a suitable replacement?
Click to expand...
2) Yes - its ok.
3) I am changed heat sink to the some one from the old video card - runs at the same temp as before lowering fan speed.
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,591
1,051
113
Hmm, so I've made some progress on the engineering sample SB7700, but not much.

I got a disk image from @bentwire which made a couple of things a bit easier

The problem: The switch gets stuck at "Modules are being configured" with not helpful output in the logs.

I have updated the BIOS from the ancient engineering sample version to the current version -> No change

And I have noticed that the ASIC Firmware was MT_1880110032, which is for an SB7790 (unmanaged)
I have flashed the MT_1870110032 firmware (SB7700, managed) because this seems to be the right one

-> No chance

I've waited for 10 minutes for the system to initialise, but nothing happens


Any ideas what I could to next or how to debug this issue?

The only issues I can see in the log right now:

Code: 
   15.406741] tc_ingress_policy[905]: Cannot find device "mgmts0"
[   15.414521] tc_ingress_policy[905]: mDNS policing rate=4000kbit burst=400k
[   15.422447] tc_ingress_policy[905]: Cannot find device "mgmts0"
[   15.430472] tc_ingress_policy[905]: Ingress policing enable on interface mgmts0 rate=9000kbit burst=900k
[  OK  ] Started SYSV: Starts and stops TC ingerss policy.
[   15.441618] tc_ingress_policy[905]: Cannot find device "mgmts0"
[   15.456622] tc_ingress_policy[905]: Cannot find device "mgmts1"
[   15.457207] tc_ingress_policy[905]: Ingress policing enable on interface mgmts1 rate=9000kbit burst=900k
[   15.457320] tc_ingress_policy[905]: Cannot find device "mgmts1"
[   15.462769]          kvmStarting SYSV: load kvm kernel modules...[971]:
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,591
1,051
113
Yes, in recovery mode (add single to bootflags) they do.

And they work. I can assign an ip address to them and use them... Actually I used them for tftp'ing the ASIC firmware and and backing up the config, because the USB port doesn't seem to work
 
B

bentwire

New Member
Feb 19, 2022
9
3
3
I wonder if the udev rules are looking for the wrong thing and can't rename them to mgmts*... What names do they have?
 
B

bentwire

New Member
Feb 19, 2022
9
3
3
@NablaSquaredG
Do you see anything like this during boot?


Code: 
[   10.378951] rename_ifs[495]: Running renaming interfaces
[   10.570080] rename_ifs[495]: Renaming: MAC: e4:1d:2d:ab:34:9a ifindex: 2 name: mgmts0
[   10.579454] rename_ifs[495]: Renaming: MAC: e4:1d:2d:ab:34:9b ifindex: 3 name: mgmts1
 
A

andvalb

Member
Feb 15, 2021
27
25
13
Ulyanovsk, Russian Federation
NablaSquaredG said:
Hmm, so I've made some progress on the engineering sample SB7700, but not much.

I got a disk image from @bentwire which made a couple of things a bit easier

The problem: The switch gets stuck at "Modules are being configured" with not helpful output in the logs.

I have updated the BIOS from the ancient engineering sample version to the current version -> No change

And I have noticed that the ASIC Firmware was MT_1880110032, which is for an SB7790 (unmanaged)
I have flashed the MT_1870110032 firmware (SB7700, managed) because this seems to be the right one

-> No chance

I've waited for 10 minutes for the system to initialise, but nothing happens


Any ideas what I could to next or how to debug this issue?

The only issues I can see in the log right now:

Code: 
   15.406741] tc_ingress_policy[905]: Cannot find device "mgmts0"
[   15.414521] tc_ingress_policy[905]: mDNS policing rate=4000kbit burst=400k
[   15.422447] tc_ingress_policy[905]: Cannot find device "mgmts0"
[   15.430472] tc_ingress_policy[905]: Ingress policing enable on interface mgmts0 rate=9000kbit burst=900k
[  OK  ] Started SYSV: Starts and stops TC ingerss policy.
[   15.441618] tc_ingress_policy[905]: Cannot find device "mgmts0"
[   15.456622] tc_ingress_policy[905]: Cannot find device "mgmts1"
[   15.457207] tc_ingress_policy[905]: Ingress policing enable on interface mgmts1 rate=9000kbit burst=900k
[   15.457320] tc_ingress_policy[905]: Cannot find device "mgmts1"
[   15.462769]          kvmStarting SYSV: load kvm kernel modules...[971]:
Click to expand...
Can you compare fru data between models?
 
Stephan

Stephan

Well-Known Member
Apr 21, 2017
1,013
781
113
Germany
Bought two more dual-PSU SX6012 i.e. EMC 100-886-236-04 from kimbrer dot com for funny 119 EUR a piece plus tax. Contact ct@ maybe they get in more. Not sure if they come with rails but those will be for fan experimentation.

Since EMC is using this monstrous metal encasing to hold a single switch, there is a possibility for fan modification. I plan to design and 3D print a fan duct and convert from internal Delta 4cm fans to external 12cm Noctua fans. Like so (excuse the mspaint job):
emc.jpg
I'll solder two small PCBs to convert the SENSE pin signal of both Noctuas from 200..2000/3000 rpm to the original Delta range (4000..18000rpm). Because otherwise the switch will complain of failed fans. Probably by using an ATtiny13a and a LP2954 to get from 12V to 5V, plus two low-ESR caps of 1uF on input and 2.2uF on output. Alert buzzer on PCB for good measure if a fan stops. I'll be feeding the modified SENSE of one fan to two open collector SENSE inputs on the Mellanox 4pin headers, so fan1+2 and 3+4 will have the same RPM signal. Will need to research first what pressure the original Deltas deliver in order to not be that much off.

Also bought two rails for the price of a switch each from Poland from ebay for 19" installment.

Then I bought this:
502585814.jpg
 
  • Like
Reactions: ttvd
S

solon

Member
Apr 1, 2021
65
5
8
For me staying within 1U wasn't neccesary, so this is how I solved it. 200mm hole in the top, cut a piece of plastic out of a folder to make an airguide which effectively blocks off the rear, air now blows onto the top of and under the pcb, and is vented out the front. These noctua fans are 17db, which is alot less than ambient.

Beware! The fan headers look like industry standard, but they're not. I have the pinout up somewhere in this topic I think. Just attaching a industry standard connector will fry your fan.

edit: these are the fans that were in my unit: FFB0412UHN-C
I compared the throughput to the 200mm noctua. Don't remember what the numbers were for that, but is was close enough that seeing as mine doesn't live in a rack and won't be doing much heavy lifting it seemed unlikely to be a problem.
 

Attachments

Last edited:
  • Like
Reactions: cy384 and Stephan
Stephan

Stephan

Well-Known Member
Apr 21, 2017
1,013
781
113
Germany
solon said:
For me staying within 1U wasn't neccesary, so this is how I solved it.
Click to expand...
Did you mod binaries to not trigger alerts because rpm is below 4000? Same pwm%/rpm curve as for original Deltas? The 3D printed baffle will also be no longer 1U, will put a 1-2U cable guide between switches.
 
S

solon

Member
Apr 1, 2021
65
5
8
So far I haven't even got as far as actually going through the guide to turn this into a proper melanox switch. I am lacking motivation to go through it since the other infiniband switch I already have already does 98% of what I require of it. If I'm honest with myself I'll probably have to run out of other projects before I'm bored enough to bother with it any further, or if my two internet connections somehow start stacking up to more than 1Gbit which would cause the good old UTP network cables to become the bottleneck. I expect it will probably protest about the less than 4000rpm fanspeeds if that's a thing. I honestly have no idea how to go about modding binaries for this purpose so my first inclinitation would probably be to see about spoofing it with some sort of multiplier circuit - assuming that the warnings actually have an effect on functioning, otherwise, for my inconsequential home network, I'd probably just leave it.

This mod probably wouldn't be great even in 2U as the 200mm noctua is about 1/2U in height. Wouldn't leave much room for airflow. On the other hand, if you have several in the rack, you could simply make holes and attach two units side by side in 1U, and if you leave 2U above, it ought to be fine and would still save you 1U of space (1x 1U + 2U airspace instead of 2x 1U+1U airspace). You could conceivably 3D print an adapter to give the noctua as much slant as will fit in 2U of space.

As a third idea, you could butcher the top and entire back of the switch case and make an adapter to give a 200mm fan enough inclination so it would fit in 3U. 3U is 13,335cm, so that should mean the fan needs about 45 degrees of inclination if you utilize the full 3U of height. The 200mm fan is just as wide as the switch case, so that should give you really good airflow. Seems like a fair solution if you have two switches side by side. Hmmm. possibly a bit fancy when just attaching directly to the lid won't make an appreciable difference.
 
S

shuguet

New Member
Apr 8, 2022
6
0
1
Hello there,
So I've got 2 of what looks like genuine SX1012, however they are running some old software versions (PPC_M460EX 3.4.1120 and SX_PPC_M460EX SX_3.4.0012 respectively), with old firmware (Firmware version: 9.3.1750 and Firmware version: 9.3.1260).

I wasn't able to find a source/link in the thread to download any/all of the intermediate software versions to get from current state to latest available.
Can someone provide that?

Also, it looks like one of the switch has VPI enabled (but the "Gateway" tab in the webUI is still grayed out), while the other only has eth-single-switch as available profile.
My goal here is to ultimately connect the 2 switch with an MLAG and use them for 40GB, 10GB and 4x10GB breakouts connections. I don't really need 56GB nor IB. Do I need to track down a license in my case or will I be able to achieve that with the current setup (once they are both upgrade to same version, ideally).

Any help appreciated!
 
Stephan

Stephan

Well-Known Member
Apr 21, 2017
1,013
781
113
Germany
shuguet said:
Except for the fact that I have no idea on how to do that, sure, happy to help.
Click to expand...
All steps expressedly without any warranty whatsoever. Another person should look over this and check that I have not made any error. Only then continue.
  1. Login and enable shell access
    1. enable
    2. conf t
    3. license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
    4. fae cable-stamping-unlock 40g_lr4
    5. conf write
    6. (wait a little?)
    7. _shell
  2. Remove U-Boot password so you can always boot/flash your way back into the switch
    1. /opt/tms/bin/mddbreq /config/db/initial set modify - /system/bootmgr/password string ''
    2. eetool -a bf -s UBPASSWD=""
  3. Dump EEPROMs
    1. /opt/tms/bin/mellaggra _read_fru 1 0x51 1000 fru_backplate.bin
    2. /opt/tms/bin/mellaggra _read_fru 0 0x50 1000 fru_cpu.bin
    3. Copy them off via SFTP etc. ZIP and please send them to me. Needed to finally clear remaining SX6012 vs true SX1012 differences...
  4. Make the switch ethernet only
    1. system profile eth-single-switch
    2. This may completely erase your config, if so, repeat what is written in Step 1.
  5. Try updating straight to e.g. through the web GUI
  6. I will send you another shell command to apply after that update via PM
  7. Quieten down the fans (this is for 2-PSU model only probably - if you get whining, increase FAN_MIN)
    1. mount -nwo remount,rw /
    2. vi /etc/rc.local
      1. I hope you know how to use vi. If not, hit d d a couple of times to delete everything, then i to insert, :wq to save
    3. Bash: 
      #!/bin/sh

touch /var/lock/subsys/local

# mount -nwo remount,rw /
# vi /etc/rc.d/rc.local

FAN_MIN="27"
FAN_MAX="50"
WAIT_MAX="10" # 5 minutes

MDREQ1="/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string"
MDREQ2="fan_number int8 1 fan_speed int8"
MDREQ3="set_max uint8"

i=1
while :; do
PID=$(pidof clusterd)
if [ -n "$PID" ]; then
sleep 60
echo "Adjusting fan speed"
$MDREQ1 "/MGMT/FAN1" $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
break
else
sleep 30
i=$((i+1))
if [ $i -gt $WAIT_MAX ]; then
echo "Timeout waiting for clusterd"
break
fi
fi
done

exit 0
 
Last edited:
  • Like
Reactions: cy384, Labs and klui
S

shuguet

New Member
Apr 8, 2022
6
0
1
Stephan said:
All steps expressedly without any warranty whatsoever. Another person should look over this and check that I have not made any error. Only then continue.
  1. Login and enable shell access
    1. enable
    2. conf t
    3. license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
    4. fae cable-stamping-unlock 40g_lr4
    5. conf write
    6. (wait a little?)
    7. _shell
  2. Remove U-Boot password so you can always boot/flash your way back into the switch
    1. /opt/tms/bin/mddbreq /config/db/initial set modify - /system/bootmgr/password string ''
    2. eetool -a bf -s UBPASSWD=""
  3. Dump EEPROMs
    1. /opt/tms/bin/mellaggra _read_fru 1 0x51 1000 fru_backplate.bin
    2. /opt/tms/bin/mellaggra _read_fru 0 0x50 1000 fru_cpu.bin
    3. Copy them off via SFTP etc. ZIP and please send them to me. Needed to finally clear remaining SX6012 vs true SX1012 differences...
  4. Make the switch ethernet only
    1. system profile eth-single-switch
  5. Try updating straight to e.g. through the web GUI
  6. I will send you another shell command to apply after that update via PM
  7. Quieten down the fans (this is for 2-PSU model only probably - if you get whining, increase FAN_MIN)
    1. mount -nwo remount,rw /
    2. vi /etc/rc.local
      1. I hope you know how to use vi. If not, hit d d a couple of times to delete everything, then i to insert, :wq to save
    3. Bash: 
      #!/bin/sh

touch /var/lock/subsys/local

# mount -nwo remount,rw /
# vi /etc/rc.d/rc.local

FAN_MIN="27"
FAN_MAX="50"
WAIT_MAX="10" # 5 minutes

MDREQ1="/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string"
MDREQ2="fan_number int8 1 fan_speed int8"
MDREQ3="set_max uint8"

i=1
while :; do
PID=$(pidof clusterd)
if [ -n "$PID" ]; then
sleep 60
echo "Adjusting fan speed"
$MDREQ1 "/MGMT/FAN1" $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
break
else
sleep 30
i=$((i+1))
if [ $i -gt $WAIT_MAX ]; then
echo "Timeout waiting for clusterd"
break
fi
fi
done

exit 0
Click to expand...
Thank you for the steps!
I’ll wait for confirmation from another veteran of the thread, then proceed.

Also, what is that «Gateway » grayed out tab on the WebUI?
Is that for IB stuff (then I do not care) or is that for L3 inter-VLAN traffic/routing (then I care a lot)?
 
Last edited:
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,591
1,051
113
shuguet said:
Also, what is that «Gateway » grayed out tab on the WebUI?
Is that for IB stuff (then I do not care) or is that for L3 inter-VLAN traffic/routing (then I care a lot)?
Click to expand...
It is if you use the SX6036 as a Gateway for IB<->Ethernet

You'd have to apply another license for that


@All:
Wouldn't it be a good idea to start a new Mellanox SX Switch megathread (similar to the ICX6610 one)?
The knowledge about those switches is spread across this thread and pretty well hidden for newbies.
 
  • Like
Reactions: nbritton and Necrotyr
Necrotyr

Necrotyr

Active Member
Jun 25, 2017
206
52
28
Denmark
It would probably be smart to combine all the tips, tricks and quirks in a single post, instead of spread over 1100-ish posts.
 
You must log in or register to reply here.
Top Bottom