Beware of EMC switches sold as Mellanox SX6XXX on eBay

[Renat]

Generally it should work: with force type.
But before try to install FDR14 Speed License.
"IB Max licensed speed: FDR10" maybe this is bug.

 

[sbonikowske]

the switch says licensed for 56g ethernet
MSX6036T-1BRR-1 [standalone: master] (config) # show system capabilities
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Not supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb

MSX6036T-1BRR-1 [standalone: master] (config interface ethernet 1/21) # speed ?
1G Port runs at 1Gbps
10G Port runs at 10Gbps
40G Port runs at 40Gbps
MSX6036T-1BRR-1 [standalone: master] (config interface ethernet 1/21) # speed 56G force
% interface Eth1/21 does not support the chosen speed (56000)

 

[Renat]

I wrote about " IB Max licensed speed: FDR10 ". Early there was a free license for FDR14.
maybe this is a bug.

With mc2207128-003 card worked on 56G Eth.

 

[sbonikowske]

yeah, I am not using IB. I am using ethernet. as of 3.4 you don't even need a different license for ethernet 56g anymore.

 

[Renat]

well, then I can`t nothing to suppose if you don`t want to read message.

 

[sbonikowske]

I have looked for an fdr14 license and as far as I can tell it doesn't exist. from what I have read there are no licenses for IB except fabric monitor. Everything else was included.

 

[Stephan]

I don't have a SX6036 at hand, try googling the two words pastebin efm_sx_eth_speed_sw_limit maybe not all ports are licensed. Cable is FDR14 with a little sticker on it? Did you force both sides to 56000? If you force the CX3 to 56000 with the cable, does link come up?

 

[NablaSquaredG]

I try to force a port to 56g and it won't let me.

MSX6036T-1BRR-1 [standalone: master] (config interface ethernet 1/21) # speed ?

SX6036T (T Suffix!) does not support 56G.
You would have to flash a different backplane firmware (the one for normal SX6036F or SX6036G) and patch the PN in the EEPROMs (and there are some magic flags which you would also have to patch)

 

[sbonikowske]

SX6036T (T Suffix!) does not support 56G.
You would have to flash a different backplane firmware (the one for normal SX6036F or SX6036G) and patch the PN in the EEPROMs (and there are some magic flags which you would also have to patch)

That explains so much! Thank you! it's odd that the switch says 56g when it's really locked to 40g.

 

[Stephan]

SX6036T (T Suffix!) does not support 56G.

Why is that even locked? Silicon binning? Marketing?

 

[sbonikowske]

Why is that even locked? Silicon binning? Marketing?

I looked the 6036G/F have 4tb/s backplanes while the T only has 2.9tb/s. so they are different. I don't know enough to say for sure, but I imagine that will be an ASIC limitation.

For me I am unlikely to ever need more than 10-15 ports bumping to 56g shouldn't cause any issues. Just have to find out how to do it.

 

[NablaSquaredG]

Why is that even locked? Silicon binning? Marketing?

Market segmentation. I did successfully "unlock" / "crossflash" 2 SX6036T to SX6036G.
I doubt there was any meaningful silicon binning going on.

I looked the 6036G/F have 4tb/s backplanes while the T only has 2.9tb/s. so they are different. I don't know enough to say for sure, but I imagine that will be an ASIC limitation.

No they are not different. Hardware-wise the switches are completely identical.

 

[kapone]

All steps expressedly without any warranty whatsoever. Another person should look over this and check that I have not made any error. Only then continue.

1. Login and enable shell access
   1. enable
   2. conf t
   3. license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
   4. fae cable-stamping-unlock 40g_lr4
   5. conf write
   6. (wait a little?)
   7. _shell
2. Remove U-Boot password so you can always boot/flash your way back into the switch
   1. /opt/tms/bin/mddbreq /config/db/initial set modify - /system/bootmgr/password string ''
   2. eetool -a bf -s UBPASSWD=""
3. Dump EEPROMs
   1. /opt/tms/bin/mellaggra _read_fru 1 0x51 1000 fru_backplate.bin
   2. /opt/tms/bin/mellaggra _read_fru 0 0x50 1000 fru_cpu.bin
   3. Copy them off via SFTP etc. ZIP and please send them to me. Needed to finally clear remaining SX6012 vs true SX1012 differences...
4. Make the switch ethernet only
   1. system profile eth-single-switch
   2. This may completely erase your config, if so, repeat what is written in Step 1.
5. Try updating straight to
   https://www.mellanox.com/downloads/Software/image-PPC_M460EX-3.6.8012.img
   e.g. through the web GUI
6. I will send you another shell command to apply after that update via PM
7. Quieten down the fans (this is for 2-PSU model only probably - if you get whining, increase FAN_MIN)
   1. mount -nwo remount,rw /
   2. vi /etc/rc.local
      1. I hope you know how to use vi. If not, hit d d a couple of times to delete everything, then i to insert, :wq to save

   3. #!/bin/sh
      
      touch /var/lock/subsys/local
      
      # mount -nwo remount,rw /
      # vi /etc/rc.d/rc.local
      
      FAN_MIN="27"
      FAN_MAX="50"
      WAIT_MAX="10" # 5 minutes
      
      MDREQ1="/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string"
      MDREQ2="fan_number int8 1 fan_speed int8"
      MDREQ3="set_max uint8"
      
      i=1
      while :; do
      PID=$(pidof clusterd)
      if [ -n "$PID" ]; then
      sleep 60
      echo "Adjusting fan speed"
      $MDREQ1 "/MGMT/FAN1" $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
      break
      else
      sleep 30
      i=$((i+1))
      if [ $i -gt $WAIT_MAX ]; then
      echo "Timeout waiting for clusterd"
      break
      fi
      fi
      done
      
      exit 0

Dude...minor correction (but I love what you did!).

On step 7.2 it should be vi /etc/rc.d/rc.local not vi /etc/rc.local

 

[kapone]

Joined the club (Got two plain vanilla SX6036s). Replacing a Brocade iCX 6610 (ran out of 10g/40g ports). Updated to the latest version...applied licenses...played with fans/PWM till I was somewhat happy...and idle power consumption is...35w. (Final tweaks. Fan script sets them to 15% PWM!)

Holy shit. This is a 36x40gb port switch!

I'm in awe.

 

[Stephan]

idle power consumption is...35w

Remember the fans are also cooling transceivers. Issue if you live in Arizona, in northern Sweden not so much. So in final habitat with a stress test running why not check transceiver temps. Could be as simple as turning off spanning tree, loop ports, and then inject a few broadcasts. Briefly also with external fan blowing at transceivers from the side, then at stock fan speeds. I'm a collector of datasheets and repair manuals and like to check maximum temperature ratings for low rate of surprises.

 

[klui]

On step 7.2 it should be vi /etc/rc.d/rc.local not vi /etc/rc.local

Doesn't matter.

[admin@sx6012c etc]# ls -ald rc*
lrwxrwxrwx  1 13 Apr 30  2018 rc.local -> rc.d/rc.local

 

[kapone]

Remember the fans are also cooling transceivers. Issue if you live in Arizona, in northern Sweden not so much. So in final habitat with a stress test running why not check transceiver temps. Could be as simple as turning off spanning tree, loop ports, and then inject a few broadcasts. Briefly also with external fan blowing at transceivers from the side, then at stock fan speeds. I'm a collector of datasheets and repair manuals and like to check maximum temperature ratings for low rate of surprises.

Understood, and I do intend to stress test it (will be hosting production loads) as this will become the core switch. (And I'm in the DC area, no Arizona level heat here... )

That said, I'm using all DAC cables, there's no transceivers involved (yet, and likely never in this house). The Mellanox will be paired with a Brocade 6450 for the 1G RJ45 ports and just two 10G-BaseT adapters, one for the WAN (Fios 2gb ONT has a 10G-T port), one for the AP (Fios CR1000B wifi unit in AP mode, also has a 10G-T for the LAN side)

Logically, the network is a few segments/VLANs and some custom/specific routing (especially on the AWS side/connect), not that bad. Now...rewriting those rules from the Brocade format to the Mellanox format...ugh.

 

[kapone]

Doesn't matter.

[admin@sx6012c etc]# ls -ald rc*
lrwxrwxrwx  1 13 Apr 30  2018 rc.local -> rc.d/rc.local

doh! Had forgotten about that, and I must have screwed up while writing the fan script. Because doing vi /etc/rc.local didn't seem to save the file, but that is likely user error.

 

[klui]

Maybe forgot to mount / rw?

/dev/mtdblock7    524288 491388     32900  94% /

/dev/mtdblock7 on / type jffs2 (ro,noatime)

I'm actually more curious if you did anything special to get your 6036's fans' PWM to 15%. I haven't unboxed my 6036 but my 6012s' fans will surge between PWM percentage of < 30% and maximum every 5-10 seconds.

 

[kapone]

Maybe forgot to mount / rw?

/dev/mtdblock7    524288 491388     32900  94% /

/dev/mtdblock7 on / type jffs2 (ro,noatime)

I'm actually more curious if you did anything special to get your 6036's fans' PWM to 15%. I haven't unboxed my 6036 but my 6012s' fans will surge between PWM percentage of < 30% and maximum every 5-10 seconds.

I don't know if it makes a difference or not...this is a brand new 6036, plain vanilla Mellanox.

That said, I started where everyone else did, at 27% and started going lower until the fans decided to not cooperate and surge. 15-16% seems right where they don't surge. Right now, I have a single PSU connected, but I might try the second one as well, to see what the power draw looks like.

Edit: At 15-16% the fans are running ~4200-4300rpm.

# mount -nwo remount,rw /
# vi /etc/rc.d/rc.local

FAN_MIN=“15”
FAN_MAX="50"
WAIT_MAX="10" # 5 minutes

MDREQ1="/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string"
MDREQ2="fan_number int8 1 fan_speed int8"
MDREQ3="set_max uint8"

i=1
while :; do
PID=$(pidof clusterd)
if [ -n "$PID" ]; then
sleep 60
echo "Adjusting fan speed"
$MDREQ1 “/FAN/FAN1" $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
$MDREQ1 “/FAN/FAN2” $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
$MDREQ1 “/FAN/FAN3” $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
$MDREQ1 “/FAN/FAN4” $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
$MDREQ1 “/PS1/FAN1" $MDREQ2 $FAN_MIN $MDREQ3 $FAN_MAX
break
else
sleep 30
i=$((i+1))
if [ $i -gt $WAIT_MAX ]; then
echo "Timeout waiting for clusterd"
break
fi
fi
done

exit 0