Beware of EMC switches sold as Mellanox SX6XXX on eBay
- Thread starter mpogr
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I know the mellanox ones do. The "T" means FDR10, and the "F" means FDR, but the EMC part numbers are just numbers. Do you know how I could look up my EMC part number to determine whether it's an FDR10 or FDR?
The spec sheet claims the 6005 will do FDR so I'm guessing it's a configuration issue.
http://www.mellanox.com/related-docs/prod_ib_switch_systems/PB_SX6005.pdf
Yes. There are two types of SX6005, one with a "T" in the part name and one with an "F". The former is FDR10 and the latter is FDR. However, the EMC switches just have "SX6005" and then a purely numerical part number on the pull out tab.
Basically, I'm trying to figure out if the switch is hardware limited to FDR10 or if there's some sort of firmware/setting incompatibility.
I tried forcing the ports with ibportstate <lid> <port> espeed 1 , but it didn't work. They autonegotiate fine to FDR when connected back to back.
Ah, gotcha. I'm seeing all sorts of model numbers listed on ebay, some specifying MSX6005F-2BFS, some MSX6005-1FS, some MSX6005-1BFS. I'm not sure where they're getting the model numbers. Can you dump product info from the switch itself? Or maybe from the bootloader?
I don't have a switch in hand yet, I'm still waiting on mine to be delivered, sorry I can't be more help.
No idea how to do that unfortunately. The only command I know for looking at switch info is "ibswitches". Know any others?
Sorry, I don't have a switch in hand to test with yet. The models listed on ebay are weird, they leave off the T or F so it's hard to tell which version you're getting. The only thing I see on the spec sheet is that there's no T version of the -2BFS.
Good call. Mine has two psus, so there's a better chance it's an FDR switch than FDR10, but EMC could have some special configuration, so who knows.
I'm going to install MFT and try some of those tools to figure out what's going on.
Ok, made a little progress on this. I installed MFT. Then I did "ibswitches" to find the LID of the switch. I then did "flint -d lid-<switch lid> query full". This returned the firmware version (9.3.2000, so not latest), as wel as the PSID, which was "EMC1260110026". Now, if you look at the mellanox firmware download page, you can see the PSID of the various switches when you click on them. EMC apparently keeps the same numbers, but changes the prefix to "EMC". This switch's PSID corresponds to the MSX6005T, which is FDR10, not FDR.
So more buyer beware: Some EMC switches are advertised as 56 Gbit/s FDR, but are actually FDR10!
I'm going to try to figure out how to burn the stock mellanox FDR firmware to it. Anyone every done this with switches? I've done it with HCA's before.
AHA! It worked! I have FDR speeds now!
You can burn mellanox .bin firmware directly to EMC SX6005 switches almost exactly like you do with rebranded HCAs. You need the LID of the switch from "ibswitches", and you need to install MFT. Then the command is:
where X is the lid number of your switch. I downloaded the FDR MSX6005F firmware from mellanox's site and successfully burned it to my FDR10 EMC SX6005 switch this way.
You should probably follow instructions like these (look for izx's post) and backup the firmware image and hw info before burning, though.
Hope this helps someone in the future.
You can burn mellanox .bin firmware directly to EMC SX6005 switches almost exactly like you do with rebranded HCAs. You need the LID of the switch from "ibswitches", and you need to install MFT. Then the command is:
Code:
flint -d lid-X -i fw.bin -allow_psid_change burnYou should probably follow instructions like these (look for izx's post) and backup the firmware image and hw info before burning, though.
Hope this helps someone in the future.
Last edited:
All EMC switches that were part of a dual fabric (A/B) are all MSX6012F-2BRS with their custom 'dumb' mlxos on it.
Last edited:
Successfully upgraded my MSX6012! Wanted to give a shout out and thanks to mpogr for putting together the guide!
I have 2 questions though:
1) I'm trying to check performance of the setup. I have 2 R610's with ConnectX-3's flashed with the latest Mellanox firmware. I'm running them as 40Gbe, 9000 MTU (on cards and switch). However, when running iperf3, it's showing a max of 14Gbps, even with parallel instances. Anything I should look at here?
2) My management interface goes down over time, and won't come back automatically. The lights on the management nic on the switch go off as well. If I unplug the network patch cable to the management nic and plug it back in, it'll come back online. It will still go offline again after a day or so. Has anyone seen this?
I have 2 questions though:
1) I'm trying to check performance of the setup. I have 2 R610's with ConnectX-3's flashed with the latest Mellanox firmware. I'm running them as 40Gbe, 9000 MTU (on cards and switch). However, when running iperf3, it's showing a max of 14Gbps, even with parallel instances. Anything I should look at here?
2) My management interface goes down over time, and won't come back automatically. The lights on the management nic on the switch go off as well. If I unplug the network patch cable to the management nic and plug it back in, it'll come back online. It will still go offline again after a day or so. Has anyone seen this?
Try 1500 MTU, for some reason I never was able to break about ~13-14Gb/s with jumbo frames. I have no idea why, this was on a point to point link directly between two machines.
That is the max speed you will get on PCIe 2.0 (if you have an x56xx series Xeon) which the r610 is. the Rx20's and newer do PCIe 3.0 (8GT/s per lane).
After overhead, your speed should be sitting ~12.7 gigabytes/s (jumbo frames or not wont make a difference)
Also, make sure the card is not using the riser that the internal storage perc card is using( which would put the card above the idrac module). that physical x8 runs as an x4. The physical x8 on the riser closest to the power supplies, is the slot that actually runs x8
If you have a x16 slot available and $200 burning a hole in your pocket you can go this route:
https://forums.servethehome.com/index.php?threads/100gbe-mellanox-180-usd.21079/
I can hit 39Gbps on an old PCIe2 system. A bit of work is required to deal with the heat pipe, but it works. Two x8 cards would be cheaper, though. You probably aren't going to hit 40Gbps single stream anyway so LAGing two cards should be a reasonable equivalent.
I'm having a couple of issues making the jffs image.
In /bin of the mellanox firmware the busybox file is a symlink, so I copied the binary from /sbin instead. However when I flash the image and boot it, it can't find the inittab.
Has anyone else had this issue?
In /bin of the mellanox firmware the busybox file is a symlink, so I copied the binary from /sbin instead. However when I flash the image and boot it, it can't find the inittab.
Has anyone else had this issue?
Attachments
Use the file within the following zip
http://www.jslenterprises.net/mlnxbase.zip
save you the trouble of building the jffs2 image.
Ok, I started completely over, substituting in that jffs2 image, and it seems to be doing the exact same thing:
And it will just reboot and go into a loop
For reference here is my printenv output
Code:
=> run flash_jffs2
Kernel panic - not syncing: No init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
CPU: 0 PID: 1 Comm: swapper Not tainted 3.10.27-MELLANOXuni-m460ex PPC_M460EX
Call Trace(ef840000): name=swapper, state=0
[ef845ea0] [c0005614] show_stack+0x54/0x150 (unreliable)
[ef845ee0] [c02758d8] panic+0xcc/0x204
[ef845f30] [c0001b90] kernel_init+0xe8/0xf8
[ef845f40] [c000afcc] ret_from_kernel_thread+0x5c/0x64
Rebooting in 180 seconds..And it will just reboot and go into a loop
For reference here is my printenv output
Code:
=> printenv
bootcmd=run flash_jffs2
bootdelay=5
baudrate=9600
loads_echo=
hostname=mlnx460ex
netdev=eth0
nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath}
ramargs=setenv bootargs root=/dev/ram rw
addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1
addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate}
addmisc=setenv bootargs ${bootargs}
initrd_high=30000000
kernel_addr_r=400000
fdt_addr_r=800000
ramdisk_addr_r=C00000
hostname=mlnx460ex
bootfile=mlnx460ex/uImage
ramdisk_file=mlnx460ex/uRamdisk
rootpath=/opt/eldk/ppc_4xxFP
flash_self=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr}
flash_nfs=run nfsargs addip addtty addmisc;bootm ${kernel_addr} - ${fdt_addr}
net_nfs=tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} - ${fdt_addr_r}
net_self_load=tftp ${kernel_addr_r} ${bootfile};tftp ${fdt_addr_r} ${fdt_file};tftp ${ramdisk_addr_r} ${ramdisk_file};
net_self=run net_self_load;run ramargs addip addtty addmisc;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
fdt_file=mlnx460ex/mlnx460ex.dtb
flash_self_old=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr}
flash_nfs_old=run nfsargs addip addtty addmisc;bootm ${kernel_addr}
net_nfs_old=tftp ${kernel_addr_r} ${bootfile};run nfsargs addip addtty addmisc;bootm ${kernel_addr_r}
load=tftp 200000 mlnx460ex/u-boot.bin
update=protect off 0xFFFA0000 FFFFFFFF;era 0xFFFA0000 FFFFFFFF;cp.b ${fileaddr} 0xFFFA0000 ${filesize};setenv filesize;saveenv
upd=run load update
dhcp_vendor-class-identifier=bootmfg:hwname:mlnx460ex:
reset_button=0
clear_filesize=setenv filesize
mfg_dir=mlnx460ex
mfg_args=setenv bootargs root=/dev/ram rw ramdisk_size=${mfg_ramdisk_size} ${mfg_extra_args}
mfg_common_args=run addtty addmisc
mfg_load=tftp ${kernel_addr_r} ${mfg_root}${mfg_dir}/${mfg_kernel_file};tftp ${fdt_addr_r} ${mfg_root}${mfg_dir}/${mfg_fdt_file};tftp ${ramdisk_addr_r} ${mfg_root}${mfg_dir}/${mfg_ramdisk_file}
mfg_nodhcp=echo "Manufacture will TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
mfg=echo "Manufacture will DHCP, TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; dhcp; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
menu_file=menu.img
menu_load=tftp ${menu_addr_r} ${mfg_root}${mfg_dir}/${menu_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
menu_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
menu_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
menu_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run menu_usb_load_ext2 ; else ; run menu_usb_load_fat ; fi
menu_usb=run clear_filesize ; run menu_usb_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
menu_nodhcp=run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
menu=dhcp ; run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
fw_file=u-boot.bin
fw_load=tftp ${fw_addr_r} ${mfg_root}${mfg_dir}/${fw_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
fw_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
fw_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
fw_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run fw_usb_load_ext2 ; else ; run fw_usb_load_fat ; fi
fw_update_raw=protect off 0xFFFA0000 FFFFFFFF;erase 0xFFFA0000 FFFFFFFF;cp.b ${fw_addr_r} 0xFFFA0000 ${filesize};cmp.b ${fw_addr_r} 0xFFFA0000 ${filesize};setenv filesize; saveenv
fw_usb_update=run clear_filesize ; run fw_usb_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
fw_update_nodhcp=run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
fw_update=dhcp ; run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
boot_common_args=run addtty addmisc
mfg_usb_dir=mlnx460ex
mfg_usb_load_ext2=usb start; echo "Loading ${mfg_kernel_file}";ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
mfg_usb_load_fat=usb start; echo "Loading ${mfg_kernel_file}";fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
mfg_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run mfg_usb_load_ext2 ; else ; run mfg_usb_load_fat ; fi
mfg_usb=echo "Manufacture will load from USB directory ${mfg_usb_root}${mfg_usb_dir}, and boot"; echo; run clear_filesize ; run mfg_usb_load; if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
kernel_addr=ff000000
fdt_addr=ff1e0000
ramdisk_addr=ff200000
fw_addr_r=400000
menu_addr_r=B00000
pciconfighost=1
pcie_mode=RP:RP
autoload=no
rootdev=/dev/mtdblock6
boot_usb_ext2_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
boot_usb_ext2_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
boot_usb_fat_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
boot_usb_fat_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
mfg_kernel_file=vmlinuz
mfg_ramdisk_file=rootfs
mfg_ramdisk_size=180224
mfg_fdt_file=fdt
mfg_usb_dev=0
mfg_usb_part=1
mfg_usb_fstype=fat
mfg_usb_root=/
boot_kernel_file=vmlinuz
boot_fdt_file=fdt
boot_usb_dev=0
boot_usb_part_loc_1=2
boot_usb_part_loc_2=3
boot_usb_root_loc_1=/dev/sda5
boot_usb_root_loc_2=/dev/sda6
usb_args_loc_1=setenv bootargs root=${boot_usb_root_loc_1} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
usb_args_loc_2=setenv bootargs root=${boot_usb_root_loc_2} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
flash_jffs2=run jffs2_args boot_common_args;bootm ${kernel_addr} - ${fdt_addr}
autostart=off
autoload=off
ethact=ppc_4xx_eth0
filesize=240000
fileaddr=400000
netmask=255.0.0.0
ipaddr=10.0.0.3
serverip=10.0.1.2
image_kernel_args=loglevel=2
jffs2_args=setenv bootargs root=/dev/mtdblock6 rootfstype=jffs2 rw reset_button=0 loglevel=2
Last edited: