Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,777
    Likes Received:
    444
    There are different versions of that switch, some support qdr+fdr10 only and the high end ones do fdr and all lower speeds.
     
    #141
  2. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    59
    Likes Received:
    11
    I know the mellanox ones do. The "T" means FDR10, and the "F" means FDR, but the EMC part numbers are just numbers. Do you know how I could look up my EMC part number to determine whether it's an FDR10 or FDR?
     
    #142
  3. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    645
    Likes Received:
    221
    The spec sheet claims the 6005 will do FDR so I'm guessing it's a configuration issue.

    http://www.mellanox.com/related-docs/prod_ib_switch_systems/PB_SX6005.pdf
     
    #143
  4. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    59
    Likes Received:
    11
    Yes. There are two types of SX6005, one with a "T" in the part name and one with an "F". The former is FDR10 and the latter is FDR. However, the EMC switches just have "SX6005" and then a purely numerical part number on the pull out tab.

    Basically, I'm trying to figure out if the switch is hardware limited to FDR10 or if there's some sort of firmware/setting incompatibility.

    I tried forcing the ports with ibportstate <lid> <port> espeed 1 , but it didn't work. They autonegotiate fine to FDR when connected back to back.
     
    #144
  5. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    645
    Likes Received:
    221
    Ah, gotcha. I'm seeing all sorts of model numbers listed on ebay, some specifying MSX6005F-2BFS, some MSX6005-1FS, some MSX6005-1BFS. I'm not sure where they're getting the model numbers. Can you dump product info from the switch itself? Or maybe from the bootloader?

    I don't have a switch in hand yet, I'm still waiting on mine to be delivered, sorry I can't be more help.
     
    #145
  6. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    59
    Likes Received:
    11
    No idea how to do that unfortunately. The only command I know for looking at switch info is "ibswitches". Know any others?
     
    #146
  7. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    645
    Likes Received:
    221
    Sorry, I don't have a switch in hand to test with yet. The models listed on ebay are weird, they leave off the T or F so it's hard to tell which version you're getting. The only thing I see on the spec sheet is that there's no T version of the -2BFS.
     
    #147
  8. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    59
    Likes Received:
    11
    Good call. Mine has two psus, so there's a better chance it's an FDR switch than FDR10, but EMC could have some special configuration, so who knows.

    I'm going to install MFT and try some of those tools to figure out what's going on.
     
    #148
  9. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    59
    Likes Received:
    11
    Ok, made a little progress on this. I installed MFT. Then I did "ibswitches" to find the LID of the switch. I then did "flint -d lid-<switch lid> query full". This returned the firmware version (9.3.2000, so not latest), as wel as the PSID, which was "EMC1260110026". Now, if you look at the mellanox firmware download page, you can see the PSID of the various switches when you click on them. EMC apparently keeps the same numbers, but changes the prefix to "EMC". This switch's PSID corresponds to the MSX6005T, which is FDR10, not FDR.

    So more buyer beware: Some EMC switches are advertised as 56 Gbit/s FDR, but are actually FDR10!

    I'm going to try to figure out how to burn the stock mellanox FDR firmware to it. Anyone every done this with switches? I've done it with HCA's before.
     
    #149
  10. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    59
    Likes Received:
    11
    AHA! It worked! I have FDR speeds now!

    You can burn mellanox .bin firmware directly to EMC SX6005 switches almost exactly like you do with rebranded HCAs. You need the LID of the switch from "ibswitches", and you need to install MFT. Then the command is:
    Code:
    flint -d lid-X -i fw.bin -allow_psid_change burn
    where X is the lid number of your switch. I downloaded the FDR MSX6005F firmware from mellanox's site and successfully burned it to my FDR10 EMC SX6005 switch this way.

    You should probably follow instructions like these (look for izx's post) and backup the firmware image and hw info before burning, though.

    Hope this helps someone in the future.
     
    #150
    Last edited: Sep 13, 2018
    Tha_14, MikeWebb and arglebargle like this.
  11. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    All EMC switches that were part of a dual fabric (A/B) are all MSX6012F-2BRS with their custom 'dumb' mlxos on it.
     
    #151
    Last edited: Sep 15, 2018
    arglebargle likes this.
  12. autodestruct

    autodestruct New Member

    Joined:
    Aug 14, 2018
    Messages:
    1
    Likes Received:
    0
    Successfully upgraded my MSX6012! Wanted to give a shout out and thanks to mpogr for putting together the guide!

    I have 2 questions though:

    1) I'm trying to check performance of the setup. I have 2 R610's with ConnectX-3's flashed with the latest Mellanox firmware. I'm running them as 40Gbe, 9000 MTU (on cards and switch). However, when running iperf3, it's showing a max of 14Gbps, even with parallel instances. Anything I should look at here?

    2) My management interface goes down over time, and won't come back automatically. The lights on the management nic on the switch go off as well. If I unplug the network patch cable to the management nic and plug it back in, it'll come back online. It will still go offline again after a day or so. Has anyone seen this?
     
    #152
  13. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    645
    Likes Received:
    221
    Try 1500 MTU, for some reason I never was able to break about ~13-14Gb/s with jumbo frames. I have no idea why, this was on a point to point link directly between two machines.
     
    #153
  14. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    That is the max speed you will get on PCIe 2.0 (if you have an x56xx series Xeon) which the r610 is. the Rx20's and newer do PCIe 3.0 (8GT/s per lane).

    After overhead, your speed should be sitting ~12.7 gigabytes/s (jumbo frames or not wont make a difference)

    Also, make sure the card is not using the riser that the internal storage perc card is using( which would put the card above the idrac module). that physical x8 runs as an x4. The physical x8 on the riser closest to the power supplies, is the slot that actually runs x8
     
    #154
    punkerman likes this.
  15. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    69
    Likes Received:
    43
    If you have a x16 slot available and $200 burning a hole in your pocket you can go this route:

    https://forums.servethehome.com/index.php?threads/100gbe-mellanox-180-usd.21079/

    I can hit 39Gbps on an old PCIe2 system. A bit of work is required to deal with the heat pipe, but it works. Two x8 cards would be cheaper, though. You probably aren't going to hit 40Gbps single stream anyway so LAGing two cards should be a reasonable equivalent.
     
    #155
  16. Skuzzi

    Skuzzi New Member

    Joined:
    Sep 20, 2018
    Messages:
    5
    Likes Received:
    1
    I'm having a couple of issues making the jffs image.

    In /bin of the mellanox firmware the busybox file is a symlink, so I copied the binary from /sbin instead. However when I flash the image and boot it, it can't find the inittab.

    Has anyone else had this issue?
     

    Attached Files:

    #156
  17. Skuzzi

    Skuzzi New Member

    Joined:
    Sep 20, 2018
    Messages:
    5
    Likes Received:
    1
    Realised my init was from the mellanox OS and not symlinked to busybox. Chanegd that and now I get a kernel panic followed by a reboot.
     

    Attached Files:

    #157
  18. JTF195

    JTF195 New Member

    Joined:
    Nov 15, 2017
    Messages:
    9
    Likes Received:
    6

    I'm having this exact same issue.
     
    #158
  19. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    Use the file within the following zip
    http://www.jslenterprises.net/mlnxbase.zip

    save you the trouble of building the jffs2 image.
     
    #159
    BeTeP, punkerman, Labs and 1 other person like this.
  20. JTF195

    JTF195 New Member

    Joined:
    Nov 15, 2017
    Messages:
    9
    Likes Received:
    6
    Ok, I started completely over, substituting in that jffs2 image, and it seems to be doing the exact same thing:
    Code:
    => run flash_jffs2
    Kernel panic - not syncing: No init found.  Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
    CPU: 0 PID: 1 Comm: swapper Not tainted 3.10.27-MELLANOXuni-m460ex PPC_M460EX
    Call Trace(ef840000): name=swapper, state=0
    [ef845ea0] [c0005614] show_stack+0x54/0x150 (unreliable)
    [ef845ee0] [c02758d8] panic+0xcc/0x204
    [ef845f30] [c0001b90] kernel_init+0xe8/0xf8
    [ef845f40] [c000afcc] ret_from_kernel_thread+0x5c/0x64
    Rebooting in 180 seconds..

    And it will just reboot and go into a loop

    For reference here is my printenv output
    Code:
    => printenv
    bootcmd=run flash_jffs2
    bootdelay=5
    baudrate=9600
    loads_echo=
    hostname=mlnx460ex
    netdev=eth0
    nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath}
    ramargs=setenv bootargs root=/dev/ram rw
    addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1
    addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate}
    addmisc=setenv bootargs ${bootargs}
    initrd_high=30000000
    kernel_addr_r=400000
    fdt_addr_r=800000
    ramdisk_addr_r=C00000
    hostname=mlnx460ex
    bootfile=mlnx460ex/uImage
    ramdisk_file=mlnx460ex/uRamdisk
    rootpath=/opt/eldk/ppc_4xxFP
    flash_self=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr}
    flash_nfs=run nfsargs addip addtty addmisc;bootm ${kernel_addr} - ${fdt_addr}
    net_nfs=tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} - ${fdt_addr_r}
    net_self_load=tftp ${kernel_addr_r} ${bootfile};tftp ${fdt_addr_r} ${fdt_file};tftp ${ramdisk_addr_r} ${ramdisk_file};
    net_self=run net_self_load;run ramargs addip addtty addmisc;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
    fdt_file=mlnx460ex/mlnx460ex.dtb
    flash_self_old=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr}
    flash_nfs_old=run nfsargs addip addtty addmisc;bootm ${kernel_addr}
    net_nfs_old=tftp ${kernel_addr_r} ${bootfile};run nfsargs addip addtty addmisc;bootm ${kernel_addr_r}
    load=tftp 200000 mlnx460ex/u-boot.bin
    update=protect off 0xFFFA0000 FFFFFFFF;era 0xFFFA0000 FFFFFFFF;cp.b ${fileaddr} 0xFFFA0000 ${filesize};setenv filesize;saveenv
    upd=run load update
    dhcp_vendor-class-identifier=bootmfg:hwname:mlnx460ex:
    reset_button=0
    clear_filesize=setenv filesize
    mfg_dir=mlnx460ex
    mfg_args=setenv bootargs root=/dev/ram rw ramdisk_size=${mfg_ramdisk_size} ${mfg_extra_args}
    mfg_common_args=run addtty addmisc
    mfg_load=tftp ${kernel_addr_r} ${mfg_root}${mfg_dir}/${mfg_kernel_file};tftp ${fdt_addr_r} ${mfg_root}${mfg_dir}/${mfg_fdt_file};tftp ${ramdisk_addr_r} ${mfg_root}${mfg_dir}/${mfg_ramdisk_file}
    mfg_nodhcp=echo "Manufacture will TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
    mfg=echo "Manufacture will DHCP, TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; dhcp; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
    menu_file=menu.img
    menu_load=tftp ${menu_addr_r} ${mfg_root}${mfg_dir}/${menu_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
    menu_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
    menu_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
    menu_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run menu_usb_load_ext2 ; else ; run menu_usb_load_fat ; fi
    menu_usb=run clear_filesize ; run menu_usb_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
    menu_nodhcp=run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
    menu=dhcp ; run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
    fw_file=u-boot.bin
    fw_load=tftp ${fw_addr_r} ${mfg_root}${mfg_dir}/${fw_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
    fw_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
    fw_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
    fw_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run fw_usb_load_ext2 ; else ; run fw_usb_load_fat ; fi
    fw_update_raw=protect off 0xFFFA0000 FFFFFFFF;erase 0xFFFA0000 FFFFFFFF;cp.b ${fw_addr_r} 0xFFFA0000 ${filesize};cmp.b ${fw_addr_r} 0xFFFA0000 ${filesize};setenv filesize; saveenv
    fw_usb_update=run clear_filesize ; run fw_usb_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
    fw_update_nodhcp=run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
    fw_update=dhcp ; run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
    boot_common_args=run addtty addmisc
    mfg_usb_dir=mlnx460ex
    mfg_usb_load_ext2=usb start; echo "Loading ${mfg_kernel_file}";ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
    mfg_usb_load_fat=usb start; echo "Loading ${mfg_kernel_file}";fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
    mfg_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run mfg_usb_load_ext2 ; else ; run mfg_usb_load_fat ; fi
    mfg_usb=echo "Manufacture will load from USB directory ${mfg_usb_root}${mfg_usb_dir}, and boot"; echo; run clear_filesize ; run mfg_usb_load; if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
    kernel_addr=ff000000
    fdt_addr=ff1e0000
    ramdisk_addr=ff200000
    fw_addr_r=400000
    menu_addr_r=B00000
    pciconfighost=1
    pcie_mode=RP:RP
    autoload=no
    rootdev=/dev/mtdblock6
    boot_usb_ext2_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    boot_usb_ext2_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    boot_usb_fat_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    boot_usb_fat_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    mfg_kernel_file=vmlinuz
    mfg_ramdisk_file=rootfs
    mfg_ramdisk_size=180224
    mfg_fdt_file=fdt
    mfg_usb_dev=0
    mfg_usb_part=1
    mfg_usb_fstype=fat
    mfg_usb_root=/
    boot_kernel_file=vmlinuz
    boot_fdt_file=fdt
    boot_usb_dev=0
    boot_usb_part_loc_1=2
    boot_usb_part_loc_2=3
    boot_usb_root_loc_1=/dev/sda5
    boot_usb_root_loc_2=/dev/sda6
    usb_args_loc_1=setenv bootargs root=${boot_usb_root_loc_1} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
    usb_args_loc_2=setenv bootargs root=${boot_usb_root_loc_2} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
    flash_jffs2=run jffs2_args boot_common_args;bootm ${kernel_addr} - ${fdt_addr}
    autostart=off
    autoload=off
    ethact=ppc_4xx_eth0
    filesize=240000
    fileaddr=400000
    netmask=255.0.0.0
    ipaddr=10.0.0.3
    serverip=10.0.1.2
    image_kernel_args=loglevel=2
    jffs2_args=setenv bootargs root=/dev/mtdblock6 rootfstype=jffs2 rw reset_button=0 loglevel=2
     
    #160
    Last edited: Oct 7, 2018
Similar Threads: Beware switches
Forum Title Date
Networking Beware - Emulex 10GbE Virtual Fabric Adapter II x8 PCI-E slot Jun 22, 2014
Networking Netgear M4500 100G switches Feb 18, 2020
Networking Accton/Edgecore/White Box vs Arista & other branded switches? Nov 23, 2019
Networking Multiple Subnets on Multiple Switches, No VLAN Sep 22, 2019
Networking [EU] Looking for cheap SFP+ switches and QSFP+ backbone switch Jun 15, 2019

Share This Page