Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    I'd grab one of the cx-3 dual port that go for about 28~35 on ebay. Cross flash it and put it on your windows box. Run opensm on the same windows box and flash it through that.
     
    #121
  2. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    Well, I literally just cleared everything. I accidentally ran envreset in uboot.

    can someone run a "printenv" within uboot and send me a copy so I can rebuild it the arguments.
    If anyone is wondering, below is the virgin uboot env without any config for the various oem's/mellanox

    Code:
    bootcmd=run flash_jffs2
    bootdelay=5
    baudrate=9600
    loads_echo=
    autoload=n
    hostname=mlnx460ex
    netdev=eth0
    nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath}
    ramargs=setenv bootargs root=/dev/ram rw
    addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:eek:ff panic=1
    addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate}
    addmisc=setenv bootargs ${bootargs}
    initrd_high=30000000
    kernel_addr_r=400000
    fdt_addr_r=800000
    ramdisk_addr_r=C00000
    hostname=mlnx460ex
    bootfile=mlnx460ex/uImage
    ramdisk_file=mlnx460ex/uRamdisk
    rootpath=/opt/eldk/ppc_4xxFP
    flash_self=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr}
    flash_nfs=run nfsargs addip addtty addmisc;bootm ${kernel_addr} - ${fdt_addr}
    net_nfs=tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} - ${fdt_addr_r}
    net_self_load=tftp ${kernel_addr_r} ${bootfile};tftp ${fdt_addr_r} ${fdt_file};tftp ${ramdisk_addr_r} ${ramdisk_file};
    net_self=run net_self_load;run ramargs addip addtty addmisc;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
    fdt_file=mlnx460ex/mlnx460ex.dtb
    flash_self_old=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr}
    flash_nfs_old=run nfsargs addip addtty addmisc;bootm ${kernel_addr}
    net_nfs_old=tftp ${kernel_addr_r} ${bootfile};run nfsargs addip addtty addmisc;bootm ${kernel_addr_r}
    load=tftp 200000 mlnx460ex/u-boot.bin
    update=protect off 0xFFFA0000 FFFFFFFF;era 0xFFFA0000 FFFFFFFF;cp.b ${fileaddr} 0xFFFA0000 ${filesize};setenv filesize;saveenv
    upd=run load update
    dhcp_vendor-class-identifier=bootmfg:hwname:mlnx460ex:
    reset_button=0
    clear_filesize=setenv filesize
    mfg_dir=mlnx460ex
    mfg_args=setenv bootargs root=/dev/ram rw ramdisk_size=${mfg_ramdisk_size} ${mfg_extra_args}
    mfg_common_args=run addtty addmisc
    mfg_load=tftp ${kernel_addr_r} ${mfg_root}${mfg_dir}/${mfg_kernel_file};tftp ${fdt_addr_r} ${mfg_root}${mfg_dir}/${mfg_fdt_file};tftp ${ramdisk_addr_r} ${mfg_root}${mfg_dir}/${mfg_ramdisk_file}
    mfg_nodhcp=echo "Manufacture will TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
    mfg=echo "Manufacture will DHCP, TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; dhcp; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
    menu_file=menu.img
    menu_load=tftp ${menu_addr_r} ${mfg_root}${mfg_dir}/${menu_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
    menu_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
    menu_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
    menu_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run menu_usb_load_ext2 ; else ; run menu_usb_load_fat ; fi
    menu_usb=run clear_filesize ; run menu_usb_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
    menu_nodhcp=run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
    menu=dhcp ; run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
    fw_file=u-boot.bin
    fw_load=tftp ${fw_addr_r} ${mfg_root}${mfg_dir}/${fw_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
    fw_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
    fw_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
    fw_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run fw_usb_load_ext2 ; else ; run fw_usb_load_fat ; fi
    fw_update_raw=protect off 0xFFFA0000 FFFFFFFF;erase 0xFFFA0000 FFFFFFFF;cp.b ${fw_addr_r} 0xFFFA0000 ${filesize};cmp.b ${fw_addr_r} 0xFFFA0000 ${filesize};setenv filesize; saveenv
    fw_usb_update=run clear_filesize ; run fw_usb_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
    fw_update_nodhcp=run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
    fw_update=dhcp ; run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
    boot_common_args=run addtty addmisc
    mfg_usb_dir=mlnx460ex
    mfg_usb_load_ext2=usb start; echo "Loading ${mfg_kernel_file}";ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
    mfg_usb_load_fat=usb start; echo "Loading ${mfg_kernel_file}";fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
    mfg_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run mfg_usb_load_ext2 ; else ; run mfg_usb_load_fat ; fi
    mfg_usb=echo "Manufacture will load from USB directory ${mfg_usb_root}${mfg_usb_dir}, and boot"; echo; run clear_filesize ; run mfg_usb_load; if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
    kernel_addr=ff000000
    fdt_addr=ff1e0000
    ramdisk_addr=ff200000
    fw_addr_r=400000
    menu_addr_r=B00000
    pciconfighost=1
    pcie_mode=RP:RP
    autoload=no
    rootdev=/dev/mtdblock6
    boot_usb_ext2_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    boot_usb_ext2_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    boot_usb_fat_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    boot_usb_fat_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
    mfg_kernel_file=vmlinuz
    mfg_ramdisk_file=rootfs
    mfg_ramdisk_size=180224
    mfg_fdt_file=fdt
    mfg_usb_dev=0
    mfg_usb_part=1
    mfg_usb_fstype=fat
    mfg_usb_root=/
    boot_kernel_file=vmlinuz
    boot_fdt_file=fdt
    boot_usb_dev=0
    boot_usb_part_loc_1=2
    boot_usb_part_loc_2=3
    boot_usb_root_loc_1=/dev/sda5
    boot_usb_root_loc_2=/dev/sda6
    usb_args_loc_1=setenv bootargs root=${boot_usb_root_loc_1} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
    usb_args_loc_2=setenv bootargs root=${boot_usb_root_loc_2} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
    jffs2_args=setenv bootargs root=${rootdev} rootfstype=jffs2 ro reset_button=${reset_button} ${image_kernel_args} ${extra_args}
    flash_jffs2=run jffs2_args boot_common_args;bootm ${kernel_addr} - ${fdt_addr}
    Looks like anyone wanting to update u-boot, it will do it automatically for you when you call on the function - best way looks to be usb based (u-boot.bin can be found within the Mellanox OS)
     
    #122
    Last edited: Sep 4, 2018
  3. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    So, no one willing to share their u-boot env args? "printenv" is all you need to do, then copy paste
     
    #123
  4. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    I will if noone else has by thursday or friday, switch is at my other house (and not online yet).
     
    #124
  5. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    Sorry mate, I'll do when I reboot my switch. I don't do this very often...
     
    #125
  6. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    87
    Likes Received:
    19
    JSLEnterprises PM sent
    Hope it helps
     
    #126
    JSLEnterprises likes this.
  7. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    87
    Likes Received:
    19
    Thanks mpogr, another +1 here.

    Finally got around to finishing this thing off (read: found ear plugs). Went smooth, guide was good but I can see why it's not or the faint of heart or the one click easy fix crowed. I had a hex head scratch moment, but a re-read fixed it.

    That moment on (the long) first boot when it all kicked in..sudden, defining and peaceful. I might still get some new fans though, I think these are shot.

    Found out my ebay card purchase was a VPI but cable purchase was FDR10. HA! was wondering why they were cheap.
    Now I'm on the hunt for FDR14 DAC's or 56GB QSFP transceivers. I might just go buy from FS.

    Now to learn about IB and let my home HCI cluster really stretch their legs.
     
    #127
  8. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    Thanks MikeWebb, i got back the missing arg's for u-boot, without re-adding the extra crap that emc added. I get it to boot normally again

    now back to my initial problem to figure out why the initialization of the ports fail while starting up (first image).

    [​IMG]


    Edit: Lets see if anything changes when changing the profile type (ib, ib-single-switch, ib-no-adaptive-routing-single-switch, eth-single-switch, vpi-single-switch)

    Edit 2: Nope, no change. however, new options are available via cli

    Edit 3: Looks like its still a no go. I definitely need to do the firmware update, as the mlnx-os cant see the asic ('show asic-version' from cli shows nothing in the display)
     
    #128
    Last edited: Sep 5, 2018
  9. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    87
    Likes Received:
    19
    Anyone here successfully auto boot MLNX? I've tried using askenv to tell flash_jffs2 to run mlxlinux but it just gets stuck with ecc errors. Seems it cant find the root directory? I can manually boot after doing a setenv jffs2_args for root= etc.
     
    #129
  10. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,466
    Likes Received:
    218
    #130
  11. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    @MikeWebb
    enter this in uboot then saveenv
    Code:
    setenv rootdev /dev/mtdblock7
    that space in the command is an = for the variable
     
    #131
    MikeWebb likes this.
  12. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    Just a heads up on these, the vendor keeps bumping available supply when they're purchased down to 1 remaining. I pulled the trigger on one earlier today because he's the last ebay vendor with stock and I was concerned about affordable supply in the future and he bumped available stock by +5 right after.
     
    #132
    Last edited: Sep 12, 2018
  13. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    87
    Likes Received:
    19
    #133
  14. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    If you google MSX6012 you see the original side by side rack mount configuration. It looks like they were designed to rack up in pairs in a carrier, the EMC mounts look like quick and dirty hack jobs to make them fit some weirdly specific depth profile.
     
    #134
    Last edited: Sep 12, 2018
  15. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,710
    Likes Received:
    423
    There are different "adapters" available from mellanox; an adapter for asingle switch (like in the ebay listing) and the dual switch, side-by-side adapter.
     
    #135
  16. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,466
    Likes Received:
    218
    I'm just worried that if I get one I won't be able to find the guide or that I'll get the wrong one for the guide.
     
    #136
  17. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    The guide is available, no need to worry. Just PM me. As for the "right or wrong", just avoid buying the SX6XX5 models (which don't have a CPU) and you will be fine, as long as it's EMC. If it's Dell or HP, then it's even better, as you won't need to do the conversion.
     
    #137
  18. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    For Dell 6012F-2BRS's the part specifically to look for (that already have the gateway license by default) is
    00YY0K_0YC5N8
     
    #138
    MikeWebb likes this.
  19. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    87
    Likes Received:
    19
    HA! yes. saveenv was the awesome sauce. I can't believe I forgot that. Honestly I don't know how I make it out of the house in the mornings sometimes.
     
    #139
    Tha_14 likes this.
  20. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    59
    Likes Received:
    11
    I bought one of these EMC SX6005 switches. My IB cards and cables will show 56 Gbit/s link speed when connected back to back, but the link speed drops to FDR10 when connected through the switch. The switch is unmanaged, so I don't think I can check its port speeds. Since it's EMC, the part number doesn't have a "T" or "F" in it, so I'm not sure if it's the switch hardware or some sort of configuration/firmware problem limiting the speed. Any ideas?

    Thanks!
     
    #140

Share This Page