Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    Not at all. You can add multipathing the same way as for iSCSI. Have a look at VMware documentation for iSER and you'll see all these details there.
     
    #261
  2. efschu2

    efschu2 Member

    Joined:
    Feb 14, 2019
    Messages:
    68
    Likes Received:
    9
    Well this would be the configuration on an active/active cluster, but unlike in my planed active/passive configuration the failover host is not representing any iSCSI/iSER LUNs while primary host is alive, so ESXi is not using multipath to two individual targets on two physical boxes, multipath is only be used in case of switch/nic death (but still connected to the same physical box, only using the failover network path (second nic/second switch)). In my iSCSI setup in case of primary failure the secondary host would import the ZFS pools, start the SCST target and then taking over the floating cluster IP. Am I thinking to much/wrong here? :confused:

    For an active/active solution I would have to take DRDB, but because of performance reason I dislike that. An active/passive solution with a "re-uptime" of a couple of seconds is enough in my case.

    Some guy wrote a guide with a very similar setup. But instead of iSCSI and a ZFS on dual-ported SAS disks I want to use iSER and a ZFS on dual-ported NVMe disks.
    https://bm-stor.com/index.php/blog/Linux-cluster-with-ZFS-on-Cluster-in-a-Box/
     
    #262
    Last edited: Feb 22, 2019
  3. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    today chat with one friend, he bricked his sx6012 after execute:
    eeprom read 0x53 0x8500000 0 1958000
    in uboot ,the switch report i2c error and freeze. he just unplug power cable
    after that the switch just dead can’t boot up,is there any chance rescue his sx6012?
     
    #263
    Last edited: Feb 22, 2019
  4. Squeakz

    Squeakz New Member

    Joined:
    Apr 12, 2018
    Messages:
    2
    Likes Received:
    0
    I recently picked up a SX6018. Will see if I can get it up and running. :)
     
    #264
  5. Labs

    Labs New Member

    Joined:
    Mar 21, 2019
    Messages:
    25
    Likes Received:
    4
    Hello,

    I am looking to buy a Mellanox SX6012 but have some questions after reading the forum and other documentation on the net.

    There are 2 models for SX6012 the "T" version which is FDR10 and "F" version which is FDR.
    Q1: Is there any hardware difference between them? Or is just software/license?

    Now about OEM ones:
    EMC models:
    100-886-236-00
    100-886-236-04

    Q2: What's the difference between them? One is FDR10 and one is FDR?

    HP Models:
    They don't have the SX6012 it seems, only the 36 ports or 216 ports models.

    Q3: Is there any HP SX6012? If yes what is the HP part number?

    IBM and Lenovo Models:
    The same like HP they don't seem to have any SX6012 but they have the SX6036 models.

    Q4: Is there any IBM\Lenovo SX6012? If yes what is the IBM/Lenovo part number?

    Dell EMC Models:
    It seems on Mellanox site there is no mention about this SX6012 from EMC.
    All Dell EMC switches in the list seems to be the externally managed ones and there is no internally managed ones.

    Q5: Is this correct?

    U-Boot and firmware questions:
    I would to know the status of the U-Boot for standard and OEM models.
    It seems only the EMC model has the U-Boot without password. In rest all have the U-Boot locked.

    Q6: Can someone comment on this? Are there any other models from OEM or not with U-Boot unlocked (without password)?

    Firmware for Mellanox switches can be in 2 categories from what I read:
    A) externally managed firmware, usually very small image around 1.5MB or less with image file name for example "fw-sx-9_4_2000-Ax.bin"
    B) internally managed firmware, more exactly the Mellanox OS. These images are 200~500MB and have the file naming for example "image-PPC_M460EX-3.5.1006.img"

    Q7: Are the "image-PPC_M460EX-3.y.abcd.img" files interchangeable between all internally managed switches like 12/18/36 ports?
    Q8: Are the "image-PPC_M460EX-3.y.abcd.img" files customized for OEMs or the Mellanox and OEM files are identical?
    Q9: Can U-Boot be unlocked if it locked? Can it be flashed with an unlocked U-Boot image?

    And last part for now regarding access to Linux shell for internally managed switches.
    Q10: How can you drop from normal switch CLI to Linux shell? I saw people posting different commands run from Linux shell.

    I saw during booting output that switch is reporting 2 storage parts a FLASH: 16 MB and a NAND: 1024 MiB.
    I suppose the 16MB is for the U-Boot and the 1024MB is for the "image-PPC_M460EX-3.y.abcd.img" which should be the Mellanox OS.
    Q11: Can someone confirm this?

    Regards!
     
    #265
  6. efschu2

    efschu2 Member

    Joined:
    Feb 14, 2019
    Messages:
    68
    Likes Received:
    9
    Finaly my switches and cards arrived, but no cables :(

    How can I find out the switchlid?

    Code:
    mst status
    MST modules:
    ------------
        MST PCI module loaded
        MST PCI configuration module loaded
    
    MST devices:
    ------------
    /dev/mst/dev-i2c-0               - Embedded I2C master
    /dev/mst/dev-i2c-1               - Embedded I2C master
    /dev/mst/mt51000_pciconf0        - PCI configuration cycles access.
                                       domain:bus:dev.fn=0001:81:00.0 addr.reg=88 data.reg=92
                                       Chip revision is: 02
     
    #266
  7. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,752
    Likes Received:
    615
    #267
  8. efschu2

    efschu2 Member

    Joined:
    Feb 14, 2019
    Messages:
    68
    Likes Received:
    9
    So for local flash the divce name is mt51000_pciconf0 ?
     
    #268
  9. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,752
    Likes Received:
    615
    /dev/mst/mt51000_pciconf0


    edit: technically you are correct, the actual device name is mt51000_pciconf0 but what -d expects is the path to the device which is the above
     
    #269
    efschu2 likes this.
  10. efschu2

    efschu2 Member

    Joined:
    Feb 14, 2019
    Messages:
    68
    Likes Received:
    9
    Another success story here. Thanks to everyone sharing information!


    I had to change the default boot env for uboot, havent found that in the guide, for anyone looking for it:

    Code:
    setenv rootdev /dev/mtdblock7
    setenv bootcmd "run mlxlinux"
    saveenv 
     
    #270
    chancedoorway, Matt G and Labs like this.
  11. Tydus

    Tydus New Member

    Joined:
    Mar 18, 2019
    Messages:
    2
    Likes Received:
    1
    Hi everyone! I've read through this thread and learned much by pieces of wisdom from Mr. mpogr and all your replies. Cheers!

    I've PMed Mr. mpogr and got a conversion guide, but with some attachments missing. I contacted him again when I finally received the switch, and it seems he's not online these days...

    I cannot wait to convert it so can anyone kindly send me the missing pieces instead? Things I need are "/etc/image_layout.sh" "chad" and "hwd".

    Thanks in advance!
     
    #271
  12. groove

    groove Member

    Joined:
    Sep 21, 2011
    Messages:
    67
    Likes Received:
    18
    Hello everyone,

    In the middle of attempting to convert. Also looking for links to "image_layout.sh" , "chad", and "hwd".

    Would really appreciate it if someone can be kind enough to post the links.

    Thanks in advance.
     
    #272
  13. Tydus

    Tydus New Member

    Joined:
    Mar 18, 2019
    Messages:
    2
    Likes Received:
    1
    OK. Finally I found another way, without touching the shipped "chad" or "hwd". It is even more "hacking" so I'll not publish it anywere...

    Now working on "image_layout.sh". Seems it should be rather simple (< 10 lines or so), but more investigation should be made for sure.
     
    #273
    Freebsd1976 likes this.
  14. Labs

    Labs New Member

    Joined:
    Mar 21, 2019
    Messages:
    25
    Likes Received:
    4
    Hi,

    Is there any way from uboot to dump the 16MB and 1GB flash chips and upload to tftp? I would like to make a backup of the 2 flash chips before attempting to erase/write other things.
     
    #274
  15. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    Do it carefully my friend bricked his 6012
     
    #275
  16. SGS

    SGS New Member

    Joined:
    May 24, 2017
    Messages:
    7
    Likes Received:
    1
    Sorry, I don't have a solution but I'm interested to know why he did that? There are no devices on slave address 0x53, unless I'm mistaken.

    If anyone is tempted to play with the i2c commands in uboot I'd strongly advise against it. I bricked mine by reading too but managed to bring it back to life with a Diolan DLN-2-U2C via one of the internal i2c headers. I think i2c support in that uboot build is broken.

    Edit: for reference, this is the command that bricked mine:

    imd 0x52 0.2 0xFFF

    That is supposed to read but actually wrote a zero to the first byte of the cpu bootstrap code in the boot rom
     
    #276
    Last edited: Apr 10, 2019
  17. groove

    groove Member

    Joined:
    Sep 21, 2011
    Messages:
    67
    Likes Received:
    18
    Hello everyone,

    Reading through the guide and am looking at the contents of MLNX-OS image (file name image-PPC_M460EX-ppc-m460ex-20160609-202426.tgz) and I can't seem to find an 'image_layout.sh' in that image. I could locate the other file chad, hwd and ibd in the image that should the guide states should be replaced.

    If I attempt the conversion, do I just copy image_layout.sh to /mnt/root2/etc even though it does not exist?

    Please advise if it's ok for the file to no exist in the original image or if I got a wrong image.

    Thanks
     
    #277
    Last edited: Apr 10, 2019
  18. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    Yes
     
    #278
  19. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    Yikes, that's really good to know about before starting on a conversion.
     
    #279
  20. groove

    groove Member

    Joined:
    Sep 21, 2011
    Messages:
    67
    Likes Received:
    18

    Thanks for the confirmation, mpogr.
     
    #280

Share This Page