Beware of EMC switches sold as Mellanox SX6XXX on eBay

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

mpogr

Active Member
Jul 14, 2016
115
95
28
53
...unlike iSCSIs IP configuration on iSER RoCE v1 this could only be done by mac address "overwrite" (in my understanding)...
Not at all. You can add multipathing the same way as for iSCSI. Have a look at VMware documentation for iSER and you'll see all these details there.
 

efschu2

Member
Feb 14, 2019
68
12
8
Well this would be the configuration on an active/active cluster, but unlike in my planed active/passive configuration the failover host is not representing any iSCSI/iSER LUNs while primary host is alive, so ESXi is not using multipath to two individual targets on two physical boxes, multipath is only be used in case of switch/nic death (but still connected to the same physical box, only using the failover network path (second nic/second switch)). In my iSCSI setup in case of primary failure the secondary host would import the ZFS pools, start the SCST target and then taking over the floating cluster IP. Am I thinking to much/wrong here? :confused:

For an active/active solution I would have to take DRDB, but because of performance reason I dislike that. An active/passive solution with a "re-uptime" of a couple of seconds is enough in my case.

Some guy wrote a guide with a very similar setup. But instead of iSCSI and a ZFS on dual-ported SAS disks I want to use iSER and a ZFS on dual-ported NVMe disks.
https://bm-stor.com/index.php/blog/Linux-cluster-with-ZFS-on-Cluster-in-a-Box/
 
Last edited:

Freebsd1976

Active Member
Feb 23, 2018
387
73
28
today chat with one friend, he bricked his sx6012 after execute:
eeprom read 0x53 0x8500000 0 1958000
in uboot ,the switch report i2c error and freeze. he just unplug power cable
after that the switch just dead can’t boot up,is there any chance rescue his sx6012?
 
Last edited:

Labs

Member
Mar 21, 2019
88
16
8
Hello,

I am looking to buy a Mellanox SX6012 but have some questions after reading the forum and other documentation on the net.

There are 2 models for SX6012 the "T" version which is FDR10 and "F" version which is FDR.
Q1: Is there any hardware difference between them? Or is just software/license?

Now about OEM ones:
EMC models:
100-886-236-00
100-886-236-04

Q2: What's the difference between them? One is FDR10 and one is FDR?

HP Models:
They don't have the SX6012 it seems, only the 36 ports or 216 ports models.

Q3: Is there any HP SX6012? If yes what is the HP part number?

IBM and Lenovo Models:
The same like HP they don't seem to have any SX6012 but they have the SX6036 models.

Q4: Is there any IBM\Lenovo SX6012? If yes what is the IBM/Lenovo part number?

Dell EMC Models:
It seems on Mellanox site there is no mention about this SX6012 from EMC.
All Dell EMC switches in the list seems to be the externally managed ones and there is no internally managed ones.

Q5: Is this correct?

U-Boot and firmware questions:
I would to know the status of the U-Boot for standard and OEM models.
It seems only the EMC model has the U-Boot without password. In rest all have the U-Boot locked.

Q6: Can someone comment on this? Are there any other models from OEM or not with U-Boot unlocked (without password)?

Firmware for Mellanox switches can be in 2 categories from what I read:
A) externally managed firmware, usually very small image around 1.5MB or less with image file name for example "fw-sx-9_4_2000-Ax.bin"
B) internally managed firmware, more exactly the Mellanox OS. These images are 200~500MB and have the file naming for example "image-PPC_M460EX-3.5.1006.img"

Q7: Are the "image-PPC_M460EX-3.y.abcd.img" files interchangeable between all internally managed switches like 12/18/36 ports?
Q8: Are the "image-PPC_M460EX-3.y.abcd.img" files customized for OEMs or the Mellanox and OEM files are identical?
Q9: Can U-Boot be unlocked if it locked? Can it be flashed with an unlocked U-Boot image?

And last part for now regarding access to Linux shell for internally managed switches.
Q10: How can you drop from normal switch CLI to Linux shell? I saw people posting different commands run from Linux shell.

I saw during booting output that switch is reporting 2 storage parts a FLASH: 16 MB and a NAND: 1024 MiB.
I suppose the 16MB is for the U-Boot and the 1024MB is for the "image-PPC_M460EX-3.y.abcd.img" which should be the Mellanox OS.
Q11: Can someone confirm this?

Regards!
 

efschu2

Member
Feb 14, 2019
68
12
8
Finaly my switches and cards arrived, but no cables :(

How can I find out the switchlid?

Code:
mst status
MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/dev-i2c-0               - Embedded I2C master
/dev/mst/dev-i2c-1               - Embedded I2C master
/dev/mst/mt51000_pciconf0        - PCI configuration cycles access.
                                   domain:bus:dev.fn=0001:81:00.0 addr.reg=88 data.reg=92
                                   Chip revision is: 02
 
  • Like
Reactions: RedX1

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
/dev/mst/mt51000_pciconf0


edit: technically you are correct, the actual device name is mt51000_pciconf0 but what -d expects is the path to the device which is the above
 
  • Like
Reactions: efschu2

Tydus

New Member
Mar 18, 2019
2
1
3
Hi everyone! I've read through this thread and learned much by pieces of wisdom from Mr. mpogr and all your replies. Cheers!

I've PMed Mr. mpogr and got a conversion guide, but with some attachments missing. I contacted him again when I finally received the switch, and it seems he's not online these days...

I cannot wait to convert it so can anyone kindly send me the missing pieces instead? Things I need are "/etc/image_layout.sh" "chad" and "hwd".

Thanks in advance!
 

groove

Member
Sep 21, 2011
90
31
18
Hello everyone,

In the middle of attempting to convert. Also looking for links to "image_layout.sh" , "chad", and "hwd".

Would really appreciate it if someone can be kind enough to post the links.

Thanks in advance.
 

Tydus

New Member
Mar 18, 2019
2
1
3
OK. Finally I found another way, without touching the shipped "chad" or "hwd". It is even more "hacking" so I'll not publish it anywere...

Now working on "image_layout.sh". Seems it should be rather simple (< 10 lines or so), but more investigation should be made for sure.
 
  • Like
Reactions: Freebsd1976

Labs

Member
Mar 21, 2019
88
16
8
Hi,

Is there any way from uboot to dump the 16MB and 1GB flash chips and upload to tftp? I would like to make a backup of the 2 flash chips before attempting to erase/write other things.
 

Freebsd1976

Active Member
Feb 23, 2018
387
73
28
Hi,

Is there any way from uboot to dump the 16MB and 1GB flash chips and upload to tftp? I would like to make a backup of the 2 flash chips before attempting to erase/write other things.
Do it carefully my friend bricked his 6012
 

SGS

Member
May 24, 2017
43
20
8
54
today chat with one friend, he bricked his sx6012 after execute:
eeprom read 0x53 0x8500000 0 1958000
in uboot ,the switch report i2c error and freeze. he just unplug power cable
after that the switch just dead can’t boot up,is there any chance rescue his sx6012?
Sorry, I don't have a solution but I'm interested to know why he did that? There are no devices on slave address 0x53, unless I'm mistaken.

If anyone is tempted to play with the i2c commands in uboot I'd strongly advise against it. I bricked mine by reading too but managed to bring it back to life with a Diolan DLN-2-U2C via one of the internal i2c headers. I think i2c support in that uboot build is broken.

Edit: for reference, this is the command that bricked mine:

imd 0x52 0.2 0xFFF

That is supposed to read but actually wrote a zero to the first byte of the cpu bootstrap code in the boot rom
 
Last edited:

groove

Member
Sep 21, 2011
90
31
18
Hello everyone,

Reading through the guide and am looking at the contents of MLNX-OS image (file name image-PPC_M460EX-ppc-m460ex-20160609-202426.tgz) and I can't seem to find an 'image_layout.sh' in that image. I could locate the other file chad, hwd and ibd in the image that should the guide states should be replaced.

If I attempt the conversion, do I just copy image_layout.sh to /mnt/root2/etc even though it does not exist?

Please advise if it's ok for the file to no exist in the original image or if I got a wrong image.

Thanks
 
Last edited:

mpogr

Active Member
Jul 14, 2016
115
95
28
53
Hello everyone,

Reading through the guide and am looking at the contents of MLNX-OS image (file name image-PPC_M460EX-ppc-m460ex-20160609-202426.tgz) and I can't seem to find an 'image_layout.sh' in that image. I could locate the other file chad, hwd and ibd in the image that should the guide states should be replaced.

If I attempt the conversion, do I just copy image_layout.sh to /mnt/root2/etc even though it does not exist?

Please advise if it's ok for the file to no exist in the original image or if I got a wrong image.

Thanks
Yes
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
If anyone is tempted to play with the i2c commands in uboot I'd strongly advise against it. I bricked mine by reading too but managed to bring it back to life with a Diolan DLN-2-U2C via one of the internal i2c headers. I think i2c support in that uboot build is broken.

Edit: for reference, this is the command that bricked mine:

imd 0x52 0.2 0xFFF

That is supposed to read but actually wrote a zero to the first byte of the cpu bootstrap code in the boot rom
Yikes, that's really good to know about before starting on a conversion.