Beware of EMC switches sold as Mellanox SX6XXX on eBay

Eong

Member
Dec 24, 2019
35
7
8
I managed to hack the SX6018 firmware to apply power to high power transceivers (eg LR4 and BiDi) but the switch would not recognize them. I verified that the transceivers were powered and emitting laser light so I think the issue is that the switch firmware won't recognize them because they aren't SR4s. I tried both the Arista QSFP-40G-UNIV and the Cisco BiDi with both my SX6012s and my SX6018. I tried switch-to-switch between the SX6012s and between SX6012 and SX6018. I think the only answer is to find a way to reprogram the transceivers to advertise themselves as SR4.
I just updated my 6012 to latest 3.6.8012 and make it work. Now I have transceiver DDM support. It may help. Do you mind to have a conversation? I sent you private message but get no reply.
 
  • Like
Reactions: klui and Rand__

Rand__

Well-Known Member
Mar 6, 2014
4,629
919
113
so you gonna write instructions on how to upgrade the converted EMC switch? :)
 
Apr 21, 2016
47
17
8
40
I just updated my 6012 to latest 3.6.8012 and make it work. Now I have transceiver DDM support. It may help. Do you mind to have a conversation? I sent you private message but get no reply.
Hello,
It should be fairly straight forward to add support in the firmware - provided you have the mlx file to generate the binary.
Of course, the hardware capability needs to be there (AFAIK 6018/6012 have that on all ports).
If you have anything newer than the 9.4.2000 ver please, share :)

If it's of interest I can come up with the ini for this.
 

Eong

Member
Dec 24, 2019
35
7
8
so you gonna write instructions on how to upgrade the converted EMC switch? :)
No need, just replace those three binary files. The other part is almost the same. Host id is wrong and of course you need new firmware.
I want to ask if anyone has used a LR4 transceiver on it successfully? And I need a FRU dump binary from original 6012.
 
Last edited:

Eong

Member
Dec 24, 2019
35
7
8
Hello,
It should be fairly straight forward to add support in the firmware - provided you have the mlx file to generate the binary.
Of course, the hardware capability needs to be there (AFAIK 6018/6012 have that on all ports).
If you have anything newer than the 9.4.2000 ver please, share :)

If it's of interest I can come up with the ini for this.
I will share the files later. I do not use IB, I am not sure if IB works or not. Need time for testing.
Any hint to add full LR4 support in ini file? The power level is already 5 there. I do not think it's related to that. I can see the transceiver is powered but no light up.
 
Last edited:

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Yes, please share the updated binaries! :)

Instructions to enable LR4 would be a life saver! Or even how to extract/obtain the ini file to modify. I've been trying to get some BiDi transceivers to work forever.
 

Eong

Member
Dec 24, 2019
35
7
8
Yes, please share the updated binaries! :)

Instructions to enable LR4 would be a life saver! Or even how to extract/obtain the ini file to modify. I've been trying to get some BiDi transceivers to work forever.
Files will be shared soon.

I can not make my LR4 modules work even after upgrade. My friend confirms that SR4 works. If someone can run commands "mlxi2c show fru chassis" on their non-EMC SX60xx switch, it may be helpful.
 

klui

Active Member
Feb 3, 2019
194
82
28
I will share the files later. I do not use IB, I am not sure if IB works or not. Need time for testing.
Any hint to add full LR4 support in ini file? The power level is already 5 there. I do not think it's related to that. I can see the transceiver is powered but no light up.
Maybe you need to use the F_WT_Ax FW? Here's the output from BeTeP's mfa_extract.py script
Code:
./mfa_extract.py fw-SX-rel-9_4_5110-FIT.mfa l
 .
 .
MT_1270110020 MSX6012F_Ax                      SwitchX-2 based FDR InfiniBand Switch; 12 QSFP ports; Managed ; Subnet Manager for 648 nodes
MT_1270111029 MSX6012F_WT_Ax                   SwitchX-2 based FDR IB; 12 QSFP ports; Managed ; Subnet Manager for 648 nodes; P1-2 LR4 enabled
MT_1270110029 MSX6012T_Ax                      SwitchX-2 based FDR10 InfiniBand Switch; 12 QSFP ports; Managed ; Subnet Manager for 648 nodes
 

Eong

Member
Dec 24, 2019
35
7
8
There is no need for the mlx files. It is possible to replace ini in a bin file. I posted the tools to do it somewhere on the forum.

Found it
The problem is, 6012 already has power level 5. And it is be able to power on the transceivers.
Maybe you need to use the F_WT_Ax FW? Here's the output from BeTeP's mfa_extract.py script
Code:
./mfa_extract.py fw-SX-rel-9_4_5110-FIT.mfa l
.
.
MT_1270110020 MSX6012F_Ax                      SwitchX-2 based FDR InfiniBand Switch; 12 QSFP ports; Managed ; Subnet Manager for 648 nodes
MT_1270111029 MSX6012F_WT_Ax                   SwitchX-2 based FDR IB; 12 QSFP ports; Managed ; Subnet Manager for 648 nodes; P1-2 LR4 enabled
MT_1270110029 MSX6012T_Ax                      SwitchX-2 based FDR10 InfiniBand Switch; 12 QSFP ports; Managed ; Subnet Manager for 648 nodes
I tried that already before I upgraded os, not working for me.
 

klui

Active Member
Feb 3, 2019
194
82
28
The problem is, 6012 already has power level 5. And it is be able to power on the transceivers.

I tried that already before I upgraded os, not working for me.
The manual for 3.6.3004 states that the 6012 supports LR4 transceivers on all ports but the FW description seems to indicate only ports 1 and 2 are supported. Did you try ports 1 and 2?
 

Eong

Member
Dec 24, 2019
35
7
8
The manual for 3.6.3004 states that the 6012 supports LR4 transceivers on all ports but the FW description seems to indicate only ports 1 and 2 are supported. Did you try ports 1 and 2?
Tried everything I know. Tried transceivers from FS and HuaWei. Tried different firmwares. Tried different ports. Tried upgrading the os to 3.6.8012.
None of them works. I am so sad.
That's why I am guessing maybe something is missing in FRU. But I do not have the original one, or information read from an original one.
 
Last edited:

Eong

Member
Dec 24, 2019
35
7
8
Anyone can host the files for 3.6.8012 conversation? I need a place to share the files properly. PM me please.
 
Last edited:
  • Like
Reactions: klui

klui

Active Member
Feb 3, 2019
194
82
28
Here are the files from Eong.
lab.netservers.ro

PS: the full image can also be downloaded from Nvidia : MLNX-OS PPC M460EX SwitchX v3.6.8012 Image

Once I have a bit of time, I'll post the other files from the thread, so that we can keep them all in one place.
If something happens with the site, let me know.
Thank you @Eong and @Gabriel Mateiciuc.

@Necrotyr since you've been hosting the guide you may want to revise it with Eong's instructions hosted by Gabriel Mateiciuc.
 

Labs

Member
Mar 21, 2019
46
7
8
Here are the files from Eong.
lab.netservers.ro

PS: the full image can also be downloaded from Nvidia : MLNX-OS PPC M460EX SwitchX v3.6.8012 Image

Once I have a bit of time, I'll post the other files from the thread, so that we can keep them all in one place.
If something happens with the site, let me know.
I had a look at the files and I saw that ibd file is missing. We don't need to replace it or we can use the old one? I guess there is an update between 1002 and 8012 versions of the ibd.

I think the best way of conversion would be to dump the flash from Mellanox 6012 or maybe it's working from a 6018 or 6036 and try to flash directly those. After that update the FRU part so it can be seen as 6012. Or maybe the management board can identify the chassis type automatically at first reboot.

I tried for a long time to find a Mellanox 6012 model but they are rare and costs a lot compared to the EMC models.
 

Labs

Member
Mar 21, 2019
46
7
8
I will share the files later. I do not use IB, I am not sure if IB works or not. Need time for testing.
Any hint to add full LR4 support in ini file? The power level is already 5 there. I do not think it's related to that. I can see the transceiver is powered but no light up.
From what I understand it says that for ethernet all ports are enabled and only in the case of infiniband you need to change the ini to allow high-power flow which is part of the link handshake.

At the end of the page in the Note is written also this "Only "Mellanox" LR4 modules are supported on these switches". Did you try with the Mellanox LR4 or with a OEM module? Looks like the switches have a white list of allowed LR4 modules or the Mellanox LR4 modules are custom made and that's why only those are allowed.
 

wanderer

New Member
Apr 29, 2020
1
0
1
Hi all,

what is the easiest (and cheapest) way to connect one of these to an existing gigabit switch?

or is there breakout cables to rj45? (i tried to find some but couldn't)
This is obviously to keep the rest of the network running if the ethernet switches have no SFP ports.

Thanks
 

Freebsd1976

Active Member
Feb 23, 2018
227
33
28
Hi all,

what is the easiest (and cheapest) way to connect one of these to an existing gigabit switch?

or is there breakout cables to rj45? (i tried to find some but couldn't)
This is obviously to keep the rest of the network running if the ethernet switches have no SFP ports.

Thanks
I use QSA(mellanox ) ->sfp to rj45 module (molex)