Beware of EMC switches sold as Mellanox SX6XXX on eBay

[oxynazin]

He probably means RDMA over Converged Ethernet ("RoCE")

But CX3 supports RoCE according to their product brief pdf. Unclear v1 or v2, but somewhere I saw that CX2 - RoCEv1, CX3 - RoCEv2.

 

[mpogr]

Yes, I meant RoCE, and, indeed, CX-3 supports v1, but only CX-3 Pro and up support v2, which is quite a bit of a difference latency-wise.
I have done some measurements with my setup using SRP a while ago, now need to redo the same baseline to see how things are different with iSER, but, from memory, there was quite a bit of performance loss, especially with multithreaded workloads.

 

[oxynazin]

Found this huge article: https://www.demartek.com/Reports_Free/Demartek_IBTA_RoCE_Deployment_Guide_2018-07.pdf Have not read all article, because there're too many new information for me. But found some interesting parts. For SMB Direct there's intersting part: "If RoCE v2 is not necessary for routing purposes, RoCE v1 may give better performance.". And in the end: "The addition of ETS and RoCEv2 Congestion Management resulted in improved RoCE throughput and decreased latency.". As I understood, v2 adds some overhead for Layer3 support but also adds "RoCEv2 Congestion Management (RCM)". So v1 must be at least as fast as v2, until you not utilize network for 100%. Am I right?

 

[zxv]

Here's some additional discussion of RoCE V1 vs RoCE v2
with diagrams of packet layout.

https://community.mellanox.com/s/article/roce-v2-considerations

 

[manfri]

Kudos to mpogr for the guide... great work... but a bit extreme for my heart, and tooo much effort for my needs...
I'm gonna pass ))

 

[Tim111]

Just signed up, is there a way I can get information for flashing EMC sx6012 please.

 

[Terry Wallace]

If you start at page 1 of this thread and read through it just about all the information has been presented here.
Good luck on conversion.
Post when you get stuck on a specific step and we'll walk you through it.

 

[Tim111]

Thanks Terry.

 

[zxv]

For those that could not open the Demartek PDF link above, there is another link:
http://www.mellanox.com/related-doc...martek_IBTA_RoCE_Deployment_Guide_2018-07.pdf

 

[i386]

Checked my ebay alarms today and found a forgotten item:
Mellanox MSX6036F 36-Port FDR/QSFP 1U Managed Fibre Channel Switch w/ 2x PSU | eBay

• 500$ or bo, free shipping
• fdr (not qdr or fdr10 like other listings) infiniband switch
• managed
  
• 36 ports
• very likely hp branded (mlnx-os updates for free from hpe)
  
• rear to front airflow
• can be "converted" to a vpi/ethernet switch

 

[Rand__]

Mellanox MSX6036F-1SFS 36 Port QSFP FDR Managed InfiniBand Switch SX6036 2 AC 7290107194792 | eBay
Also seems to be HP, $425

 

[Freebsd1976]

an off-topic question , howto connect sx6012 to internet?
my sx6012 only for spark cluster in my home lab, no internet access,.
now for system update which need internet access,I consider connect it to other switch for internet access, my other switch have rj45 1G port, but the question is how to connect two switch , sx6012 only the management port is rj45?

 

[i386]

You could use a qsfp to sfp Adapter and a rj45 sfp Module

 

[Freebsd1976]

You could use a qsfp to sfp Adapter and a rj45 sfp Module

thanks a lot ，I found the other switch has one 10g port, so I think I can buy one qsfp+ to four sfp fan out cable uplink for internet access

 

[Freebsd1976]

Another question , I know we could not upgrade os ,but could we upgrade the firmware, anyone try this.

 

[Freebsd1976]

So, I also manage to modify one of these switches
Thanks mpogr for sharing your knowledge!

The one thing that really pissed me off was how noisy these little things are!
Instead of buying new low speed fans which would be just as expensive as the switch, I decided I would have to use the Fans that came with it. I did a lot of experiments and I came up with a solution to keep them quiet without modifying the switch irreversibly. So in order to keep them quiet I decided to lower their voltage thus making them spin slower. A 5 volt linear regulator could do the job except that it would get very hot, and switched regulators are usually big (compared with a 7805).
But then I found these on ebay:
10pcs DC-DC 12-24V To 5V 3A Mini Step Down Module Buck Converter Adjustable | eBay

These small PCB's are cheap as beans and they are the same size as a 7805 plus they allow for variable output (which allows for tuning fan speed to almost any noise desired). I striped 4 noctua NA-RC7 low-noise adapters I had (they are useless with these switches by the way...) and made 4 adapters of them. The small PCB's have a potentiometer that allows to change output voltage. 5V seems to be a good ratio between noise/speed that will allow the switch to operate safelly if you are not going to use it with 12 ports on all the time at full speed.

I'm getting temps bellow 40ºC on all boards except CPU Board Monitor 2 which is at 52ºC. This is with 4 passive DACs and one QSFP+ adapter connected. PWM and Tacometer will work as well with this mod (although PWM will not make fan speed change a lot at low voltages). Also the switch will complain (you will get an alarm) of low Fan speed. Haven't figured a way of getting rid of this, but it doesn't bother me at all.

Anyway, just decided to share this, since it might be usefull to someone. Please use this with care, reducing FAN speeds to very low speed/air flow can kill the switch.


PS: the voltage regulators have an enable pin whitch should be left floating.
PPS: also another nice thing about this is that power consumption drops by almost 20W, making this switch running at arround 30W... that's insane for the speeds this thing can do.

-Edit- correction regarding Enable pin on these devices.

thank a lot , this is want I want. Btw how to do make this output fixed 5v ?the default output seems adjustable .

PS : find howto fix output ,
5PCS Mini DC to DC Adjustable Step Down Module 12/24V to 5V 3A Buck Converter Power Supply Module
There is a YouTube video demo howto do this

 

[llowrey]

High power transceivers (eg LR4 and Cisco BiDi) are not supported by the SX6018. Interestingly, the SX6012 supports them on all ports. I verified that (SX6012 and SX6018) with Arista QSFP-40G-UNIV transceivers (LR4 but supporting duplex MMF in addition to SMF).

It should be a simple matter of adding "module_power_level_supported=5" to the "[module#]" sections for each port in the firmware config. The problem is that without the .mlx firmware source it does not appear to be possible to generate a firmware image (.bin) with a custom config.

Questions:

1. Do any of you have, or know where to find, a source .mlx for the SX6018?
2. Is there a way to modify the config entries of a .bin firmware image?
3. Do any of you have a support contract that would allow you to request an .mlx for the SX6018?

 

[RageBone]

i think that there was a tool to read configs from network-cards, might work with switches too.

EDIT:
i think it was one of the few standard tools, mstflint or so.

Edit:
http://www.mellanox.com/page/management_tools
Debug-Utils sounds interesting.
I think i used MLXBurn before but its to long ago.
mlxdump, mstdump sound very promising

 

[llowrey]

I used flint to pull the config from the current firmware image .bins for the SX6018 and the SX6012 which is how I learned that the magic entry is "module_power_level_supported=5". My problem now is how to either load an updated config into the switch or to modify the firmware with a new config.

As far as I can tell the only way to modify a firmware is to generate a new one using mlxburn and that requires an .mlx source firmware file, which I don't have. I have not seen any tool which will burn just the config.

It seems Mellanox used to publish firmware sources (.mlx files) on their website, at least for adapter cards, until just a couple of years ago.

 

[Freebsd1976]

I used flint to pull the config from the current firmware image .bins for the SX6018 and the SX6012 which is how I learned that the magic entry is "module_power_level_supported=5". My problem now is how to either load an updated config into the switch or to modify the firmware with a new config.

As far as I can tell the only way to modify a firmware is to generate a new one using mlxburn and that requires an .mlx source firmware file, which I don't have. I have not seen any tool which will burn just the config.

It seems Mellanox used to publish firmware sources (.mlx files) on their website, at least for adapter cards, until just a couple of years ago.

Yeah , if you find a way to modify it , maybe also could set fan speed, this from config.ini for fan control

[fan]

fan_en=1

fan0_mode = PWM_OUT

fan1_mode = TACHO_IN

fan2_mode = NOT_USED

fan3_mode = TACHO_IN

fan4_mode = NOT_USED

fan5_mode = TACHO_IN

fan6_mode = NOT_USED

fan7_mode = TACHO_IN

fan8_mode = NOT_USED

fan9_mode = NOT_USED


fan1_hi_limit_rpm = 23000

fan3_hi_limit_rpm = 23000

fan5_hi_limit_rpm = 23000

fan7_hi_limit_rpm = 23000