Beware of EMC switches sold as Mellanox SX6XXX on eBay

mpogr

Active Member
Jul 14, 2016
113
80
28
49
Yes, I meant RoCE, and, indeed, CX-3 supports v1, but only CX-3 Pro and up support v2, which is quite a bit of a difference latency-wise.
I have done some measurements with my setup using SRP a while ago, now need to redo the same baseline to see how things are different with iSER, but, from memory, there was quite a bit of performance loss, especially with multithreaded workloads.
 

oxynazin

Member
Dec 10, 2018
37
20
8
Found this huge article: https://www.demartek.com/Reports_Free/Demartek_IBTA_RoCE_Deployment_Guide_2018-07.pdf Have not read all article, because there're too many new information for me. But found some interesting parts. For SMB Direct there's intersting part: "If RoCE v2 is not necessary for routing purposes, RoCE v1 may give better performance.". And in the end: "The addition of ETS and RoCEv2 Congestion Management resulted in improved RoCE throughput and decreased latency.". As I understood, v2 adds some overhead for Layer3 support but also adds "RoCEv2 Congestion Management (RCM)". So v1 must be at least as fast as v2, until you not utilize network for 100%. Am I right?
 

manfri

Member
Nov 19, 2015
44
7
8
53
Kudos to mpogr for the guide... great work... but a bit extreme for my heart, and tooo much effort for my needs...
I'm gonna pass :cool:))
 

Tim111

New Member
Jan 9, 2019
11
0
1
Just signed up, is there a way I can get information for flashing EMC sx6012 please.
 

Terry Wallace

PsyOps SysOp
Aug 13, 2018
138
78
28
51
Allen, TX
If you start at page 1 of this thread and read through it just about all the information has been presented here. :)
Good luck on conversion.
Post when you get stuck on a specific step and we'll walk you through it.
 

Freebsd1976

Active Member
Feb 23, 2018
202
30
28
an off-topic question , howto connect sx6012 to internet?
my sx6012 only for spark cluster in my home lab, no internet access,.
now for system update which need internet access,I consider connect it to other switch for internet access, my other switch have rj45 1G port, but the question is how to connect two switch , sx6012 only the management port is rj45?
 

Freebsd1976

Active Member
Feb 23, 2018
202
30
28
Another question , I know we could not upgrade os ,but could we upgrade the firmware, anyone try this.
 

Freebsd1976

Active Member
Feb 23, 2018
202
30
28
So, I also manage to modify one of these switches :)
Thanks mpogr for sharing your knowledge!

The one thing that really pissed me off was how noisy these little things are!
Instead of buying new low speed fans which would be just as expensive as the switch, I decided I would have to use the Fans that came with it. I did a lot of experiments and I came up with a solution to keep them quiet without modifying the switch irreversibly. So in order to keep them quiet I decided to lower their voltage thus making them spin slower. A 5 volt linear regulator could do the job except that it would get very hot, and switched regulators are usually big (compared with a 7805).
But then I found these on ebay:
10pcs DC-DC 12-24V To 5V 3A Mini Step Down Module Buck Converter Adjustable | eBay

These small PCB's are cheap as beans and they are the same size as a 7805 plus they allow for variable output (which allows for tuning fan speed to almost any noise desired). I striped 4 noctua NA-RC7 low-noise adapters I had (they are useless with these switches by the way...) and made 4 adapters of them. The small PCB's have a potentiometer that allows to change output voltage. 5V seems to be a good ratio between noise/speed that will allow the switch to operate safelly if you are not going to use it with 12 ports on all the time at full speed.

I'm getting temps bellow 40ºC on all boards except CPU Board Monitor 2 which is at 52ºC. This is with 4 passive DACs and one QSFP+ adapter connected. PWM and Tacometer will work as well with this mod (although PWM will not make fan speed change a lot at low voltages). Also the switch will complain (you will get an alarm) of low Fan speed. Haven't figured a way of getting rid of this, but it doesn't bother me at all.

Anyway, just decided to share this, since it might be usefull to someone. Please use this with care, reducing FAN speeds to very low speed/air flow can kill the switch.


PS: the voltage regulators have an enable pin whitch should be left floating.
PPS: also another nice thing about this is that power consumption drops by almost 20W, making this switch running at arround 30W... that's insane for the speeds this thing can do.:D

-Edit- correction regarding Enable pin on these devices.
thank a lot , this is want I want. Btw how to do make this output fixed 5v ?the default output seems adjustable .

PS : find howto fix output ,
5PCS Mini DC to DC Adjustable Step Down Module 12/24V to 5V 3A Buck Converter Power Supply Module
There is a YouTube video demo howto do this
 
Last edited:

llowrey

Member
Feb 26, 2018
82
53
18
High power transceivers (eg LR4 and Cisco BiDi) are not supported by the SX6018. Interestingly, the SX6012 supports them on all ports. I verified that (SX6012 and SX6018) with Arista QSFP-40G-UNIV transceivers (LR4 but supporting duplex MMF in addition to SMF).

It should be a simple matter of adding "module_power_level_supported=5" to the "[module#]" sections for each port in the firmware config. The problem is that without the .mlx firmware source it does not appear to be possible to generate a firmware image (.bin) with a custom config.

Questions:
  1. Do any of you have, or know where to find, a source .mlx for the SX6018?
  2. Is there a way to modify the config entries of a .bin firmware image?
  3. Do any of you have a support contract that would allow you to request an .mlx for the SX6018?
 

RageBone

Active Member
Jul 11, 2017
291
77
28
i think that there was a tool to read configs from network-cards, might work with switches too.

EDIT:
i think it was one of the few standard tools, mstflint or so.

Edit:
http://www.mellanox.com/page/management_tools
Debug-Utils sounds interesting.
I think i used MLXBurn before but its to long ago.
mlxdump, mstdump sound very promising
 
Last edited:

llowrey

Member
Feb 26, 2018
82
53
18
I used flint to pull the config from the current firmware image .bins for the SX6018 and the SX6012 which is how I learned that the magic entry is "module_power_level_supported=5". My problem now is how to either load an updated config into the switch or to modify the firmware with a new config.

As far as I can tell the only way to modify a firmware is to generate a new one using mlxburn and that requires an .mlx source firmware file, which I don't have. I have not seen any tool which will burn just the config.

It seems Mellanox used to publish firmware sources (.mlx files) on their website, at least for adapter cards, until just a couple of years ago.
 

Freebsd1976

Active Member
Feb 23, 2018
202
30
28
I used flint to pull the config from the current firmware image .bins for the SX6018 and the SX6012 which is how I learned that the magic entry is "module_power_level_supported=5". My problem now is how to either load an updated config into the switch or to modify the firmware with a new config.

As far as I can tell the only way to modify a firmware is to generate a new one using mlxburn and that requires an .mlx source firmware file, which I don't have. I have not seen any tool which will burn just the config.

It seems Mellanox used to publish firmware sources (.mlx files) on their website, at least for adapter cards, until just a couple of years ago.
Yeah , if you find a way to modify it , maybe also could set fan speed, this from config.ini for fan control

[fan]

fan_en=1

fan0_mode = PWM_OUT

fan1_mode = TACHO_IN

fan2_mode = NOT_USED

fan3_mode = TACHO_IN

fan4_mode = NOT_USED

fan5_mode = TACHO_IN

fan6_mode = NOT_USED

fan7_mode = TACHO_IN

fan8_mode = NOT_USED

fan9_mode = NOT_USED


fan1_hi_limit_rpm = 23000

fan3_hi_limit_rpm = 23000

fan5_hi_limit_rpm = 23000

fan7_hi_limit_rpm = 23000