Beware of EMC switches sold as Mellanox SX6XXX on eBay

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Hindsight

Member
Mar 28, 2016
55
14
8
42
Is there any way to modify the ini file for the SX6018? I want to use a few Cisco QSFP-40G-SR-BDs in it, but it identifies it as a high power transceiver. LR4 Support on Mellanox Switches | Mellanox Interconnect Community says that it's possible to edit the ini in the SX6012 to enable it but doesn't mention the 6018. Going to pick up some Astria QSFP-40G-UNIV and hope those work for now.
At what price? for $39 you can get mellonox branded ones from fs.com
 

mb300sd

Active Member
Aug 1, 2016
204
80
28
34
At what price? for $39 you can get mellonox branded ones from fs.com
I need BiDi or UNIV to run 40G over duplex LC fiber. Unfortunately the $39 ones are MPO and my house is wired with LC, did it during a renovation and there's no way to change it without ripping out all the walls again.
 

Hindsight

Member
Mar 28, 2016
55
14
8
42
I need BiDi or UNIV to run 40G over duplex LC fiber. Unfortunately the $39 ones are MPO and my house is wired with LC, did it during a renovation and there's no way to change it without ripping out all the walls again.
Ahh that's always fun. If you ever re-do it add some conduit so you can run new stuff easily.
 

im2geek4you

Member
Feb 27, 2017
30
10
8
43
So, I also manage to modify one of these switches :)
Thanks mpogr for sharing your knowledge!

The one thing that really pissed me off was how noisy these little things are!
Instead of buying new low speed fans which would be just as expensive as the switch, I decided I would have to use the Fans that came with it. I did a lot of experiments and I came up with a solution to keep them quiet without modifying the switch irreversibly. So in order to keep them quiet I decided to lower their voltage thus making them spin slower. A 5 volt linear regulator could do the job except that it would get very hot, and switched regulators are usually big (compared with a 7805).
But then I found these on ebay:
10pcs DC-DC 12-24V To 5V 3A Mini Step Down Module Buck Converter Adjustable | eBay

These small PCB's are cheap as beans and they are the same size as a 7805 plus they allow for variable output (which allows for tuning fan speed to almost any noise desired). I striped 4 noctua NA-RC7 low-noise adapters I had (they are useless with these switches by the way...) and made 4 adapters of them. The small PCB's have a potentiometer that allows to change output voltage. 5V seems to be a good ratio between noise/speed that will allow the switch to operate safelly if you are not going to use it with 12 ports on all the time at full speed.

I'm getting temps bellow 40ºC on all boards except CPU Board Monitor 2 which is at 52ºC. This is with 4 passive DACs and one QSFP+ adapter connected. PWM and Tacometer will work as well with this mod (although PWM will not make fan speed change a lot at low voltages). Also the switch will complain (you will get an alarm) of low Fan speed. Haven't figured a way of getting rid of this, but it doesn't bother me at all.

Anyway, just decided to share this, since it might be usefull to someone. Please use this with care, reducing FAN speeds to very low speed/air flow can kill the switch.


PS: the voltage regulators have an enable pin whitch should be left floating.
PPS: also another nice thing about this is that power consumption drops by almost 20W, making this switch running at arround 30W... that's insane for the speeds this thing can do.:D

-Edit- correction regarding Enable pin on these devices.
 

Attachments

Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
So these do VPI (Both IB and Ethernet) after converting to Mellanox's firmware? That price on the sx6012 is really tempting in that case, holy cow...

Are FDR DACs actually a different pinout or construction from standard QSFP or do the cables just have a capability bit set in firmware? If it's a bit flag it'd be interesting to find a source of cheap DACs that aren't write protected, flip the capability bit and see how they perform.

So, I also manage to modify one of these switches :)
Thanks mpogr for sharing your knowledge!

The one thing that really pissed me off was how noisy these little things are!
Instead of buying new low speed fans which would be just as expensive as the switch, I decided I would have to use the Fans that came with it. I did a lot of experiments and I came up with a solution to keep them quiet without modifying the switch irreversibly. So in order to keep them quiet I decided to lower their voltage thus making them spin slower. A 5 volt linear regulator could do the job except that it would get very hot, and switched regulators are usually big (compared with a 7805).
But then I found these on ebay:
10pcs DC-DC 12-24V To 5V 3A Mini Step Down Module Buck Converter Adjustable | eBay

These small PCB's are cheap as beans and they are the same size as a 7805 plus they allow for variable output (which allows for tuning fan speed to almost any noise desired). I striped 4 noctua NA-RC7 low-noise adapters I had (they are useless with these switches by the way...) and made 4 adapters of them. The small PCB's have a potentiometer that allows to change output voltage. 5V seems to be a good ratio between noise/speed that will allow the switch to operate safelly if you are not going to use it with 12 ports on all the time at full speed.

I'm getting temps bellow 40ºC on all boards except CPU Board Monitor 2 which is at 52ºC. This is with 4 passive DACs and one QSFP+ adapter connected. PWM and Tacometer will work as well with this mod (although PWM will not make fan speed change a lot at low voltages). Also the switch will complain (you will get an alarm) of low Fan speed. Haven't figured a way of getting rid of this, but it doesn't bother me at all.

Anyway, just decided to share this, since it might be usefull to someone. Please use this with care, reducing FAN speeds to very low speed/air flow can kill the switch.


PS: the voltage regulator have an enable pin whitch must be connected to Vin.
PPS: also another nice thing about this is that power consumption drops by almost 20W, making this switch running at arround 30W... that's insane for the speeds this thing can do.:D
That is super slick, nice work! I'm going to keep these in mind the next time I'm quieting a switch.
 

Hindsight

Member
Mar 28, 2016
55
14
8
42
The one thing that really pissed me off was how noisy these little things are!
Thanks, That could have been useful for another switch I replaced fans in, will keep it for future. This switch to me isn't that bad when booted into mlnx-os. The EMC boot is extremely loud though.
 

JSLEnterprises

New Member
May 4, 2018
20
22
3
41
Anyone having the issue of the management site being stuck "initializing" after successfully doing the conversion?
I can manage it via CLI, but the management site seems stuck.
 

Hindsight

Member
Mar 28, 2016
55
14
8
42
Anyone having the issue of the management site being stuck "initializing" after successfully doing the conversion?
I can manage it via CLI, but the management site seems stuck.
I think it might have taken a long time (30-40 minutes) I was about to power it off when it came up, after the first time it has always been quick.
 

JSLEnterprises

New Member
May 4, 2018
20
22
3
41
Yeah, still no luck, it might be because I haven't updated the firmware yet. I don't have the ability to update it via inband methods (im preparing my switches via laptop at my other residence, while my homelab is a good 3 hours away.
is there a method to update it out of band through the management port?

@Hindsight, does your emc-mellanox switch also have the part number starting 100-886- ?
 
Last edited:

Hindsight

Member
Mar 28, 2016
55
14
8
42
Yeah, still no luck, it might be because I haven't updated the firmware yet. I don't have the ability to update it via inband methods (im preparing my switches via laptop at my other residence, while my homelab is a good 3 hours away).
is there a method to update it out of band through the management port?

@Hindsight, does your emc-mellanox switch also have the part number starting 100-886- ?
Yes it does, And you can update the firmware on the switch directly but I haven't tried it. You can pm mpogr for instructions.
 

Hindsight

Member
Mar 28, 2016
55
14
8
42
I finally got my FDR Cables and when I attached them they connect at 40GB. I switch them to 56 GB in the web console and they now show down. VMware only lets me select 40000 for speed. Is there anything else I need to do to get this to work?

My nics are cx-3
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
  • Like
Reactions: JSLEnterprises

mpogr

Active Member
Jul 14, 2016
115
95
28
53
@Hindsight this is strange. The only case I had such a problem was when I tried putting CX-3 Pro firmware on my CX-3 (non-Pro) cards. Then they lost 56Gbps speed. But, otherwise, never had issues like this...
 

mb300sd

Active Member
Aug 1, 2016
204
80
28
34
Does anyone know the arguments to the i2c command on the switch? Apparently the BiDi QSFPs I'm using are writable, but only temporarily, and reset on power down. I want to try and send an i2c write in the init script and see if that'll get the switch to activate them.

Code:
[admin@SX6018 mlnx_i2c]# i2c 0 r 50 1 0 256
All parameters are interpreted as hexa values. Application return
code is zero only when read/write was successfully completed.

[admin@SX6018 mlnx_i2c]# i2c 0 r 50 1 0 256

Usage:

        i2c [OPTIONS] <device> <cmd> <i2c_addr> <addr> [<data>]


OPTIONS are:
        -h              - Print this message
        -v              - Print Version info
        -a <addr_width> - Set address width to specified value. May
                          be 0, 1, 2 or 4. Default is 1.
        -d <data_width> - Set data width to specified value. May
                          be 1, 2 or 4. Default is 1.
        -x <data_len>   - Present data as two hexa digit string
                          (like 01202040ac). Note that this option is
                          mutually exclusive with "-d" option

Parameters are:
        <device>        - Valid MST device.
        <cmd>           - Command. May be "r[ead]" or "w[rite]".
        <i2c_addr>      - I2C slave address.
        <addr>          - Address (addr_width bytes length) inside I2C
                          target device to read/write operation.
        <data>          - Data (data_width bytes length) to write to
                          target device.

Note that <addr> is ignored if <addr_witdh> == 0
 

JSLEnterprises

New Member
May 4, 2018
20
22
3
41
I give up. I cant get any of the ports to initialize (qsfp), so I cant fw update via inband.
I cant update the fw from the switch, even though flint is literally sitting on the switch ( /dev/mtdblock7/usr/bin/ ), and so is the god damn bin.
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
If you're willing to mail it to me instead of tossing it I'll pay postage :p
 

Hindsight

Member
Mar 28, 2016
55
14
8
42
I give up. I cant get any of the ports to initialize (qsfp), so I cant fw update via inband.
I cant update the fw from the switch, even though flint is literally sitting on the switch ( /dev/mtdblock7/usr/bin/ ), and so is the god damn bin.
Just give it a day or two break and try some more, it took me 2 days, loud as hell in my room. One of the guys last week had the same problems with port not coming up. Are you running the open subnet manager? ports won't come on without it. He had to buy a DAC before it worked for him.
 

JSLEnterprises

New Member
May 4, 2018
20
22
3
41
Just give it a day or two break and try some more, it took me 2 days, loud as hell in my room. One of the guys last week had the same problems with port not coming up. Are you running the open subnet manager? ports won't come on without it. He had to buy a DAC before it worked for him.
Using monoprice dac's (have a qdr IBM switch which my 2 mel/emc switches are going to replace, and the dac's work fine on it. however, it only does l2 ethernet switching when not doing ib which is why I want to swap it out)
Open Subnet manager is running on one of my servers, connection doesn't come up.
cards i use are are cx-3 bcct's (which don't do InfiniBand)
 
Last edited: