SOLVED Mellanox SX6036 + LR4 Transceivers does not work
- Thread starter NablaSquaredG
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
WARNING THIS MIGHT BREAK YOUR SWITCH BE SURE YOU KNOW WHAT YOU ARE DOING -- I TAKE NO RESPONSIBILITY FOR THE FOLLOWING FW FILE OR COMMANDS OR IF YOU PERMANENTLY DAMAGE YOUR SWITCH
I was in search for the firmware as well so I reached out to @llowrey as he was the first to mention that the custom firmware that exists and there are some limitations that you need to keep track of that I have been told but never confirmed.
Max watts per QSFP port: 3.5W
Max watts per 4 port QSFP group: 14W (1-4, 5-8, 9-12, etc. etc.)
I have been running the firmware for a week now but others have been running it for a while without any problems. You should still consider this to be risky. I haven't heard any reports of the firmware or process bricking switches.
You will need to know the process that EMC to Mellanox switch flashing goes as you will need to run shell commands, link can be found here on google docs v1.2 since the main thread often has the site down.
You'll need to be able to get to a shell prompt via the
Here are the steps I took to get this to work.
Reboot, and once restarted log back in and enable LR4
That is the steps I took to do it. Good luck!
All credit is to @llowrey as he reached out and gave me the steps.
I was in search for the firmware as well so I reached out to @llowrey as he was the first to mention that the custom firmware that exists and there are some limitations that you need to keep track of that I have been told but never confirmed.
Max watts per QSFP port: 3.5W
Max watts per 4 port QSFP group: 14W (1-4, 5-8, 9-12, etc. etc.)
I have been running the firmware for a week now but others have been running it for a while without any problems. You should still consider this to be risky. I haven't heard any reports of the firmware or process bricking switches.
You will need to know the process that EMC to Mellanox switch flashing goes as you will need to run shell commands, link can be found here on google docs v1.2 since the main thread often has the site down.
You'll need to be able to get to a shell prompt via the
_shell command in order to run flint.Here are the steps I took to get this to work.
- Updated to latest MLNX-OS 8012
- Download HP Firmware here.
- It contains firmware for SX6018 and SX6036 (I am unsure about SX6012)
MT_1240212020for SX6018MT_1010310020for SX6036
- Console into switch, and get to the shell by running
_shell - I mounted the file system in rw
mount -nwo remount,rw / - Created a temp directory
mkdir /mnt/tempand copied the firmware file over and named itMT_1010310020_HP.bin(for my SX6036) - ran the flashing command
flint --override_cache_replacement --allow_psid_change -d /dev/mst/mt51000_pciconf0 -i /mnt/temp/MT_1010310020_HP.bin b
- clean up folder
rm -rf /mnt/temp
Code:
-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
Current FW version on flash: 9.4.5110
New FW version: 9.4.5110
Note: The new FW version is the same as the current FW version on flash.
Do you want to continue ? (y/n) [n] : y
You are about to replace current PSID on flash - "DEL1010310020" with a different PSID - "MT_1010310020".
Note: It is highly recommended not to change the PSID.
Do you want to continue ? (y/n) [n] : yfae cable-stamping-unlock 40g_lr4That is the steps I took to do it. Good luck!
All credit is to @llowrey as he reached out and gave me the steps.
Thanks!
Couple things your aws blob storage doesnt have anonymous access. No worries though as you can get the files from here:
Anyways I got it all working… well kinda… i got past the “High power transceiver is not supported.” But now on switch it shows admin state as up though operational state is down.
I got to troubleshoot more because I hooked this into windows 11 via a dual port Mellanox CX314A (which I got a few others in linux systems without issue), with Cisco QSFP-40G-SR-BD gbics, and a 10m LC-LC Duplex 50/125 Multimode OM3 fiber cable. My linux systems are rack mounted so they use qsfp to qsfp copper passive cables.
Couple things your aws blob storage doesnt have anonymous access. No worries though as you can get the files from here:
Anyways I got it all working… well kinda… i got past the “High power transceiver is not supported.” But now on switch it shows admin state as up though operational state is down.
I got to troubleshoot more because I hooked this into windows 11 via a dual port Mellanox CX314A (which I got a few others in linux systems without issue), with Cisco QSFP-40G-SR-BD gbics, and a 10m LC-LC Duplex 50/125 Multimode OM3 fiber cable. My linux systems are rack mounted so they use qsfp to qsfp copper passive cables.
I can confirm that Cisco BiDi and Arista UNIV transceivers work in CX-4 Lx cards at 40GbE over duplex MMF.
Do you any information on how you were able to dump the firmware? I 've been looking through one of the update images for any mentions of fans and rpm so that I could remove the fan rpm minimum. This thing is too loud even at 15% from the fae fan speed modifications and I would like to be able to stop the fan from spinning up at low rpm so I can do a fan replacement and add a heatsink if it makes sense
Last edited:
I got my shiny old SX6036 in the rack today. I have a bunch of 3m QSFP+ DACs working to several servers with ConnectX-3 Pro NICs. I also got 4x Cisco QSFP-40G-SR-BD modules. I intend to use these to connect to my desktops with ConnectX-3 Pro NICs when they arrive in the mail this week.
Anyway, I am a little unclear of the firmware "mod." Does it just unlock the ability to have high-power optics on all ports or does it unlock something more? I installed the modded firmware successfully, so I guess it's silly I'm asking now, but I'd like to know.
Additionally, I have one QSFP-40G-SR-BD in port 33 and fae cable-stamping-unlock 40g_lr4 is set but I am not getting any light from the QSFP module (using a phone camera to check). Am I doing something wrong or is this incompatible?
Anyway, I am a little unclear of the firmware "mod." Does it just unlock the ability to have high-power optics on all ports or does it unlock something more? I installed the modded firmware successfully, so I guess it's silly I'm asking now, but I'd like to know.
Additionally, I have one QSFP-40G-SR-BD in port 33 and fae cable-stamping-unlock 40g_lr4 is set but I am not getting any light from the QSFP module (using a phone camera to check). Am I doing something wrong or is this incompatible?
I tried ConnectX3 to SX6036 using the Cisco MMF BiDi optics and neither end appears to light up. Any way to get these to work or is this a losing battle?
EDIT: I ordered a CX4 card for my remote desktop. I still want to know if anyone else got a MMF BiDi working with the SX6036.
EDIT: I ordered a CX4 card for my remote desktop. I still want to know if anyone else got a MMF BiDi working with the SX6036.
Last edited:
Interesting. I currently use UNIV in my workstation but it has a CX4 Lx connected to the sx6012.
Here's the connection on one of my lab systems with a CX3 using MMF in loopback. According to ethtool it's 2.5W maximum.
Here's the connection on one of my lab systems with a CX3 using MMF in loopback. According to ethtool it's 2.5W maximum.
Code:
# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4099_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:05:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 01
/dev/mst/mt4099_pci_cr0 - PCI direct access.
domain:bus:dev.fn=0000:05:00.0 bar=0xd3300000 size=0x100000
Chip revision is: 01
# lspci|grep -iv intel
04:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 430] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GF108 High Definition Audio Controller (rev a1)
05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
07:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
# dmesg|grep renamed
[ 7.105062] igb 0000:02:00.1 enp2s0f1: renamed from eth1
[ 7.385009] igb 0000:02:00.0 enp2s0f0: renamed from eth0
[ 10.765596] mlx4_core 0000:05:00.0 enp5s0: renamed from eth0
[ 11.116003] mlx4_core 0000:05:00.0 enp5s0d1: renamed from eth0
# ethtool enp5s0
Settings for enp5s0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
56000baseCR4/Full
56000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 40000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000014 (20)
link ifdown
Link detected: yes
# ethtool -m enp5s0d1
Identifier : 0x0d (QSFP+)
Extended identifier : 0x80
Extended identifier description : 2.5W max. Power consumption
Extended identifier description : No CDR in TX, No CDR in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 40G Ethernet: 40G Base-LR4
Encoding : 0x05 (64B/66B)
BR, Nominal : 10300Mbps
Rate identifier : 0x00
Length (SMF,km) : 1km
Length (OM3 50um) : 150m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1310.000nm
Laser wavelength tolerance : 6.500nm
Vendor name : Arista Networks
Vendor OUI : 00:1c:73
Vendor PN : QSFP-40G-UNIV
Vendor rev : 20
Vendor SN : XLA100000000
Date code : 17073000
Revision Compliance : SFF-8636 Rev 1.5
Module temperature : 41.20 degrees C / 106.16 degrees F
Module voltage : 3.2594 V
Alarm/warning flags implemented : No
Laser tx bias current (Channel 1) : 51.746 mA
Laser tx bias current (Channel 2) : 60.846 mA
Laser tx bias current (Channel 3) : 51.016 mA
Laser tx bias current (Channel 4) : 45.952 mA
Transmit avg optical power (Channel 1) : 0.8690 mW / -0.61 dBm
Transmit avg optical power (Channel 2) : 0.8958 mW / -0.48 dBm
Transmit avg optical power (Channel 3) : 0.9884 mW / -0.05 dBm
Transmit avg optical power (Channel 4) : 0.8667 mW / -0.62 dBm
Rcvr signal avg optical power(Channel 1) : 1.2307 mW / 0.90 dBm
Rcvr signal avg optical power(Channel 2) : 1.3125 mW / 1.18 dBm
Rcvr signal avg optical power(Channel 3) : 1.3604 mW / 1.34 dBm
Rcvr signal avg optical power(Channel 4) : 1.5093 mW / 1.79 dBm
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:25:90:0f:eb:b0 brd ff:ff:ff:ff:ff:ff
3: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether 00:25:90:0f:eb:b1 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether f4:52:14:81:a9:c1 brd ff:ff:ff:ff:ff:ff
5: enp5s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether f4:52:14:81:a9:c2 brd ff:ff:ff:ff:ff:ff
#Same module using SMF
Code:
# ethtool -m enp5s0d1
Identifier : 0x0d (QSFP+)
Extended identifier : 0x80
Extended identifier description : 2.5W max. Power consumption
Extended identifier description : No CDR in TX, No CDR in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 40G Ethernet: 40G Base-LR4
Encoding : 0x05 (64B/66B)
BR, Nominal : 10300Mbps
Rate identifier : 0x00
Length (SMF,km) : 1km
Length (OM3 50um) : 150m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1310.000nm
Laser wavelength tolerance : 6.500nm
Vendor name : Arista Networks
Vendor OUI : 00:1c:73
Vendor PN : QSFP-40G-UNIV
Vendor rev : 20
Vendor SN : XLA100000000
Date code : 17073000
Revision Compliance : SFF-8636 Rev 1.5
Module temperature : 41.20 degrees C / 106.16 degrees F
Module voltage : 3.2606 V
Alarm/warning flags implemented : No
Laser tx bias current (Channel 1) : 52.084 mA
Laser tx bias current (Channel 2) : 60.846 mA
Laser tx bias current (Channel 3) : 51.162 mA
Laser tx bias current (Channel 4) : 45.934 mA
Transmit avg optical power (Channel 1) : 0.8706 mW / -0.60 dBm
Transmit avg optical power (Channel 2) : 0.8931 mW / -0.49 dBm
Transmit avg optical power (Channel 3) : 0.9855 mW / -0.06 dBm
Transmit avg optical power (Channel 4) : 0.8652 mW / -0.63 dBm
Rcvr signal avg optical power(Channel 1) : 0.9197 mW / -0.36 dBm
Rcvr signal avg optical power(Channel 2) : 1.0490 mW / 0.21 dBm
Rcvr signal avg optical power(Channel 3) : 1.1932 mW / 0.77 dBm
Rcvr signal avg optical power(Channel 4) : 0.8918 mW / -0.50 dBmI checked a few pictures on ebay and there seems to be different versions, Made in USA and Made in Malaysia stands out on the labels. Later chips less power would be my theory. Or happy accident.
I bought a CX4 card and is on its way to run the 40G-SR-BD. What about the SX6036 side? I still can not get the Cisco QSFP-40G-SR-BD to light up. Has anyone?
I can confirm both Arista and Cisco BiDi with CX4 Lx and SX6018. The SX6036 will work, assuming you're run the
fae command. The stock firmware only allows high-power modules on certain ports. If you're running my modified firmware then you should be able to run high-power on all ports. But, a good place to start would be the ports that are enabled in the stock firmware.Which cheap LR4 transceivers are you using or are known to work? The original Mellanox MC2210511-LR4 (and MC2210511-IR4) I see are several hundred $/€. Even the clone transceivers from fs.com and qsfptek.com are still over $100.
Cheapest I could find (new) was a Mellanox MC2210511-IR4 clone from qsfptek.com for $84.
I ordered two of these and they work in my SX6036 on the specific ports after setting the "fae cable-stamping-unlock 40g_lr4" command. They seem to draw around 4 Watt while idle (haven't tested under load yet) using a 3 meter fiber patch (both OS2 and OM4).
Cheapest I could find (new) was a Mellanox MC2210511-IR4 clone from qsfptek.com for $84.
I ordered two of these and they work in my SX6036 on the specific ports after setting the "fae cable-stamping-unlock 40g_lr4" command. They seem to draw around 4 Watt while idle (haven't tested under load yet) using a 3 meter fiber patch (both OS2 and OM4).
Can't go wrong with Kaiam QSFP+40G-LR4 Lite for under $10. These are low power (hence 'Lite') so you're not going to be able to reach the building down the street with them.
KAIAM XQX2502 QSFP+40G-LR4 Lite OPTICAL MODULE | eBay
Find many great new & used options and get the best deals for KAIAM XQX2502 QSFP+40G-LR4 Lite OPTICAL MODULE at the best online prices at eBay! Free shipping for many products!
www.ebay.com