SOLVED Mellanox SX6036 + LR4 Transceivers does not work

[NablaSquaredG]

Hey,

So I was trying to pair one of my SX6036 with LR4 transceivers - Never done this before

I know that LR4 transceivers only work in Ports 1, 3, 33, 35, so I used those ports

Mellanox switch systems offer high power transceiver (e.g. LR4) support in the following ports:
•SX6036 – ports 1, 3, 33, 35
•SX6012/SX6710/SX6720 – all ports
•SB7700 – all ports
•TX6000/TX6100 – all uplink ports
If a high power transceiver (e.g. LR4) is inserted to a port that does not support it, the link does
not go up, and the following warning message is displayed: “Warning: High power transceiver is
not supported” when the command “show interfaces ib” is run.

But the transceivers do not link up. It does not matter whether the Port is in IB or Eth mode (Switch is in VPI mode)

Firmware is 3.6.8012

Any ideas?

 

[NablaSquaredG]

Solution:

Use the magic 0x66 0x61 0x65 manufacturer command to remove the cable-stamping-lock

 

[Stephan]

What's with all the LR4 / LR4 Lite stuff on here recently anyway? Everyone built a tree house 1 km away during pandemic downtime that really could use 40 Gbps ethernet to the house?

 

[NablaSquaredG]

What's with all the LR4 / LR4 Lite stuff on here recently anyway? Everyone built a tree house 1 km away during pandemic downtime that really could use 40 Gbps ethernet to the house?

MTP cables got incredibly expensive... So Single Mode LC Fibre cables it it!

 

[BoredSysadmin]

What's with all the LR4 / LR4 Lite stuff on here recently anyway? Everyone built a tree house 1 km away during pandemic downtime that really could use 40 Gbps ethernet to the house?

Community or DIY ISP perhaps?

 

[Stephan]

Would definitly like more community ISP videos on youtube, using Aristas or Mellanoxes from ebay. ;-)

 

[BoredSysadmin]

Would definitly like more community ISP videos on youtube, using Aristas or Mellanoxes from ebay. ;-)

 

[llowrey]

Solution:

Use the magic 0x66 0x61 0x65 manufacturer command to remove the cable-stamping-lock

Although this thread is for the SX6036, I would like to mention that the SX6018 does not support high-power transceivers (eg LR4) on any port but can be modified to do so. The magic cable stamping command only works on ports that have high power enabled. Theoretically, the same mod could be used to enable LR4 transceivers on more than the 4 ports enabled by default on the SX6036. The asic firmware contains settings for each port and just needs module_power_level_supported to be set to 5 to enable high power transceivers.

I couldn't find the firmware code necessary to generate a bin with a modified config ini so I resorted to patching the bin directly. PM if you'd like a copy of the modded bin for the SX6036 or SX6018.

 

[veegee]

Solution:

Use the magic 0x66 0x61 0x65 manufacturer command to remove the cable-stamping-lock

Can you please explain how to do this?

 

[veegee]

Figured it out, feel free to message me if you need help.

 

[saratoga]

For anyone who is puzzled by this, the command only works on newer firmware versions. 3.4.x have a lock command but no unlock, at least for me. Once I updated to 3.6.3 it appeared.

Edit: Ran through the sequence on a second sx6036. Steps are:

Update to at least 3.5.1006.

Then in the console:

enable
configure terminal
fae
cable-stamping-unlock 40g_lr4
exit
write memory

After that my Intel LR4 modules are recognized in ports 1, 3, 33 and 35.

 

[unphased]

Thanks for the steps. I have some CWDM4 transceivers I'm testing on my SX6036 and they were showing the "high power transceiver is not supported" when opening the port in the webui. When I plug to port 33 it no longer shows this error! but does not come online.

I have applied the steps and waiting for switch to reboot now.

My hope is that the "40g_lr4" is more of a notion of high power, aka LR4 transceivers are high power, rather than just about LR4 (of which I have no such equipment... yet) and so, the later 100G CWDM4 technology is also high power but hopefully still backward compatible.

 

[Prophes0r]

What's with all the LR4 / LR4 Lite stuff on here recently anyway? Everyone built a tree house 1 km away during pandemic downtime that really could use 40 Gbps ethernet to the house?

Dirt cheap transceivers that should work without attenuators?
Why use MPO/MTP fiber when you can get LC/UPC fiber and $3.50 100G LR4 Lite transceivers?

@llowrey Are you saying you modded the SX6036 switch ACIC firmware to accept "high power" transceivers on all ports when used in conjunction with the `cable-stamping-unlock 40g_lr4` fae commands?

If so, I am very interested, since I'm finishing up my new networking closet and I want 6 machines using the above-mentioned transceivers.

 

[unphased]

I don't need this yet, but I can see it happening for sure, given how well my $8 kaiam LR4L transceivers work from my brief testing, being able to run these on every port would be amazing! Please provide more info.

 

[llowrey]

@llowrey Are you saying you modded the SX6036 switch ACIC firmware to accept "high power" transceivers on all ports when used in conjunction with the `cable-stamping-unlock 40g_lr4` fae commands?

Yes. The modded firmware removes the power restriction on all ports. That, combined with the above command enables LR4 optics on all ports. I don't want to publish the firmware publicly since there is some risk so feel free to DM me and I'll send you the info. More than a dozen people have used this firmware without incident and I've been running it for years so the risk is low, but it's not zero.

 

[BeTeP]

I couldn't find the firmware code necessary to generate a bin with a modified config ini so I resorted to patching the bin directly

I don't know how I missed this thread last year but the whole reason why I wrote my GitHub - BeTeP-STH/mft-scripts: Mellanox firmware files and MFT related scripts was to solve this specific problem with my switch 4 years ago.

 

[Bad Apple]

Yes. The modded firmware removes the power restriction on all ports. That, combined with the above command enables LR4 optics on all ports. I don't want to publish the firmware publicly since there is some risk so feel free to DM me and I'll send you the info. More than a dozen people have used this firmware without incident and I've been running it for years so the risk is low, but it's not zero.

Hi, I think I need this firmware to update my sx6036 switch, can you provide me with one? Thank you so much

 

[ClumsyAssassin]

Yes. The modded firmware removes the power restriction on all ports. That, combined with the above command enables LR4 optics on all ports. I don't want to publish the firmware publicly since there is some risk so feel free to DM me and I'll send you the info. More than a dozen people have used this firmware without incident and I've been running it for years so the risk is low, but it's not zero.

Hi, this is my first post here. After reading up a lot on Mellanox switches via the forums (thanks a lot to all involved), I've ended up with a SX6036 switch. I've performed the lr4 unlock, so would like to try out the firmware.

 

[marg]

I am looking for some help here.

I have a stock SX6036 running Mellanox/Nvidia published image " PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc " on the device.

I have tried to install the LR4 modded firmware suggested here but it results in a kernel panic -- see below

[admin@switch-a7a7a8 ~]# flint -override_cache_replacement  -d /dev/mst/mt51000_pciconf0 q

-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
Image type:          FS2
FW Version:          9.4.5110
FW Release Date:     12.2.2019
Device ID:           51000
Description:         Node             Sys image
GUIDs:               e41d2d03003dba00 e41d2d03003dba00
Description:         Base             Switch
MACs:                    e41d2d3dba00     e41d2d3dba60
VSD:                 n/a
PSID:                MT_1010310020



[admin@switch-a7a7a8 ~]# flint --override_cache_replacement --allow_psid_change -d /dev/mst/mt51000_pciconf0 -i "./MT_1010310020.bin" b

-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.

    Current FW version on flash:  9.4.5110
    New FW version:               9.4.5110

    Note: The new FW version is the same as the current FW version on flash.

 Do you want to continue ? (y/n) [n] : y

Burning FS2 FW image wOops: Machine check, sig: 7 [#1]
PREEMPT mlnx460ex
Modules linked in: af_packet mst_pciconf(O) mst_pci(O) oops_dump_reg(O) sx_pra(O) int_hndl_main(O) act_police cls_u32 sch_ingress ip6table_filter ip6_tables ib_cm(O) ib_sa(O) inet_lro ib_uverbs(O) ib_umad(O) sx_ib(O) ib_mad(O) ib_core(O) ib_addr(O) ipv6 compat(O) sx_bfd(O) sx_netdev(O) sx_core(O) 8021q switchx() sx_glue_if(O) i2c_mux_pca954x i2c_mux ehci_hcd usbcore lm87 hwmon_vid hwmon usb_common unix
CPU: 0 PID: 6132 Comm: flint_oem Tainted: G           O 3.10.94-MELLANOXuni-m460ex PPC_M460EX
task: eae08ea0 ti: efff6000 task.ti: eb7b6000
NIP: c016079c LR: c0017454 CTR: c0017438
REGS: efff7f10 TRAP: 0214   Tainted: G           O  (3.10.94-MELLANOXuni-m460ex)
MSR: 00021000 <CE,ME>  CR: 44244448  XER: 00000003

GPR00: c0009f50 efff3fd0 eae08ea0 c0386e08 0000001e 00000000 c0017438 01400001
GPR08: 00000001 ef400001 c0386e08 eb7b7d40 eae090a0 10754e48 00000000 00000000
GPR16: 100e8570 100e8280 100e8588 1074d24d bf9fd538 bf9fd438 482a8608 00000000
GPR24: 00050215 104fb180 eb7b6040 00000000 00000000 0000001e c0384060 ef804900
NIP [c016079c] radix_tree_lookup_element+0x14/0xa4
LR [c0017454] uic_irq_cascade+0x1c/0x144
Call Trace(eae08ea0): name=flint_oem, state=0
[efff3fd0] [c0017540] uic_irq_cascade+0x108/0x144 (unreliable)
[efff3ff0] [c0009f50] call_handle_irq+0x18/0x28
[eb7b7d50] [c000333c] do_IRQ+0xc4/0x118
[eb7b7d80] [c000b538] ret_from_except+0x0/0x18
--- Exception: 501 at pci_bus_write_config_dword+0x68/0x94
    LR = pci_bus_write_config_dword+0x60/0x94
[eb7b7e60] [fc632680] ioctl.isra.4+0x208/0x48c [mst_pciconf]
[eb7b7ea0] [c00bd818] vfs_ioctl+0x38/0x58
[eb7b7eb0] [c00be3ec] do_vfs_ioctl+0x59c/0x6d8
[eb7b7f10] [c00be568] SyS_ioctl+0x40/0x68
[eb7b7f40] [c000ae74] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0xfe00638
    LR = 0xfe97320
Instruction dump:
3903000c 8143000c 912a0004 91490000 9103000c 91030010 4e800020 81230008
7c6a1b78 2f890000 419e0090 71280001 <40820020> 2f840000 409e0080 2f850000
---[ end trace c3e441706158c72d ]---

Kernel panic - not syncing: Fatal exception in interrupt
Rebooting in 1 seconds..

Can somebody please tell me what I could be doing wrong?

If you need more details please let me know.

Thanks in advance for your time and attention.

 

[tarunspandit]

Yes. The modded firmware removes the power restriction on all ports. That, combined with the above command enables LR4 optics on all ports. I don't want to publish the firmware publicly since there is some risk so feel free to DM me and I'll send you the info. More than a dozen people have used this firmware without incident and I've been running it for years so the risk is low, but it's not zero.

Hi, would anyone be able to provide me the modded firmware as well?