Patching Intel X520 EEPROM to unlock all SFP+ transceivers

[seb101]

Started to look at the same fix for the X710 series. https://forums.servethehome.com/ind...ng-any-brand-sfp-modules-on-intel-x710.29040/

Thanks for the inspiration!

 

[ixs]

So thanks for the research, that's highly appreciated as I have a few Cisco UCS machines with the x520 card with Cisco stickers on it, Cisco SFP+ modules installed and the card not liking them.
Bitflipped and all works fine.

To make life easier for others, here's a quick-and-dirty patcher written in python that does the bitstring calculations and the flashing in one go: Intel x520 EEPROM Patcher allows to unlock the x520 network card to work with non-intel branded SFP modules.

It's Linux only, bugfixes and improvements are welcome.

 

[smccloud]

Has anyone tried the FreeBSD fix in pfSense? My x520 card is in my router and I'd like to use Fiber instead of a DAC.

 

[lte]

Interesting case over here: got one Dell X520 (XYT17) that while giving ethtool the exact same output as with many others here (fc) it refuses to accept the bitflip

root@ubuntu-management-station:~# ethtool -e ens161f0 offset 0x58 length 1
Offset Values
------ ------
0x0058: fc

root@ubuntu-management-station:~# ethtool -E ens161f0 magic 0x10fb8086 0x58 value 0xfd
ethtool: bad command line argument(s)

Have read over the ethtool manual now for 3 times, but still can't get my head around this
I suspect that my magic number potentially differs, the python script by @ixs wasn't able to detect any x520 adapters
How did you derive the magic number @NathanA?

Thanks!

 

[ske4za]

Have read over the ethtool manual now for 3 times, but still can't get my head around this
I suspect that my magic number potentially differs, the python script by @ixs wasn't able to detect any x520 adapters
How did you derive the magic number @NathanA?

Thanks!

I think you missed this part:

In the case of the ixgbe driver, it wants you to also supply a "magic" (a.k.a. "secret") value at the time you request the modification in order to confirm the change, and this "magic" value turns out to be the PCI device ID and vendor ID concatenated together, in that order, so you'll want to verify this first with 'lspci', though Intel's PCI vendor ID is 8086 and odds are good that device ID for most X520 models is going to be 10FB:

root@server:~# lspci -nn
[...]
01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

Look at the output of your lspci and see if your device ID and vendor ID are different.

 

[lte]

I think you missed this part:


Look at the output of your lspci and see if your device ID and vendor ID are different.

Good catch, thanks!


04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)
04:00.1 Ethernet controller [0200]: Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)

thus my magic should be 0x154d8086

and it works - thanks a lot @ske4za

 

[ske4za]

You're missing the argument offset in your command

 

[llblasi]

Hello, recently I bought two X520 cards in AliExpress, and they stop working.

The first time I plugged them the unsupported SFP message appeared, but they worked, and I was able to communicate between them. Then I reboot the machines and hell started.

The first thing I tried was adding "allow_unsupported_sfp=1" and it didn't work, then if I read the EEPROM in my card the byte in 0x58 is 0xff saying that it's unlocked, but in every reboot I get the infamous message.

Any suggestions? I'm missing something? Thanks

 

[Falloutboy]

Hello, recently I bought two X520 cards in AliExpress, and they stop working.

The first time I plugged them the unsupported SFP message appeared, but they worked, and I was able to communicate between them. Then I reboot the machines and hell started.

The first thing I tried was adding "allow_unsupported_sfp=1" and it didn't work, then if I read the EEPROM in my card the byte in 0x58 is 0xff saying that it's unlocked, but in every reboot I get the infamous message.

Any suggestions? I'm missing something? Thanks

A little more information would be useful...
What are the specs for the two machines that these cards are running in..
Include Manufacturer/Motherboard Model/CPU and what slots are populated by what cards. OS Type, Version and skew. You are probably all over it but I like to know that all other problem sources have been eliminated first.

Note: I don't have X520 cards and am just trying to be helpful.

 

[llblasi]

A little more information would be useful...
What are the specs for the two machines that these cards are running in..
Include Manufacturer/Motherboard Model/CPU and what slots are populated by what cards. OS Type, Version and skew. You are probably all over it but I like to know that all other problem sources have been eliminated first.

Note: I don't have X520 cards and am just trying to be helpful.

The cables are two JT-DAC-SDP-2 from Aliexpress.

First Machine:
PowerEdge R620 I have tried with proxmox and ubuntu 20.04

Second Machine:
MSI B450a Pro with AMD 2700X with TrueNAS-12.0-RELEASE.

They are connected to a Mikrotik CRS305-1G-4S+.

In ubuntu 20.04 dmesg says allow_unsupported_sfp Enabled

Thanks

Lluis

 

[llblasi]

The cables are two JT-DAC-SDP-2 from Aliexpress.

First Machine:
PowerEdge R620 I have tried with proxmox and ubuntu 20.04

Second Machine:
MSI B450a Pro with AMD 2700X with TrueNAS-12.0-RELEASE.

They are connected to a Mikrotik CRS305-1G-4S+.

In ubuntu 20.04 dmesg says allow_unsupported_sfp Enabled

Thanks

Lluis

Now they are working with intel DAC cables, models XDACBL1M and XDACBL2M .

Thanks for all

Lluís

 

[Tom5051]

fd did the trick here. x520-sr2 was fc and locked down.

 

[ancan]

I've got two x520 dual ports cards in the system, that identifies themself as "Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)".

The cards are enp3s0f0, enp3s0f1, enp129s0f0 and enp129s0f1, and I noticed that when changing the bit on enp3s0f0 also did reflect on enp3s0f1, which to me indicates that the same EEPROM controls both ports. But unfortunately the one bit at offset 0x55 only seem to enable the first port, so I guess each port have their own configuration structure. Has anyone figured out how to enable the second port on these cards as well?

 

[mha]

I've got two x520 dual ports cards in the system, that identifies themself as "Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)".

The cards are enp3s0f0, enp3s0f1, enp129s0f0 and enp129s0f1, and I noticed that when changing the bit on enp3s0f0 also did reflect on enp3s0f1, which to me indicates that the same EEPROM controls both ports. But unfortunately the one bit at offset 0x55 only seem to enable the first port, so I guess each port have their own configuration structure. Has anyone figured out how to enable the second port on these cards as well?

8086:154d here.

As far as I can see the EEPROM is shared. Under 6.1 EEPROM General Map you can see the headers;
Word Address, Used By, Field Name, LAN 0 / 1, Reserved

Looking at the table we see that LAN 0/1 column reads either; Shared Logic, SW, Port 0, Port1. As for 0x29 - 0x2E (where 0x2C is located) it does not indicate port 0 or 1. As such the theory is shared.

Prior to modifying EEPROM;

[   66.538947] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.9.4
[   66.538949] Copyright(c) 1999 - 2020 Intel Corporation.
[   66.539358] ixgbe 0000:05:00.0 0000:05:00.0 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled
[   66.570765] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[   66.570854] ixgbe 0000:05:00.0: Reload the driver after installing a supported module.
[   66.571679] ixgbe 0000:05:00.1 0000:05:00.1 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled
[   66.605079] ixgbe 0000:05:00.1: failed to load because an unsupported SFP+ or QSFP module type was detected.
[   66.605166] ixgbe 0000:05:00.1: Reload the driver after installing a supported module.

EEPROM status:

root@test:~# ethtool -e enp5s0f0 offset 0x58 length 1
Offset          Values
------          ------
0x0058:         fc

Now patching the ixgbe's ixgbe_get_device_caps_generic() with the following:

s32 ixgbe_get_device_caps_generic(struct ixgbe_hw *hw, u16 *device_caps)
{
        DEBUGFUNC("ixgbe_get_device_caps_generic");

        hw->eeprom.ops.read(hw, IXGBE_DEVICE_CAPS, device_caps);
-->     printk("DEVICE_CAPS: %x\n", *device_caps);

        return IXGBE_SUCCESS;
}

On load prints this;

[ 1737.349279] ixgbe 0000:05:00.0 0000:05:00.0 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled
[ 1737.380162] DEVICE_CAPS: fffc
[ 1737.380202] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 1737.380257] ixgbe 0000:05:00.0: Reload the driver after installing a supported module.
[ 1737.381053] ixgbe 0000:05:00.1 0000:05:00.1 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled
[ 1737.414462] DEVICE_CAPS: fffc
[ 1737.414502] ixgbe 0000:05:00.1: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 1737.414576] ixgbe 0000:05:00.1: Reload the driver after installing a supported module.

The 0xfffc is the device capabilities printed for the respective device.

root@test:~/ixgbe-5.9.4/src# ethtool -e enp5s0f0 offset 0x58 length 2
Offset          Values
------          ------
0x0058:         fc ff

Patching the 0xfc to 0xfd;

root@test:~/ixgbe-5.9.4/src# ethtool -E enp5s0f0 magic 0x154d8086 offset 0x58 length 1 value 0xfd
root@test:~/ixgbe-5.9.4/src#

Shows the following (without any SFP override parameter);

[ 2005.697450] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.9.4
[ 2005.697451] Copyright(c) 1999 - 2020 Intel Corporation.
[ 2005.697882] ixgbe 0000:05:00.0 0000:05:00.0 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled
[ 2005.729641] DEVICE_CAPS: fffd
[ 2005.860150] DEVICE_CAPS: fffd
[ 2005.887835] ixgbe 0000:05:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 2005.890334] DEVICE_CAPS: fffd
[ 2005.897481] ixgbe 0000:05:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ 2005.897868] ixgbe 0000:05:00.0 eth0: MAC: 2, PHY: 19, SFP+: 5, PBA No: G73131-008
[ 2005.897870] ixgbe 0000:05:00.0: b4:96:91:17:45:84
[ 2005.897878] ixgbe 0000:05:00.0 eth0: Enabled Features: RxQ: 4 TxQ: 4 FdirHash
[ 2005.898312] ixgbe 0000:05:00.0 enp5s0f0: renamed from eth0
[ 2006.112000] ixgbe 0000:05:00.0 enp5s0f0: Intel(R) 10 Gigabit Network Connection
[ 2006.112304] ixgbe 0000:05:00.1 0000:05:00.1 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled
[ 2006.139540] DEVICE_CAPS: fffd
[ 2006.292076] DEVICE_CAPS: fffd
[ 2006.319839] ixgbe 0000:05:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 2006.322781] DEVICE_CAPS: fffd
[ 2006.329290] ixgbe 0000:05:00.1: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ 2006.329675] ixgbe 0000:05:00.1 eth0: MAC: 2, PHY: 19, SFP+: 6, PBA No: G73131-008
[ 2006.329677] ixgbe 0000:05:00.1: b4:96:91:17:45:86
[ 2006.329679] ixgbe 0000:05:00.1 eth0: Enabled Features: RxQ: 4 TxQ: 4 FdirHash
[ 2006.331881] ixgbe 0000:05:00.1 eth0: Intel(R) 10 Gigabit Network Connection
[ 2006.335675] ixgbe 0000:05:00.1 enp5s0f1: renamed from eth0

As such both enp5s0f0 and enp5s0f1 were affected on my system by the bit flip.

Could you give some output on the scenario you are experiencing?

 

[mha]

Also. Why does nothing complain about the EEPROM checksum at 0x3F for this modification? Aren't we putting some OSes at risk by not correcting the checksum word?

 

[mha]

I am also curious how the capabilities will be represented on my X552;

root@fancy:~# ethtool -e ext0 offset 0x50 length 16
Offset          Values
------          ------
0x0050:         4c 01 86 00 06 00 ff ff ff ff b9 05 00 80 00 07

 

[mha]

Could you give some output on the scenario you are experiencing?

Given that these interpretations are made by the driver maybe the OS driver which you are trying to unlock (suspected not Linux as you have the workaround parameter) is doing something incorrectly in this instance?

 

[mha]

Also playing around a bit more with different types of SFPs I noted that if I plugged in a Cisco FET-10G module it would fail loading even with the module parameter or modified EEPROM.

[ 2200.741520] ixgbe 0000:05:00.0 0000:05:00.0 (uninitialized): ixgbe_check_options: FCoE Offload feature enabled
[ 2200.741521]  - Running inside ixgbe_reset_hw_82599
[ 2200.745835]  - Running inside of ixgbe_identify_phy_generic()
[ 2200.752927]  -- sfp_type did not match, setting ixgbe_phy_sfp_unsupported
[ 2200.752927]  - hw->phy.type was populated with ixgbe_phy_sfp_unsupported in ixgbe_identify_phy_82599()
[ 2200.752928]  - phy->ops.identify(hw) returned IXGBE_ERR_SFP_NOT_SUPPORTED
[ 2200.752928] hw->phy.ops.init(hw) returned IXGBE_ERR_SFP_NOT_SUPPORTED
[ 2200.752931] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 2200.752977] ixgbe 0000:05:00.0: Reload the driver after installing a supported module.

By adding some debug statements I found out that it for some reason thought my SFP+ module was a 1Gbit/s module, and triggered on this PHY initialization code:

/* Verify supported 1G SFP modules */
if (comp_codes_10g == 0 &&
   !(hw->phy.sfp_type == ixgbe_sfp_type_1g_cu_core1 ||
   hw->phy.sfp_type == ixgbe_sfp_type_1g_cu_core0 ||
   hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core0 ||
   hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core1 ||
   hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core0 ||
   hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core1)) {
      printk(" -- sfp_type did not match, setting ixgbe_phy_sfp_unsupported\n");
      hw->phy.type = ixgbe_phy_sfp_unsupported;
      status = IXGBE_ERR_SFP_NOT_SUPPORTED;
      goto out;
}

As such just patching the EEPROM or setting allow_unsupported_sfp will not have an effect if the driver determines that it's not a 10GE module. As such after patching the EEPROM ensure to try out multiple modules. Else Intel's driver will block initialization.

 

[TheNamesJosh_]

Hello!
So I'm having the same issue with my Dell X520-da2 and transceivers that don't work, I have never used Linux before. Do I install Linux and open command prompt with the nic installed and start typing the commands as mentioned? Or is there a dumb down version for a new Linux user to follow. Like a step by step.

 

[Tom5051]

Hello!
So I'm having the same issue with my Dell X520-da2 and transceivers that don't work, I have never used Linux before. Do I install Linux and open command prompt with the nic installed and start typing the commands as mentioned? Or is there a dumb down version for a new Linux user to follow. Like a step by step.

You can download a linux version that can run from a USB stick, use Rufus to make a bootable usb stick from the linux ISO, run commands from the terminal window with the X520 installed.

Create a bootable USB stick with Rufus on Windows | Ubuntu

Ubuntu is an open source software operating system that runs from the desktop, to the cloud, to all your internet connected things.

ubuntu.com