Patching Intel X520 EEPROM to unlock all SFP+ transceivers

[troupbargain]

I've applied this patch to two separate Intel X520's, but I still get the dreaded "failed to load because an unsupported SFP+ or QSFP module type was detected" message. I'm using Cisco coded 10G BiDi SFP+ modules from FS.com. Unfortunately, I don't have any other SFP+ optics to test at the moment (DACs work fine however).

I'm wondering if these optics really are incompatible for some reason, or maybe the ixgbe driver has been patched to ignore the bit in EEPROM? I've searched around but I can't find another example of this patch not working on an Intel brand X520.

I'm also having this issue with BiDi transceivers, even with the flag set.

[   13.030567] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[   13.030568] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[   13.050528] ixgbe 0000:02:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[   13.052026] ixgbe 0000:02:00.0: Reload the driver after installing a supported module.
[   13.052785] ixgbe: probe of 0000:02:00.0 failed with error -95

Are BiDi transceivers not supported?
Edit: They worked once I used the FS Box to change the brand to Intel and the part number to SFP-10GLR-31.

 

[jjakob]

I have a HP 560SFP+ that was already unlocked with 0xFD at EEPROM address 0x58.
However, I still needed to load the ixgbe module with allow_unsupported_sfp=1 to get non-OEM 10Gbase-SR modules to work.
LR modules still didn't work. So I tried setting that to 0xFF and LR modules now work!
Bit 2 (b00000010) seems to enable or disable LR module support. At least on the HP 560SFP+ with the latest firmware from HP (firmware-nic-intel-1.27.30-1.1.x86_64.rpm). I did a cold boot after setting that again and it still works.
I also have some Silicom PE210G2SPI9A-XR that did work with LR modules, they already had that 0xFF at 0x58, so that's where I got the idea that maybe that bit is for LR support. I think it's safe to set that to FF on other Intel 82599ES cards as well.
I don't know why allow_unsupported_sfp=1 is still required after doing that though. Maybe a change in the ixgbe driver.

 

[AntoineNet]

I recently had the need to put together a cheap 10Gbit/s lab, including a server with a NIC. I really only needed a single interface on the server, and I preferred it to be SFP+ rather than twisted-pair for flexibility (I figured I could always stuff a TP module into an SFP+ slot if need be). While researching the various PCIe NIC options available, I arrived at the conclusion that although the X520 might be neither the absolute best performing sub-25gig adapter on the market nor the cheapest 10gig adapter on the market (second-hand or otherwise), it seemed to strike the best balance of performance, stability, wide first-party (and *ongoing*) platform support, and price of any of the ones I looked at.

There was only one potential wrinkle: SFP+ module compatibility. Word on the street was that some of these 82599ES-based cards were known to be "picky". And not necessarily because there was an actual interop issue with modules that were discovered not to work, but because Intel apparently was (at least as the story goes) merely afraid of having to support untested modules that _might_ prove to have interop issues, so they artificially blocked all but the ones on their tested-and-supported list. (*sigh* There always has to be some catch, right? Other old cards of a similar vintage by the likes of Chelsio and Mellanox are long since EOL'd but will supposedly take just about any transceiver you throw at them. With Intel, though, you get widespread platform/OS compatibility with continued vendor support, only to be forced to either play "optics roulette" or go out of your way to source known-compatible modules.) But it was very difficult to ascertain specifically which cards were "problem" cards (all of the 82599s, just the Intel-branded ones, just some of the Intel-branded ones...?).

If you were a Linux user with one of these cards, and discovered that the SFP you wanted to populate your card with was either (seemingly) artificially "blacklisted" or "not whitelisted", you were lucky: the Linux driver actually has a parameter you can pass to it that will override that check. However, other platforms, such as ESXi and Windows, are apparently _not_ so lucky.

But the mere existence of that override parameter in the Linux driver was interesting, and got me thinking. And after taking delivery of a card, playing with it, and doing some additional research and testing, I think I may have discovered a way to instruct any X520 that rejects non-officially-supported transceivers by default to instead allow them, regardless of which driver you are using on your operating system of choice.

My problem is that right now I lack the time and resources to fully test this theory. So instead, I'm going to document my findings here and explain how (I think) this works, and hopefully others here who would like to be able to use their cards with unsupported optics on non-Linux OSes will stumble upon this, be willing to give this a shot, post their findings, and thus either prove this theory right or lay it to rest.

One of the reasons that I can't conclusively test this theory is because it turns out that the card I bought, which is a second-hand, Yottamark-verified genuine Intel X520-DA1, has no issues with any SFP+ module I feed it, even when I don't supply the Linux driver with the allow_unsupported_sfp=1 parameter. So I seemingly lucked out and got one of the "good out-of-box" ones. The thing I was able to accomplish that I feel halfway proves my theory, though, is that I managed to "convert" this unlocked card to a locked version -- a card that refused to work with some of my modules _until_ I supplied that parameter to the driver -- and then convert it back to an unlocked card again. What I don't know for sure is if the mechanism described here is applicable to all X520s that are transceiver-restricted or not...apparently some (the X520-SRx and -LRx, which are basically the -DAx pre-populated with an Intel transceiver) are locked to specific Intel-manufactured SFPs while some of the -DAx models are still restricted but will accept a wider range of approved modules? It's all still a bit unclear to me. (I also haven't been able to test my card on an OS other than Ubuntu Server LTS, either.)

Anyway, on with the show...

----​

The key to "unlocking" an X520 appears to be an undocumented bit within the card EEPROM. Most of the functions of the EEPROM are described in the 82599 datasheet (https://www.intel.com/content/dam/w...asheets/82599-10-gbe-controller-datasheet.pdf), but this one seems to be completely undocumented. The clue came from reading the sources to the Linux ixgbe driver: there is a bitfield in the EEPROM that the driver is checking which the driver source calls IXGBE_DEVICE_CAPS. (DEVICE_CAPS == "device capabilities") So it would seem that the card uses this bitfield to inform the driver about some of its features (which presumably the OEM of each card that uses an 82599 decides for). There are other preprocessor #defines for the various features that are represented by this bitfield that are all named IXGBE_DEVICE_CAPS_*; one of them is IXGBE_DEVICE_CAPS_ALLOW_ANY_SFP, which is the first/least-significant bit.

So it seems reasonable to assume that if one could permanently flip that bit within the EEPROM of an X520 that rejects non-whitelisted SFP+ modules, then you might be able to permanently unlock that card. One of the reasons for assuming this is that the code in the Linux driver (which Intel themselves wrote large portions of) that checks the IXGBE_DEVICE_CAPS_ALLOW_ANY_SFP bit in the EEPROM long predates the addition of the "allow_unsupported_sfp" option in the driver, which wass a relatively late addition (you can see this for yourself by reading through the discussion in the thread at Intel Ethernet Drivers and Utilities / [E1000-devel] [PATCH RFC] ixgbe: Module param "allow_any_sfp" for allowing unsupported SFP+ modules which is interesting reading anyway, if only for people's reactions to the news after they learned what Intel was doing). And if Intel snuck a check of this undocumented field into the Linux driver, it also seems reasonable to assume that they make similar checks in drivers that they have written for other platforms, so the Intel-written drivers for other platforms will *likely* honor the "ALLOW_ANY_SFP" bit in a given card's EEPROM if it is set, even if that driver has no way of allowing the user to override the check.

So what's the exact offset of the bit in question, and how do we change it?

This post from the Intel Linux driver development mailing list gives us some clues: Intel Ethernet Drivers and Utilities / [E1000-devel] ixgbe fcoe disabled by eeprom

As mentioned, the offset of IXGBE_DEVICE_CAPS is indeed 0x2C: Linux source code: drivers/net/ethernet/intel/ixgbe/ixgbe_type.h (v5.1.5) - Bootlin

The device_caps field from the EEPROM is read in by this function: Linux source code: drivers/net/ethernet/intel/ixgbe/ixgbe_common.c (v5.1.5) - Bootlin

From this, we can see that EEPROM data on the Intel NICs consists of 16-bit words. It turns out that they are words stored in little-endian format (least-significant byte first).

On Linux, we can usually use the 'ethtool' utility to dump out the EEPROM of a supported ethernet card either in whole or in part...here are the first 96 bytes of my X520-DA1:

root@server:~# ethtool -e enp1s0​

Offset Values​

------ ------​

0x0000: 60 07 00 00 00 00 40 00 6d 00 fd 00 8d 01 a3 01​

0x0010: a9 01 af 01 b7 01 bf 01 c7 01 cf 01 09 02 38 05​

0x0020: ff f7 ff ff ff ff ff ff ff ff fa fa 10 0e 48 02​

0x0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff​

0x0040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff​

0x0050: 0a 0e 93 00 ff ff bc 0c fd ff 01 00 11 5e ff ff​

[...]​

Note that the offsets displayed here are are byte offsets, not word offsets. So to find the *byte* offset that needs to be changed, we take 0x2C and multiply it by 2 to get 0x58. At word 0x2C (byte 0x58), as you can see above, we have word value (if displayed in big-endian order) 0xFFFD.

Since the word is in fact stored in little-endian order, the byte that we're really interested in is the one immediately at offset 0x58, with value 0xFD. If we represent that as bits (11111101), we can see that bit 1 (the last bit, the one in the "1s place"), which is the bit that supposedly controls whether the SFP+ slot will allow for "any" module or not, is already "on". This is exactly the state that I found my card in when it arrived.

Now, if I wanted to *lock* the SFP+ slot so that it only allows you to use the transceivers that Intel officially *supports*, I'd have to write a new byte value to offset 0x58 which would change bit 1 to "off". If bit 1 is the only bit of that byte that I change, the hex value of that would be 0xFC. It turns out that I can, in fact, also modify the EEPROM using the 'ethtool' utility (assuming the driver supports it, which in the case of the ixgbe driver, it does...in fact, the driver will even recompute the EEPROM checksum for you after making the changes you ask, and update it for you!). In the case of the ixgbe driver, it wants you to also supply a "magic" (a.k.a. "secret") value at the time you request the modification in order to confirm the change, and this "magic" value turns out to be the PCI device ID and vendor ID concatenated together, in that order, so you'll want to verify this first with 'lspci', though Intel's PCI vendor ID is 8086 and odds are good that device ID for most X520 models is going to be 10FB:

root@server:~# lspci -nn​

[...]​

01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)​

Now we know all that we need to in order to make this change (which will need to be done as root, naturally), so first we will modify the EEPROM with the -E parameter, and then read back the EEPROM again with -e in order to verify that the change took:

root@server:~# ethtool -E enp1s0 magic 0x10fb8086 offset 0x58 value 0xfc​

root@server:~# ethtool -e enp1s0​

Offset Values​

------ ------​

0x0000: 60 07 00 00 00 00 40 00 6d 00 fd 00 8d 01 a3 01​

0x0010: a9 01 af 01 b7 01 bf 01 c7 01 cf 01 09 02 38 05​

0x0020: ff f7 ff ff ff ff ff ff ff ff fa fa 10 0e 48 02​

0x0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff​

0x0040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff​

0x0050: 0a 0e 93 00 ff ff bc 0c fc ff 01 00 11 5e ff ff​

[...]​

...and as you can see above, byte 0x58 now shows a value of 0xFC. Success. If I reboot now with an "unauthorized" transceiver still in the slot, I will no longer have network connectivity and I will see the infamous "failed to load because an unsupported SFP+ module type was detected" in my dmesg output.

Re-unlocking it (in my case) would be as simple as writing 0xFD back to the same offset.

In order to unlock your locked card so that it (perhaps? hopefully? maybe?) will work under any OS/driver with any reasonable SFP+ module, you would just need to follow the same basic steps: first, read back byte 0x58 from your card's EEPROM:

root@server:~# ethtool -e enp1s0 offset 0x58 length 1​

Offset Values​

------ ------​

0x0058: fe​

...then second, do the math on the existing value to change the first bit from 0 to 1, and finally, write the new value back to the card's EEPROM:

root@server:~# ethtool -E enp1s0 magic 0x10fb8086 offset 0x58 value 0xff​

I would advise you to NOT simply re-use the "known-working" values for this byte shown here (e.g., 0xFD or 0xFF) on your card...since you don't know what behavior in the card's firmware the other bits change, you shouldn't change the value of any bit *other* than the first on your particular card. Also, don't simply assume that you should add 0x01 to the existing value to achieve what you want. Let's say bit 1 was *already* set "on", and byte 0x58 has, say, a value of 0xEF, and your transceiver isn't working for some other reason: if you make an incorrect assumption and change that to 0xF0 then you have just (unintentionally) changed a whole range of bits, and at the same time turned the first bit "off", too. Actually read the current byte back and do the math correctly.

If you aren't planning to use the card in a Linux box, and don't even have a Linux host to throw your card in temporarily in order to make the modification, I'm not (yet) sure how you would accomplish the same thing in Windows, but if your card is installed in an ESXi host, it appears that the local/remote "Tech Support" ESXi shell also has 'ethtool' and that it works exactly the same way as the Linux one does, as far as I have been able to tell. Other open-source *nix-like or Unix-derived OSes such as the *BSDs likely have 'ethtool' as well or something similar. (Also, macOS X apparently has 'ethtool', too, for what it's worth.)

----​

As far as I can tell, the only kinds of modules that a "locked" X520 blocks are unapproved, active SFP+ modules, while virtually all SFP modules and passive SFP+ DACs from any vendor will work even in a card with DEVICE_CAPS bit 1 set to 0.

One thing I'm hoping to establish through both my experimentation and yours is exactly which Intel parts have this bit flipped "off", and whether flipping this bit "on" is a complete unlock in all circumstances or whether other things factor in. /u/eruffini posted in this Reddit thread (Intel X520-DA2 10G NIC ESXi 6u2 woes. : vmware) that he'd been told that more recently manufactured X520s are the ones with the problem. On the other hand, we also know that X520-SR1/2 and X520-LR1/2 came with Intel optics and are intended to only be used with those. Is this bit in the EEPROM what is used to lock those models down? Does setting this bit to "on" essentially convert an X520-SR1 to an X520-DA1, for example? Or are there X520-DA1/2 cards that are also locked down from the factory? Is the enforcement to use Intel optics in the SR/LR cards done via the same mechanism or a totally separate one? Are 82599-based cards manufactured by other OEMs known to restrict SFP+s in the same way, and using the same mechanism? Etc.

The other scenario that I'm hoping doesn't end up being the case -- but certainly is within the realm of possibility -- is that it seems feasible that Intel *could* have written and released a driver that ignores this bit in the EEPROM completely and always enforces SFP+ restrictions regardless. If the driver on the host has enough influence over the card's firmware that the SFP+ check can be overridden *despite* that bit (e.g. "allow_unsupported_sfp" on Linux), it is certainly plausible that this could cut both ways...

Definitely looking forward to everybody's feedback and reports, & good luck!

-- Nathan

Hey! Does this also work with a Dell X520-DA2?
(Dell 0XYT17 0942V6 Intel X520-DA2)

Thanks in advance

 

[senor-hobbes]

Hi All,

Came across this fix to allow 3rd party modules on Intel X520s. I went through the process for each port on my two port card but only one port seems to link and establish a connection. Both 0x58 bits were "fc" so I changed them to "fd". Port 1 works and immediately finds a link/coannection but port 0 doesnt. I dont have any first party modules to test but i tried several of my other modules which all work in port 1 but not port 0.

Any ideas on what might be going on here? I see the two ports of the card in device manager (windows) with no errors.

 

[domination2700]

Can this be done on windows

 

[blunden]

Can this be done on windows

Not in the way described, no. I haven't come across a Windows tool with the same feature set as ethtool.

The easiest way is probably to simply live boot a user friendly Linux distro like Ubuntu or Mint and perform the unlock that way. You don't actually need to install a different OS, just boot one to perform the unlock.

 

[domination2700]

Would it be possible to do this for a NAS

 

[blunden]

Would it be possible to do this for a NAS

Is the NIC in the NAS even vendor locked?

Is it a custom built NAS or an appliance of some kind?

 

[douteiful]

I have 2 Intel x520-da1 which both contain FD in 0x58. They both refuse to work in windows or pfsense while my ODI DFP-34X-2C2 is inserted.
It works just fine in Linux OSes such as unraid or with a live image of debian.

Hi, I have the same exact SFP module and a card with the same chipset (Intel X520-da2) but the module doesn't work for me even on Linux with allow_unsupported_sfp=1. What's your environment and did you do anything else to make it work?

 

[ppptran]

I have a HP 560SFP+ that was already unlocked with 0xFD at EEPROM address 0x58.
However, I still needed to load the ixgbe module with allow_unsupported_sfp=1 to get non-OEM 10Gbase-SR modules to work.
LR modules still didn't work. So I tried setting that to 0xFF and LR modules now work!
Bit 2 (b00000010) seems to enable or disable LR module support. At least on the HP 560SFP+ with the latest firmware from HP (firmware-nic-intel-1.27.30-1.1.x86_64.rpm). I did a cold boot after setting that again and it still works.
I also have some Silicom PE210G2SPI9A-XR that did work with LR modules, they already had that 0xFF at 0x58, so that's where I got the idea that maybe that bit is for LR support. I think it's safe to set that to FF on other Intel 82599ES cards as well.
I don't know why allow_unsupported_sfp=1 is still required after doing that though. Maybe a change in the ixgbe driver.

I saw that you update this on Jan 2025.

Could please tell me what is your machines OS . ex, promox ?...

I have the cwwk n100 with 2sfp+ unit.
This card is 82599ES.

And in the eeproom. And At 0x58, there is already an FD.

I use promox 8.3.4 as of April 6. I already set allow unsupported sfp = 1.
I also saw that being called in the dmesg.

But the card as of right now with FD at 0x58, allow.igxbe unsupported module=1 .

It still show the unwelcome message. Unable to load , unsupport sfp module.

I use BIDI 1.25G 1310/1550 from my ISP.

I try the SFP to RJ45 (1G module), unbranded Uni something, it does work. It pickup link.

But as soon as i insert BIDI module, it pop up the unwelcome message in dmesg.

So, i am thinking, should i replace FD with FF at 0x58 as same as your card.

Updated: i tried ff. Regular sfp ( not plus) does not load the igxbe in kernel.

I dont know what happened with regular 1G Bidi sfp ( not plus )

////////////////////////////////////////////////////////////////////////////////////////////////////////

UPdated: for anyone want to use 82599ES with Bidi from ISP

I got this card work with Bidi 1G SFP. The sfp module i use is Huawei Bidi 1310/1550 SM 20km. ( On the other end is from my ISP which has 1550/1330 on their end)

 

[ruvanillabear]

Hello everybody.
I'm try flash HP 560sfp+ in to original Intel X520-DA2 and broke firmware (want activate iSCSI and FCoE BootROM).
Card initialisation don't finish on boot screen and system stuck.
Can enybody give full eeprom image from original card for flashing via SPI programmer (just full dump from ethtool)?

 

[MPServers]

You're in luck (hopefully). I got a CH341 a while back and when testing it out I was grabbing dumps of various things, including an Intel X520-DA2 (pulled from a Dell Poweredge if it matters... I have another X520-DA2 that I pulled from an old Netapp and though similar, they did have different dumps, possibly/probably different firmware versions).

No idea what version firmware is on here or if this is one I'd already patched to allow other SFPs. I'm going to say probably not. And it'll have whatever serial # and MAC addresses. If you're able to get that info from yours, I don't know if the ethtool lets you update those after the fact to reset them back to your own? I haven't messed with it to that extent.

I have a bunch of these cards so I can't recall which one I grabbed it from but if this didn't work for whatever reason, I could try popping one of these into something to get the details, and then dump that particular SPI.

It's a 512KB chip (don't recall the part # but I could look it up, I'm guessing all of those X520-DA2 boards are the same).

 

[ruvanillabear]

@MPServers


Thanks, but the chip size is not the same as on the HP card (the EEPROM chip size is 64KB).

I'm looking for a dump of the original Intel X520-DA2 card, not HP, IBM, Dell, etc., because those versions may have certain functions blocked.
When flashing a combo image, only PXE and UEFI are available.

Maybe someone can explain how BootUtil checks and determines the available ROMs for the card. Is it based on the VID/DID or specific bits in the EEPROM?

 

[MPServers]

Oh, well, I can't help you there, all I have are the Dell variety (Dell part # 0942V6).

If it helps anyway, the BIN from the chip (it's a 25P40VP) although it's a 512KB chip, it has #39A49 bytes (236,105) and then the rest is just FF.

I'm not sure how that could get squished into just 64 KB, just eyeballing the hex.

Now, the Netapp version of the X520 DA2, I'd dumped that a ways back as well and it's also a 512KB SPI, but in that case, it has data up to #F000 and then the rest is FF. Even then, the data really goes just to #E923, zeroes up to #EFFF, and then those FFs. So that would fit under 64KB if you just truncated at #FFFF and called it good.

The first part of that dump reads like this:
Copyright (C) 1997-2009, Intel Corporation
Initializing Intel(R) Boot Agent XE v2.1.40
IBA XE Slot 0000 v2140
PXE 2.1 Build 086 (WfM 2.0)

Whereas that same section of the Dell branded version, with up to date firmware (22.5.7, updated using the Dell updater) is:
Copyright (C) 1997-2023, Intel Corporation
Initializing Intel(R) Boot Agent XE v2.4.51
IBA XE Slot 0000 v2451
PXE 2.1 Build 092 (WfM 2.0)

 

[MPServers]

Oh, while I was checking, I pulled the flash from a couple of those Dell versions and compared them. They're identical so kind of like I suspected, the serial and MAC address info isn't stored in the SPI. I wasn't sure beforehand but that confirms it. That's good to know.

 

[ruvanillabear]

On the board, there are two EEPROM chips.
The first one has a capacity of 64 KB and stores the MAC address, Vendor ID (VID), Device ID (DID), and other configuration data.
The second EEPROM contains the Option ROM and additional firmware modules.

The first EEPROM can be dumped using ethtool in linux.
The second one can be dumped using Intel's bootutil tool (via the -SAVEIMAGE option).
Both EEPROMs can also be directly read using an SPI programmer.

 

[Dracarris]

Hey there,

after having tried everything, I hope I can get some help here.. Situation is X520-DA2 with an sf.com BiDi LR (10km) SFP+ module.

I patched the EEPROM, I added the allow_unsupported_sfp=1,1 option to the driver module - no luck. Every time the same message "unsupported SFP+ module type was detected". When I manually reload the driver with the module inserted, it even seems to crash, pooping out a stack trace..

So, is the only option to get these modules to work really reflashing with an Intel brand code? I bought the X520 due to its compatibility with True NAS, but if I would've known about this lockdown shit show, I would've picked up a Mellanox, they seem to be happy no matter which SFP+ you throw at it..

 

[Quantamix]

my apologies posted in the wrong thread.

 

[Dracarris]

Are BiDi transceivers not supported?
Edit: They worked once I used the FS Box to change the brand to Intel and the part number to SFP-10GLR-31.

Thanks mate, this actually worked - wild! Still, the X520 refuses (with any SFP+ I own) to connect to my QNAP switch, while my noname-China one works just fine. With the QNAP, the status LEDs on the X520 turn super dim and flash very fast a few seconds after connecting the fibre/inserting the SFP. I have absolutely none of any of these issues with my Mellanox ConnectX 4, boi these Intel cards really are special snowflakes, sigh.

 

[pconwell]

For what it's worth, I just tried this on a knockoff X520-DA1 clone and it worked! I was seeing inconsistent errors that didn't really match with the errors others are seeing here (for example, I was NOT seeing the "failed to load because an unsupported SFP+ module" - wasn't really getting any errors actually, I just couldn't get the card to see any SFP modules).

So I applied this hack, rebooted the server, and it started working as expected. Quick recap of what I did:

$sudo ethtool -e YOUR_NIC offset 0x58 length 1

add 1 to the result (e.g fd -> fe)

$sudo lspci -nn

find the device ID, mine was [8086:1557]. Flip the quads, like this -> 0x15578086

$sudo ethtool -E YOUR_NIC magic 0x15578086 offset 0x58 value 0xfe length 1

$sudo reboot

$sudo ifup YOUR_NIC