Patching Intel X520 EEPROM to unlock all SFP+ transceivers

ZFSZealot

New Member
Aug 16, 2021
21
3
3
Does anyone know what the second least significant bit does at 0x53 on the x520? i.e. I'm wondering the difference between 0xFC/0xFD and 0xFE/0xFF is. Obviously the least significant bit is the "Allow unapproved SFP+", but what's the one next to it?

I ran ethtool, saw either 0xFC or 0xFE on a Dell 0C63DV and in my excessive enthusiasm just slammed 0xFF in there, despite the clear warning in the OP (yes, I'm stupid). A reboot didn't bring up the link and I realized I wasn't sure if it was FC or FE so I put 0xFD in there, still didn't work but it wound up being a port speed problem on the switch, once that was corrected everything was fine with 0xFD.

Now I'm second guessing myself if it was originally 0xFC or 0xFE and what if anything is going to be affected if I flipped that second bit as well. The links are up but I haven't done much testing with actual traffic yet.

EDIT: Oh, and by the way, I did this using ethtool on ESXi 6.5, the "ethtool" instructions in the OP worked perfectly. Just had to use "vmnic0" as the interface name. Reading the value back from "vmnic1" once I wrote to "vmnic0" also showed the value I wrote, obviously it's just one ROM for both interfaces.

EDIT 2: Spoke too soon. This does NOT seem to work with ESXi 6.5 build 17477841 in a Dell Poweredge r620. I think when I first saw it work I had a connection up with DAC cables and then hot plugging the unsupported SFP+ modules in worked for some reason. But after a reboot it won't load the driver if it sees the module, and even hot plugging after another power off reboot seems to be hit and miss whether or not it will accept the module or just shut down the port. I checked this with the 0C63DV LOM as well as a separate x520-da2 card, both were set to 0xfd, and I did check after a power off reboot to make sure that the change had persisted. Take this with a grain of salt, but it did not work for me. Firmware on both cards is Dell 19.5.12. I'm just going to use DAC cables but hope this is useful for someone.
 
Last edited:

Hotpaganini

New Member
Sep 17, 2021
1
0
1
I have very strange error... When I try to patch my card. Command
Bash:
ethtool -E ens106f0 magic 0x10FB8086 offset 0x58 value 0xfd
cause reply: offset & length out of bounds
Please, help! I need this patch!!
 

ZFSZealot

New Member
Aug 16, 2021
21
3
3
I have very strange error... When I try to patch my card. Command
Bash:
ethtool -E ens106f0 magic 0x10FB8086 offset 0x58 value 0xfd
cause reply: offset & length out of bounds
Please, help! I need this patch!!
Can you provide more information about what OS you're trying this from and exactly which card you have?
 

Eric Stuyvesant

New Member
Nov 2, 2018
3
4
3
Can you provide more information about what OS you're trying this from and exactly which card you have?
I am not the original poster, but can confirm that I get the same error message under xubuntu 21.10 with kernel 5.15.2.

My card:
lshw -C network -short
H/W path Device Class Description
====================================================================
[...]
/0/100/3.2/0 enp17s0f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/100/3.2/0.1 enp17s0f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection

lspci -nn | egrep 82599ES
75:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
75:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

ethtool -i enp17s0f0
[...]
version: 5.1.0-k
firmware-version 0x00015e0b
[...]


I have also now replicated the bug on the Dell version of the card, the 0XYT17, which has a PCI ID of 8086:154d.

It seemingly works successfully under an xubuntu 20.04.1 livecd with the same hardware (I also note ubuntu assigns a shorter device id under 20.04.1). This is true with both hardware varieties.

I am surmising the mismatch between the installed version of ethtool under 21.10 (ethtool 5.9) and the running kernel (5.15.2 from the kernel PPA in my case) is the root cause of the problem. Something about the writing-eeprom API must have changed between kernel 5.9 (which ethtool was compiled against) and 5.15.
 
Last edited:
  • Like
Reactions: ZFSZealot

Eric Stuyvesant

New Member
Nov 2, 2018
3
4
3
I am surmising the mismatch between the installed version of ethtool under 21.10 (ethtool 5.9) and the running kernel (5.15.2 from the kernel PPA in my case) is the root cause of the problem. Something about the writing-eeprom API must have changed between kernel 5.9 (which ethtool was compiled against) and 5.15.
Apparently old ethtool versus modern kernel ABI is not the root cause. I downloaded the daily XUbuntu 22.04 and tried it in a liveboot environment, and it still fails. Today's liveboot has an ethtool version of 5.16 and a kernel of 5.15.17.

Interesting.
 
  • Like
Reactions: ZFSZealot

Eric Stuyvesant

New Member
Nov 2, 2018
3
4
3
I have very strange error... When I try to patch my card. Command
Bash:
ethtool -E ens106f0 magic 0x10FB8086 offset 0x58 value 0xfd
cause reply: offset & length out of bounds
Please, help! I need this patch!!
You might also want to review my prior couple posts for the backstory, but here's the resolution:

As I was filing a bugreport with ubuntu against their ethtool package, I decided to RTFM first. I noted that the ethtool manpage lists several valid parameters of the -E option, and length is the only parameter which the previously mentioned command left to whatever ethtool defaults to. Turns out, specifying length 1 explicitly in the ethtool -E command line resolves the error and is successful.

Therefore, instead of:
Code:
ethtool -E ens106f0 magic 0x10FB8086 offset 0x58 value 0xfd
(which fails)

do this:
Code:
ethtool -E ens106f0 magic 0x10FB8086 offset 0x58 value 0xfd length 1
Verified under XUbuntu 22.04 daily and 21.10. The addition of length 1 also works as before under 20.04.1.
 

DavidRa

Infrastructure Architect
Aug 3, 2015
303
137
43
Central Coast of NSW
www.pdconsec.net
I can confirm the new instructions for current ethtool work as Eric describes.
You might also want to review my prior couple posts for the backstory, but here's the resolution:

do this:
Code:
ethtool -E ens106f0 magic 0x10FB8086 offset 0x58 value 0xfd length 1
However, having applied 0xFD to both my X520 cards, I no longer get the unsupported SFP message (on Windows so I can't use the option for ixgbe) but I also do not get a successful link. I'm also interested therefore in other possible bits - as well as the specific drivers people have used on Windows for 10Gb unsupported SFPs.
 
  • Like
Reactions: ZFSZealot

ZFSZealot

New Member
Aug 16, 2021
21
3
3
I can confirm the new instructions for current ethtool work as Eric describes.

However, having applied 0xFD to both my X520 cards, I no longer get the unsupported SFP message (on Windows so I can't use the option for ixgbe) but I also do not get a successful link. I'm also interested therefore in other possible bits - as well as the specific drivers people have used on Windows for 10Gb unsupported SFPs.
Not sure if this helps, this seems to work with the Fujitsu version of the card I have in my one of my workstations:

1645119661234.png1645119676157.png

(Yes, it's still Windows 7 because reasons).
 

nscannell

New Member
May 19, 2022
1
0
1
Oops... Just wiped the EEPROM.

Very tired and forgot to add the offset so I just wrote "value 0xfd" out to it.

Driver won't load now although I have another working one and an EEPROM dump

Does anyone know how to restore the EEPROM via sysfs through the PCI bus?

I want to manually bind the PCI device to the ixgbe driver module then restore the EEPROM but I can't bind the PCI device.

Code:
echo -n "0000:01:00.0" > /sys/bus/pci/drivers/ixgbe/bind
-bash: echo: write error: No such device
 
Last edited:

ixs

New Member
Aug 14, 2020
3
1
3
Oops... Just wiped the EEPROM.

Very tired and forgot to add the offset so I just wrote "value 0xfd" out to it.

Driver won't load now although I have another working one and an EEPROM dump

Does anyone know how to restore the EEPROM via sysfs through the PCI bus?
Haven't had to do that with the 10GbE card, but I did that with an Intel 1GbE many years back.

My notes are at Unbricking an Intel Pro/1000 (e1000) network interface - ixs' Vodkamelone, maybe they can help you.
Dump the flash from a working card, modify the flash to have the "right" MAC address.
Then patch the intel driver to remove the flashrom consistency checks to load with the wiped card and then bit-bang the working dump back in.

Good luck.
 

ixs

New Member
Aug 14, 2020
3
1
3
Also. Why does nothing complain about the EEPROM checksum at 0x3F for this modification? :) Aren't we putting some OSes at risk by not correcting the checksum word?
That's a good point actually. Do you have a bit of docs about the EEPROM checksum calculation?
 

autoturk

Member
Sep 1, 2022
25
28
13
EDIT, original post below: I figured it out after sleuthing through the Linux drive coder that the transceiver I have is so old/janky that it doesn't have a "Transceiver Compliance Code" which means the driver doesn't recognize it as a 10 Gb transceiver and drops the port, regardless of the support status in the driver of EEPROM. I initially kludged together a fix by patching the driver, but found a more proper fix in this gist here. The tricky part was getting this fix working in pfSense, which required setting up a FreeBSD 12.3 VM and making the fix and compiling the driver in there since apparently pfsense strips all compilers by default from FreeBSD. If anyone needs help doing this I'm happy to type out more detailed steps.

------------------

I'm having a bit of an issue that's making me scratch my head. I have a "Proline SFP-10G-T" transceiver. If I connect it directly to my switch ( TL-SG3210XHP-M2) then it auto negotiates to 10 gb and things seem to be working fine.

Everything below was done on an Ubuntu 22.04 live session on an H370 chipset motherboard with an 8700k CPU.

Whenever I connect it to either an Fujitsu X520-DA2 (not identical to Intel X520 reference redesign) or an IBM X520-DA2 (identical to Intel), then the port drops from the kernel and I get the unsupported message in dmesg. Oddly enough, however, the bit referenced above is set to either 0xff (Fujitsu) or 0xfd (IBM) so the unsupported module should be allowed. Setting the `allow_unsupported_sfp` flag seems to do nothing as well -- plugging it in immediately leads to the port dropping and dmesg spitting out that error.

I'm super confused. Could the transceiver just not be working? It works fine connected to directly to my switch and seems to be transferring at 10 Gb (I have an rj45 X552 I connected it to to test) when connected to it, so I'm not sure what's going on here.
 
Last edited:

ChromeTO

New Member
Mar 12, 2020
5
5
3
Just followed this guide for my Supermicro AOC-STGN-I2S Rev 2.0 (Intel X520-DA2). Works perfectly now with "non-supported" SFPs. I have both supported and non-supported. Now the card will accept everything without complaining and shutting down the port.

The original post matched my card exactly.

Thanks to this community!
 
  • Like
Reactions: epicurean

Craig Curtin

Member
Jun 18, 2017
94
19
8
58
Thanks for all the hard work on this one OP - great work getting this far.

A quick question - does anyone know if this will work for onboard Intel 82599 adapters ?

I have a Dell 730xd that has 2 x Intel 10G and 2 x Intel 1Gb on the motherboard - the onboard 10G are rejecting most of the transceivers i throw at them.

I have a 10M 10G - AOC cable that i know works with the flashed cards (i have two of these cables) and they work with other Intel 520-Da2 after flashing - so i was going to buy two more of these cables - but wanted to make sure i could first flash the onboard - as the hosts are running ESXi - i will have to arrange downtime and a bootable Linux Key

Craig
 

Bomberski

New Member
Nov 22, 2022
2
1
3
Lot of info for a new person to take on,
this might help some one.
(Copy & Paste ;))

Might have to run sudo before every command(I don't remember if i did for every command)

I had problems with the live Ubuntu on usb so I just installed it on a old HD I had laying around.(20.04.1 was used) did the apt-update and apt-upgrade and you will need to install ethtool in Ubuntu.

-‐----------------------------------------------------------------------


Dell XYT17 Intel X520-DA2


‐---------------------------------‐-------------------------------------
sudo ip link show (find your cards port one number)

2-15-1024x358.png
‐----------------------------------‐---------------------------------
(Find current value)
Sudo ethtool -e PUTCARDNUMBERHERE offset 0x58 length 1

Offset Values
------ ------
0x0058: fc

‐---------------------------------‐-------------------------------------

(To Change value)

sudo lspci -nn

This will give you the manufacturer ID and the PCI ID( thanks Craig!) Like the text below.

(04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)

The magic number was made from this line above [8086:154d] to 0x154d8086 To use below.

Sudo ethtool -E PUTCARDNUMBERHERE magic 0x154d8086 offset 0x58 value 0xfd length 1

‐---------------------------------‐-------------------------------------

(Check if it took)
Sudo ethtool -e PUTCARDNUMBERHERE offset 0x58 length 1

Offset Values
------ ------
0x0058: fd

Should be good

‐---------------------------------‐-------------------------------------

I am just a home lab user with very little Linux experience. This took me hours to figure out skimming the 3 pages here and basic Linux commands at others sites.
This only works for the dell version listed.
No spoon feeding at this site lol

Installed in a R210ii with pfsense and set my interfaces after boot. Works great so far, sfp+ rj45 adapter worked perfectly with my xb7 modem. Modem shows 2.5g and pfsense shows 10g.
 
Last edited:
  • Like
Reactions: safado

Craig Curtin

Member
Jun 18, 2017
94
19
8
58
Lot of info for a new person on this site. So this might help some one. Might have to run sudo before ever command(I don't remember)
Will need to install ethtool in Ubuntu.
Maybe some updates?




Dell XYT17 Intel X520-DA2

sudo ip link show (find your cards port one number)

View attachment 25704

(Find current value)
Sudo ethtool -e PUTCARDNUMBERHERE offset 0x58 length 1

Offset Values
------ ------
0x0058: fc

(To Change value)

(04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)

The magic number was made from this line above [8086:154d] to 0x154d8086 To use below.


Sudo ethtool -E PUTCARDNUMBERHERE magic 0x154d8086 offset 0x58 value 0xfd length 1

(Check if it took)
Sudo ethtool -e PUTCARDNUMBERHERE offset 0x58 length 1

Offset Values
------ ------
0x0058: fd

Should be good

I am just a home lab user with very little Linux experience. This took me hours to figure out skimming the 3 pages here and basic Linux commands at others sites.
This only works for the dell version listed.
No spoon feeding at this site lol

I have not added this card to my r210ii to see if it work in pfsense yet... wish me luck
Good one - the only bit to add to that is how to get the info for the magic number

type

sudo dmesg : lspci -nn

This will give you the manufacturer ID and the PCI ID

Craig
 
  • Like
Reactions: Bomberski