I350-F4 EEPROM Investigation + 1G Intel NIC Power Consumption Comparison + Enabling 1G DAC for Intel X520

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

artorias

New Member
May 8, 2022
12
9
3
Hey everyone,
I also posted this on reddit and got a promising response but since this might also be interesting for this community and to increase my chances of succeeding, I decided to try my luck and also post this here.

The backstory goes like this: On the quest to build a low power pfsense box with an SFP NIC I got my hands on a 10gtek i350-gr-4s-x4 for super cheap on ebay. It works for the most part but unfortunately, there are some weird issues with this card.

Some examples:
  • ethtool enp2s0f0 reports that the card has internal transceivers (doesn't make sense, it has SFP cages)
  • getting information from an SFP module with ethtool -m enp2s0f0 doesn't work, it says that the operation is not supported
    It works with another NIC I have, so the problem is not the SFP module.
  • trying to update the card with Intel's bootutil doesn't work, it says no flash present (flash or option ROM is used to store UEFI, PXE, iSCSI boot configs etc.). Older versions of it say LOM, flash erased. I can physically see the flash chips on the PCB. One for the EEPROM and one for the flash/option ROM ...so that doesn't make any sense either
  • RJ-45 SFPs work but when I reboot with one of them inserted the port gets disabled due to some error

After some research I found out that the configuration of the i350 controller is stored on an EEPROM. I dumped the contents of it with ethtool -e enp2s0f0 and did some comparison against the i350 datasheet (Section 6) which describes many of the configuration parameters and their possible values.

Well, turns out the current configuration is wrong in many ways. I'm surprised it even works in this state. (LOM mode instead of NIC, wrong flash size, fiber only mode instead of SerDes or SGMII, etc...)

I want to keep the card and try to fix it. To make this a littble bit easier it would be awesome if one of you happens to have access to such an i350 based SFP card and can share a dump of the EEPROM so that I can hack together my own config.

The NICs I'm looking for that might contain the correct configuration are:
  • Supermicro AOC-AG-i4S(M)
  • Dell XV0JH
  • Silicom PE2G4SFPI35L
  • Silicom PE2G2SFPI35
  • Silicom PE2G2SFPI35-GXX
  • Axiomtek AX93322
  • Axiomtek AX93336
  • IEI PulM-1G4SF
  • IEI PulM-1G8SF
  • ALLNET ALL0136-2SFP
  • Advantech PCIE-1131
  • Advantech PCIE-2130
  • Addon Networks I350F4-AO
  • Addon Networks ADD-PCIE-4SFP
  • LR-Link LREC9714HF-4SFP
  • Inter-Tech Argus ST-7212
  • Edgeoptic 1G-NIC-I350-2O
  • Edgeoptic 1G-NIC-I350-4O

To dump the eeprom I used this command:
ethtool -e enp2s0f0 raw on > eeprom.bin

Another thing that would help me are the i350 eeprom config templates from Intel. Would be nice to have a clean reference from which I could customize my config from.

EDIT:
From what I can tell it's a genuine i350 chip. I removed the heatsink and the chip has the same markings as my i350-t4. Also, the eeprom and flash chip are the exact ones that the datasheet lists as compatible. The values and memory addresses from my eeprom dump match those of the datasheet.
Compared to the copper variants there are no "Intel official" versions of the SFP one. You can only get these from OEMs. The i350-f4 variant from Intel has non-removable transceivers built-in with LC connectors. I guess the configuration I have was taken from one of those without further customizing.
 
Last edited:

artorias

New Member
May 8, 2022
12
9
3
Ah, when I was searching for .eep files, I was wondering why the files are not included in any newer versions of this driver package. Good to know that they were included in an older version.

I'll check those files. Let's see if I can use them....

EDIT:
Unfortunately, all of them are for the copper/RJ45 variants but thank you for the link. Maybe someone with a broken RJ45 variant will find it useful.
 
Last edited:

joeribl

Active Member
Jun 6, 2021
129
45
28
Question, there exist an official Intel Eeprom tool: EEUPDATE, maybe you could try with that to see what the output is? I can provide you with it if you need.
 

azhiltsov

New Member
Aug 19, 2022
2
0
1
I have received the card and dumped the eeprom with
ethtool -e enpXXXX raw on > eeprom.bin

I shared the eeprom in a private message.
 

artorias

New Member
May 8, 2022
12
9
3
Hey @azhiltsov, thank you very much for the eeprom dump. This will definitely help.
It's been a few months and I made some progress. I'm not finished but the information I gathered so far might still be valueable to others.
I'll soon write a post with all the things I discovered.

@joeribl I'll write it in my next post but here's the short version:
I have EEUPDATE v5.38.10.00 and NVMUpdate version 1.33.15.1 (have not tested newer version of nvmupdate yet). You can indeed use both of these to talk to the i350 eeprom. With NVMUpdate you have to modify nvmupdate.cfg to make it detect the i350 card.

The dumps of ethtool, NVMUpdate and EEUPDATE contain the same data but in different formats.

I'll keep you updated...
 
  • Like
Reactions: joeribl

artorias

New Member
May 8, 2022
12
9
3
Ok so the tl;dr is:
The card is perfectly fine except for it not recognizing the flash chip, which I was able to fix. Most of the issues stem from the SFP modules I was using. The card works with:
  • 1G SFP-DAC
  • 10G SFP+-DAC on 1G speed
  • 1000base-T RJ-45 SerDes SFP
  • 10/100/1000base-T RJ-45 SGMII SFP
  • 1000base-SX SFP
  • 10Gbase-SX SFP on 1G
Can't test 1000base-LX SFP and 10Gbase-LX SFP on 1G, since I don't have such modules but most likely work also. Also haven't tested active DAC cables.

The ethtool outputs are correct and make sense after further investigation.

I spent quite some time reading the i350 datasheet and other sources until I understood all of this. I'll give you a summary.

----------------

PART 1: Clearing up the problems I had:


Why was the SFP module EEPROMM dump not working?

Well, as it turns out there was something I forgot. I was using an out-of-tree version of the Intel igb driver. After uninstalling that one, I was able to use ethtool -m enp2s0f0 again to obtain the module eeprom information.​
I discovered this through this discussion on sourceforge. To be specific, it works on my Proxmox system with kernel 5.15.53-1-pve and same version igb driver 5.15.53-1-pve.​

Which modules work and why did some of mine not work?

For more details like ethtool -m output for these modules and PHY register dumps, see the next post.​
These are the modules, that I tested an worked:​
  • Intel branded Finisar FTLX8571D3BCV-IT 10Gbase-SX on 1G speed
  • Cisco MGBSX1 1000base-SX
  • QSFPTEK QT-SFP-M CO SFP 1000M RJ45 100M in SerDes mode and SGMII mode (Marvell Alaska Ultra 88E1111 PHY chip) also work with 10/100 speeds if in SGMII mode
  • 10Gtek CAB-1GSFP-P 1GBase-CU DAC
  • HiFiber CAB-10GSFP-P 10Gbase-CU DAC on 1G speed
Modules that work with quirks:​
  • QSFPTEK QT-SFP-T CO SFP 10/100/1000 RJ45 in SGMII mode (Marvell Alaska Ultra 88E1111 PHY chip)
  • Cisco MGBT1 SFP RJ45 (unknown PHY chip)
Modules that don't work:​
  • FS.com SFP-10G-T 10GBASE-T-SFP+ (Marvell Alaska X 88X3310/40P PHY chip)
Why does the FS 10Gbase-T module not work?
Problem is that the i350 chip has to agree on a speed with the PHY in the module. This happens over MDIO. MDIO has two flavors, Clause 22 and Clause 45.​
The problem here is that the i350 can only speak Clause 22 and the module PHY only Clause 45. To make it work, a translation needs to happen. The driver should handle this. As far as I understand, the igb driver can't do this. I'm not sure but if it would use phylink and phylib, it could potentially work. Intel uses their on implementation according to this discussion.​
I had an idea how to fix this but I don't know if it's possible. If there was a way to manually programm the PHY registers in the module, the module might work together with the i350 if it's forced to 1G. As for as I understand, it's in 10G mode by default.​
Here is what happens whens I try to establish a link between the FS module and the QSFPTEK module:​
Code:
igb 0000:02:00.0 enp2s0f0: After fix-ups FlowControl is now = 3
igb 0000:02:00.0 enp2s0f0: Configuring Autoneg:PCS_LCTL=0x0217000C
igb 0000:02:00.0 enp2s0f0: Initializing the Flow Control address, type and timer regs
igb 0000:02:00.0 enp2s0f0: PCS Auto Neg has not completed.
igb 0000:02:00.0 enp2s0f0: PCS Auto Neg has not completed.
igb 0000:02:00.1 enp2s0f1: hw->fc.current_mode = 3
igb 0000:02:00.1 enp2s0f1: Flow Control = RX PAUSE frames only.
igb 0000:02:00.1 enp2s0f1: 1000 Mbs,
igb 0000:02:00.1 enp2s0f1: Full Duplex
igb 0000:02:00.1 enp2s0f1: hw->fc.current_mode = 1
igb 0000:02:00.1 enp2s0f1: 1000 Mbs,
igb 0000:02:00.1 enp2s0f1: Full Duplex
igb 0000:02:00.1 enp2s0f1: igb: enp2s0f1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f1: link becomes ready
igb 0000:02:00.0 enp2s0f0: PCS Auto Neg has not completed.
igb 0000:02:00.0 enp2s0f0: PCS Auto Neg has not completed.
igb 0000:02:00.0 enp2s0f0: PCS Auto Neg has not completed.
The QSFPTEK module can establish a link with the FS module but the FS module can't establish a link with the i350, it gets stuck on PCS Auto Neg.
Whats wrong with the other modules?
The Cisco MGBT1 works in SerDes mode but doesn't in SGMII mode.​
SerDes mode is activated when you insert the module on a booted system with the driver loaded. If the driver is not loaded and you insert the module and then load the driver, the driver tries to operate in SGMII mode but fails.​
Here ist the error:​
Code:
igb 0000:02:00.2 0000:02:00.2 (uninitialized): Vendor ID 0x00008000 read at address 1
igb 0000:02:00.2 0000:02:00.2 (uninitialized): Vendor ID 0x00008000 read at address 2
igb 0000:02:00.2 0000:02:00.2 (uninitialized): Vendor ID 0x00008000 read at address 3
igb 0000:02:00.2 0000:02:00.2 (uninitialized): Vendor ID 0x00008000 read at address 4
igb 0000:02:00.2 0000:02:00.2 (uninitialized): Vendor ID 0x00008000 read at address 5
igb 0000:02:00.2 0000:02:00.2 (uninitialized): Vendor ID 0x00000000 read at address 6
igb 0000:02:00.2 0000:02:00.2 (uninitialized): I2CCMD Error bit set
igb 0000:02:00.2 0000:02:00.2 (uninitialized): PHY address 7 was unreadable
igb: probe of 0000:02:00.2 failed with error -2
It can't read PHY register 7 for some reason and produces a hardware initialization failure. I suspect there is something wrong with the PHY register configuration.​
Removing the module and reloading the driver doesn't solve the issue, you have to reboot.​
The QSFPTEK QT-SFP-T CO seems to be SGMII only. Meaning you have to reload the driver after inserting it to make it work. It doesn't work in SerDes mode.​
One thing about the QSFPTEK QT-SFP-M CO module:​
This one seems to be able to work in SerDes and SGMII mode. When inserting it with the driver loaded, it operates at 1G fixed speed. When reloading the driver it changes to SGMII mode and works with 10/100/1000 speeds.​
This is how this looks like:​
Code:
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Vendor ID 0x0000FFFF read at address 1
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Vendor ID 0x0000FFFF read at address 2
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Vendor ID 0x0000FFFF read at address 3
igb 0000:02:00.1 0000:02:00.1 (uninitialized): I2CCMD Error bit set
igb 0000:02:00.1 0000:02:00.1 (uninitialized): PHY address 4 was unreadable
igb 0000:02:00.1 0000:02:00.1 (uninitialized): I2CCMD Error bit set
igb 0000:02:00.1 0000:02:00.1 (uninitialized): PHY address 5 was unreadable
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Vendor ID 0x00000141 read at address 6
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Masking off all interrupts
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Masking off all interrupts
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Initializing the IEEE VLAN
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Programming MAC Address into RAR[0]
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Clearing RAR[1-31]
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Zeroing the MTA
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Zeroing the UTA
igb 0000:02:00.1 0000:02:00.1 (uninitialized): After fix-ups FlowControl is now = 3
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Configuring Autoneg:PCS_LCTL=0x0213000C
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Soft resetting SGMII attached PHY...
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Reconfiguring auto-neg advertisement params
igb 0000:02:00.1 0000:02:00.1 (uninitialized): autoneg_advertised 2f
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Advertise 10mb Half duplex
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Advertise 10mb Full duplex
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Advertise 100mb Half duplex
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Advertise 100mb Full duplex
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Advertise 1000mb Full duplex
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Auto-Neg Advertising de1
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Restarting Auto-Neg
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Unable to establish link!!!
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Initializing the Flow Control address, type and timer regs
igb 0000:02:00.1 0000:02:00.1 (uninitialized): Phy info is only valid if link is up
igb 0000:02:00.1: added PHC on eth0
igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:02:00.1: eth0: (PCIe:5.0Gb/s:Width x1) xx:xx:xx:xx:xx:xx
igb 0000:02:00.1 eth0: NVM PBA number is not stored as string
igb 0000:02:00.1: eth0: PBA No: 106300-000
igb 0000:02:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
igb 0000:02:00.1 enp2s0f1: renamed from eth0
Also, this is how the ethtool output looks like after that:​
Code:
root@Proxmox:~# ethtool enp2s0f1
Settings for enp2s0f1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 6
        Transceiver: internal
        MDI-X: off (auto)
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00007fff (32767)
                               drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol
        Link detected: no
Why does the 10G Fiber module and 10G DAC cable work?

Many people are preaching that SFP+ modules don't work in SFP slots. Doesn't seem to be true here.​
If both sides operate on 1G, the SFP+ fiber module seems to have no issues.​
Short DAC cables are entirely passive, except the eeprom that has no active function for link establishing, there a no active components in it.​
You can compare it to a normal Cat6A cable. Those also don't care at what speed they operate.​
Haven't tested active DAC cables.​


How did I make the card recognize the flash?
Reference was the EEPROM dump from @azhiltsov.​
After making the following changes to the eeprom, the flash was accessible again and I was able to flash PXE etc. to it.​

Word Name and AdressWord in HEX / Binary before ChangeWord in HEX / Binary after ChangeChanges
Compatibility Word (Word 0x03)0800 (0000100000000000)0000 (0000000000000000)Bit 11:
1 -> 0
LOM mode to NIC.
EEPROM Sizing and Protected Fields (Word 0x12)5C80 (0101110010000000)6080 (0110000010000000)Bit 13-10:
0111 -> 1000
EEPROM size 16kbytes -> 32kbytes according to eeprom chip datasheet
PCIe Init Configuration Word 2 (Word 0x19)7000 (0111000000000000)3000 (0011000000000000)Bit 14:
1 -> 0
IO_Sup (I/O Support) disabled

PCIe Control 2 (Word 0x28)
1C40 (0001110001000000)1C4D (0001110001001101)Bit 0:
0 -> 1 Flash BAR in the PCI configuration space enabled.

Bit 1-3:
000 -> 110
Flash Size 64KB to 512KB according to flash chip datasheet
What's up with the confusing ethtool output?

All of the information is based on this video, which explains how Layer 1 and 2 work in Linux.​
The ethtool output is this:​
Code:
root@Proxmox:~# ethtool enp2s0f0
Settings for enp2s0f0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: on
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00007fff (32767)
                               drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol
        Link detected: no
Let's first see what the igb driver reports to ethtool:​
Line 148-174 in igb_ethtool.c tells us what happens.​
Pretty straightforward, if the NIC'S media type is not copper, it's fibre, does 1000baseKX_full and supports auto negotiation.​
Is that actually correct? Kind of... It seems like the SFP variants of the i350 cards are kind of an afterthought. Intel's i350-f4 doesn't have SFP modules and not too many OEMs made this SFP variant...so I guess no one ever asked for a more correct reporting in ethtool when it works just fine either way.​
Let's look at the lines that might be confusing more closely:
Supported ports: [ FIBRE ]
Port: FIBRE
We're using SFP modules, so it doesn't have to be fiber. DAC and RJ45 SFPs are also working.​
What are the other options? Let's look at the ethtool source code.​
This section lists the possible options for "Supported Ports" :​
TP, AUI, BNC, MII, FIBRE and Backplane​
And this section lists the possible options for "Port":​
Twisted Pair, AUI, BNC, MII, FIBRE, Diract Attach Copper, None, Other and Unknown!​
I guess it would make sense listing multiple supported ports.​
TP for RJ45 SerDes SFPs, MII for RJ-45 SGMII SFP, FIBRE for SX and LX SFPs, and Direct Attach Copper for DAC cables.​
Backplane, as far as I understand, would only fit for PCI mezzanine cards for blade servers.​
On the other hand, Intel x520 and x710 also show FIBRE as port.​
Also, see i350 datasheet page 699 for the different modes the card can operate in. The term fiber is used for SerDes operation here.​
Supported link modes: 1000baseKX/Full
Advertised link modes: 1000baseKX/Full
1000baseKX actually is for backplane connections.​
What are the options in ethtool?​
This section lists them:​
Possible ones for this card could be:​
  • 10/100/1000base-T for RJ45 SFPs
  • 1000base-X for fiber SFPs
  • 1000base-CU is missing
  • 1000base-KX
So why use KX?
I guess they use KX because backplane connection could also be interpreted as the SerDes lanes on the card from the i350 chip to the SFP cages.​
Again, this doesn't seem to be much of a concern. The x520 reports 1000baseT and 10000baseT. The x710 reports 10000baseT/Full, 1000baseX/Full, 10000baseSR/Full and 10000baseLR/Full.​
Supports auto-negotiation: Yes
Advertised auto-negotiation: Yes

Why auto negotiation if the card only operates at 1G speed with non SGMII (10/100/1000) modules?​
The process of auto negotiation still happens, but the card just reports only the 1G capability to the link partner.​
Transceiver: internal

This one was very confusing. "SFP module" is often used as a synonym for "transceiver", so it should make sense to have "external" here. What I have found out is that transceiver is also used as a synonym for the PHY (component).​
You should watch the video (the whole one, do it, it's worth the time) I linked to fully understand this but in summary:​
The SFP module is not the whole transceiver and the PCS portion of the PHY/transceiver that's integrated into the i350 is always active to enable the communication over the SerDes lines to whatever type of module there is in the SFP cage.​
I can only imagine this to report "external" if the card is in SGMII mode with an external PHY chip (not inside an SFP module but on the main PCB) directly connected to the (SG)MII pins of the i350 controller.​
Also, X520 and X710 report "internal" here as well.​

--------------

PART 2 EEPROM Flashing:


Let's start with how to obtain debug information from the igb driver.

1. How to get igb debug information:
  1. unload the driver rmmod igb
  2. load the driver with options and dynamic debugging on modprobe igb debug=16 dyndbg==p
  3. view the information with demsg | grep igb
If you don't see additional messages you might also enter dmesg -n 8 and then repeat the previous steps. This might also print the messages on your current console screen. You'll need to do this after every reboot. This was all I needed to do on my Proxmox system. You might need to jump through some more hoops on different systems.​
I also found this command in my notes, but i don't remember when I needed it:​
echo -n 'module igb +p' > /sys/kernel/debug/dynamic_debug/control
More information on dynamic debugging of kernel modules here.​

---

2. How to read/dump the i350 EEPROM:
There are multiple tools to do this. You can do it with ethtool, nvmupdate and eeupdate. ethtool and nvmupdate are publicly available, eeupdate is confidential. Your company needs to sign an NDA to get access. You don't need it but it has some additional functionalities when it comes to interfacing the card. You can find its readme file out there to find out more. I searched it for hours until if found out I can use nvmupdate. You can nonetheless find eeupdate via google. Two sources are leaking it under a different name if you really need it.​
I think eeupdate was used for intel network chips until the X520 series and nvmupdate is now the successor for X550 series and newer. Luckily, we can use nvmupdate also for the older cards.​
If you are on BSD I think there is nothing like ethtool. In this case use NVMUpdate.​
You can also use a CH341a bios flasher. More info on that in section 4.​
2.1. Installing the Intel® Ethernet Adapter Debug Driver for Linux (iqvlinux):
I don't know if this is really necessary. I have not tested all of this without it installed and I don't want to test all this all over again...so just install it to be safe. eeupdate wil throw a warning if this is not installed. You will also need it if you have a disabled option ROM flash and want to dump the EEPROM with NVMUpdate.​
Finding the newest version that is compatible with the current Proxmox 5.15.x kernel was quite tedious. The repository on sourceforge is outdated. You can't compile the old driver on current kernel versions.​
Eventually, I found it. The newest version is packaged with the Intel Ethernet Connections Boot Utility, Preboot Images, and EFI Drivers.​
  1. Download the latest version from Intel here.
  2. unpack tar -xzvf Preboot.tar.gz
  3. cd ./PREBOOT/APPS/Bootutil/Linux_x64/DRIVERS
  4. chmod +x install
Before installing, you need to install your kernel headers and build-essential.​
5. apt install build-essential
6. For Proxmox do apt install pve-headers or apt install pve-headers-$(uname -r) Adjust according to your system. If the download fails, make sure you have the pve-no-subscription repository in your /etc/apt/sources.list
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
7. run ./install
If this still fails and it complains about a missing autoconf.h file, try reinstalling the kernel headers. apt reinstall pve-headers-$(uname -r) or apt reinstall pve-headers. This was an issue I had. Then run the install script again.​
The driver should be installed now.​
2.1. EEPROM dump with ethtool:
The dump from ethtool comes in a format that needs modification before you can edit and flash the image with the other tools to the eeprom.​
  1. Read with ethtool -e enp2s0f0 raw on > eeprom.bin (you might need to change the interface name)
  2. Then hexdump -ve '8/2 "%04X " " \n"' eeprom.bin > eeprom.eep
Now it's in the format that the nvmupdate and eeupdate can use. (Same format as these tools' dump command produces)​
2.2 EEPROM dump with NVMUpdate:
note: I stated in the original post that NVMUpdate would not detect the card. That was wrong. It detects it and prints information about it but does not read or flashthe EEPROM without modification of the nvmupdate.cfg file.​
This only works if you have a detected and enabled option rom flash or already have a dump from ethtool.​
In case the flash is not activated, you need to activate it using intel BootUtil (see 2.1).​
1. run BootUtil to identify your nic ./bootutil64e
2. activate the flash ./bootutil64e /NIC=X /FE
If that doesn't work, try to do it with IBAUtil.exe DOS tool from Dell.​
Create a bootable FreeDOS USB Stick with Rufus to use it. Commands should be the same like BootUtil.​
We have to do this because there is now way in NVMUpdate to just dump the EEPROM. You always have to update flash or eeprom in the same step.​
Now to NVMUpdate:​
  1. Download NVMUpdate from the intel website.
  2. unzip it unzip 700Series_NVMUpdatePackage_v9_01.zip might need to apt install unzip first.
  3. unpack the .tar tar -xvzf 700Series_NVMUpdatePackage_v9_01_Linux.tar.gz
  4. cd ./700Series/Linux_x64/
  5. chmod +x nvmupdate64e
  6. run ./nvmupdate, this should display all intel NICs in the system
  7. run ./nvmupdate64e -i -l to display more information of all intel NICs in the system
  8. Add the configuration for your device to nvmupdate.cfg
Should look like this:​
Code:
CURRENT FAMILY: 1.0.0
CONFIG VERSION: 1.14.0

;Release 19.3/19.4 to Release 20.0

BEGIN DEVICE
DEVICENAME: DEVICENAME: Intel(R) I350 Gigabit Fiber Network Connection
VENDOR: 8086
DEVICE: 1522
SUBVENDOR: 8086
SUBDEVICE: 1522
SKIP OROM: FALSE
OROM IMAGE: BootIMG.FLB
;EEP IMAGE:
;EEPID:
;REPLACES:
RESET TYPE: POWER
END DEVICE

BEGIN DEVICE
DEVICENAME: XL710
...
The information to insert here is based on the previous output of ./nvmupdate64e -i -l . Lines with a semicolon as the first character are commented out.​
8. Now update the option ROM and dump the eeprom and flash with ./nvmupdate64e -l -b -u
This creates a folder with the name of your NICs MAC adress containing the macadress.flb and macadress.eep file.​
.eep is the eeprom dump, .flb is the flash dump.​
If it doesn't work try with the -f flag to force the flashing of the OROM.​
Full documentation of NVMUpdate here.​
2.3 dump with eeupdate:
  1. Make sure you installed the iqv drivers, see 2.1.
  2. Run eeupdate once to identify nic
  3. run again ./eeupdate64e /NIC=x /DUMP to dump the eeprom
To be on the safe side, run ./eeupdate64e /NIC=ALL /DUMP , in case you **** up and select the wrong NIC to flash in the flashing step.​

3. Editing the EEPROM image:

Now to edit it I like to copy it on a Windows machine, open the .eep file in editor and copy the single words into HxD.​
Then open the i350 controller datasheet chapter 6 eeprom map and find what you want to change.​
Copy the word into HxD and click on it. On the right side you see the bits its made of.​
One 4 character group in the eeprom.eep file is one word. The first word of the first line is address 0X00 the last address 0x07. Second line starts with 0x08 to 0x0F. Third line starts with 0x10 and so on.​
Let's look at one example word:​
0A37​
In HxD default viewing mode you will see:​
0A 37​
If you click on 0A, the data inspector will show it in Binary 8 bit form. You read from right to left. In this case:​
bit 15 -> 00001010 <- bit 8​
And 37 will be:​
bit 7 -> 00110111 <- bit 0​
Now look into the datasheet and identify which bit you need to change, highlight it in the data inspector, change it and press enter. The word should have changed now, just apply the change accordingly in the eeprom.txt file.​
When you made all your changes, save them and copy the file back to your Linux machine.​


4. Flashing the eeprom:

Before doing this, please be aware that flashing eeproms from other NICs, like flashing i210 eeprom to i350 eeprom might break the card. Not completely but it might not show up on the PCIE-Bus anymore what also means, you can't flash it with any of these tools anymore. This is because parts of the eeprom configure the initial state of some PCIE control structures/registers aka CSR space. This is the only part of this I don't really understand right now.​
If you break it, you have two options.​
  1. Rip the flash from the board and force it to operate in eeprom-less mode. This is bad. The default configuration in eeprom-less mode, probably makes it show up in the PCIE-Bus again but it also won't work properly anymore since the configuration is wrong.
  2. Use a CH341 based BIOS-chip flasher. I recommend this one that has a voltage switch:

    The other popular one floating around is the black one in this video:

    But be careful with this one. Like the video says, this one sends 5V over the data lines even if it's in 3.3V mode. This might kill the chip. Make sure to find the datasheet to your eeprom chip to find out the correct voltage. There is a big community around this so I won't explain this in detail. I tested it on a i350-t2 and it worked. Sometimes in-circuit flashing doesn't work and you have to remove the eeprom from the board, flash it and resolder it.
Anyway...​
Before you flash, write down your NIC's MAC-adresses. Usually they won't be overwritten but I haven't tested this thoroughly. Flashing the .eep file from someone else with different MAC-Adresses should not change your MAC adress...should.​
4.1 Flashing with ethtool:
i didn't bother learning how to do this, but others have done this with X520s to unlock it for other brand SFPs. See this thread.​
4.2 Flashing with nvmupdate:
Modify nvmupdate.cfg again to something like this:​
Code:
CURRENT FAMILY: 1.0.0
CONFIG VERSION: 1.14.0

;Release 19.3/19.4 to Release 20.0

BEGIN DEVICE
DEVICENAME: DEVICENAME: Intel(R) I350 Gigabit Fiber Network Connection
VENDOR: 8086
DEVICE: 1522
SUBVENDOR: 8086
SUBDEVICE: 1522
SKIP OROM: TRUE
;OROM IMAGE: BootIMG.FLB
EEP IMAGE: eeprom.eep
;EEPID:
;REPLACES:
RESET TYPE: POWER
END DEVICE

BEGIN DEVICE
DEVICENAME: XL710
...
After that, run ./nvmupdate -l -b -u.​
Because we are using the same eep image revision, this will fail, because the programm thinks the current image and the modified file are the same. To force the flashing, use ./nvmupdate -l -b -u -f
4.3 Flashing with eeupdate:
Be carefull here and make sure to select the correct NIC.​
./eeupdate64e /nic=X /D eeprom.eep


5. Reboot

That's it.
 
Last edited:

artorias

New Member
May 8, 2022
12
9
3
Here are the ethtool -m dumps of the SFP modules:

Dumping the Cisco MGBSX EEPROM didn't work for some reason.

Intel / Finisar FTLX8571D3BCV-IT 10Gbase-SR/1000base-SX :
Code:
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x10 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 10G Ethernet: 10G Base-SR
        Transceiver type                          : Ethernet: 1000BASE-SX
        Encoding                                  : 0x06 (64B/66B)
        BR, Nominal                               : 10300MBd
        Rate identifier                           : 0x02 (8/4/2G Rx Rate_Select only)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 80m
        Length (62.5um)                           : 30m
        Length (Copper)                           : 0m
        Length (OM3)                              : 300m
        Laser wavelength                          : 850nm
        Vendor name                               : Intel Corp
        Vendor OUI                                : 00:1b:21
        Vendor PN                                 : FTLX8571D3BCV-IT
        Vendor rev                                : A
        Option values                             : 0x00 0x3a
        Option                                    : RX_LOS implemented
        Option                                    : TX_FAULT implemented
        Option                                    : TX_DISABLE implemented
        Option                                    : RATE_SELECT implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : REDACTED
        Date code                                 : 170711
        Optical diagnostics support               : Yes
        Laser bias current                        : 7.604 mA
        Laser output power                        : 0.5942 mW / -2.26 dBm
        Receiver signal average optical power     : 0.0008 mW / -30.97 dBm
        Module temperature                        : 29.25 degrees C / 84.66 degrees F
        Module voltage                            : 3.3145 V
        Alarm/warning flags implemented           : Yes
        Laser bias current high alarm             : Off
        Laser bias current low alarm              : Off
        Laser bias current high warning           : Off
        Laser bias current low warning            : Off
        Laser output power high alarm             : Off
        Laser output power low alarm              : Off
        Laser output power high warning           : Off
        Laser output power low warning            : Off
        Module temperature high alarm             : Off
        Module temperature low alarm              : Off
        Module temperature high warning           : Off
        Module temperature low warning            : Off
        Module voltage high alarm                 : Off
        Module voltage low alarm                  : Off
        Module voltage high warning               : Off
        Module voltage low warning                : Off
        Laser rx power high alarm                 : Off
        Laser rx power low alarm                  : On
        Laser rx power high warning               : Off
        Laser rx power low warning                : On
        Laser bias current high alarm threshold   : 13.200 mA
        Laser bias current low alarm threshold    : 4.000 mA
        Laser bias current high warning threshold : 12.600 mA
        Laser bias current low warning threshold  : 5.000 mA
        Laser output power high alarm threshold   : 1.0000 mW / 0.00 dBm
        Laser output power low alarm threshold    : 0.2512 mW / -6.00 dBm
        Laser output power high warning threshold : 0.7943 mW / -1.00 dBm
        Laser output power low warning threshold  : 0.3162 mW / -5.00 dBm
        Module temperature high alarm threshold   : 78.00 degrees C / 172.40 degrees F
        Module temperature low alarm threshold    : -13.00 degrees C / 8.60 degrees F
        Module temperature high warning threshold : 73.00 degrees C / 163.40 degrees F
        Module temperature low warning threshold  : -8.00 degrees C / 17.60 degrees F
        Module voltage high alarm threshold       : 3.7000 V
        Module voltage low alarm threshold        : 2.9000 V
        Module voltage high warning threshold     : 3.6000 V
        Module voltage low warning threshold      : 3.0000 V
        Laser rx power high alarm threshold       : 1.0000 mW / 0.00 dBm
        Laser rx power low alarm threshold        : 0.0100 mW / -20.00 dBm
        Laser rx power high warning threshold     : 0.7943 mW / -1.00 dBm
        Laser rx power low warning threshold      : 0.0158 mW / -18.01 dBm
QSFPTEK QT-SFP-M CO SFP 1000M RJ45 SERDES:
Code:
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x00 (unknown or unspecified)
        Transceiver codes                         : 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x01
        Transceiver type                          : Ethernet: 1000BASE-T
        Transceiver type                          : Extended: 100G AOC or 25GAUI C2M AOC with worst BER of 5x10^(-5)
        Encoding                                  : 0x01 (8B/10B)
        BR, Nominal                               : 1300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 0m
        Length (62.5um)                           : 0m
        Length (Copper)                           : 100m
        Length (OM3)                              : 0m
        Laser wavelength                          : 0nm
        Vendor name                               : QSFPTEK
        Vendor OUI                                : 00:00:00
        Vendor PN                                 : QT-SFP-M
        Vendor rev                                : B
        Option values                             : 0x00 0x10
        Option                                    : TX_DISABLE implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : REDACTED
        Date code                                 : 220531
        Optical diagnostics support               : No
mii-tool -vvv output:
Code:
Using SIOCGMIIPHY=0x8947
enp2s0f0: negotiated 1000baseT-FD flow-control, link ok
  registers for MII PHY 6:
    1140 796d 0141 0cc2 0de1 cde1 000d 2801
    41e8 0e00 7800 0000 0000 0000 0000 f000
    0878 ac4c 0000 1c40 0878 0000 0000 0000
    4100 0000 000a 8084 0000 0000 0000 0000
  product info: Yukon-EC 88E1111 rev 2
  basic mode:   autonegotiation enabled
  basic status: autonegotiation complete, link ok
  capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
  advertising:  1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
  link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
QSFPTEK QT-SFP-T CO SFP 10/100/1000 RJ45 SGMII:
Code:
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x00 (unknown or unspecified)
        Transceiver codes                         : 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x01
        Transceiver type                          : Ethernet: 1000BASE-T
        Transceiver type                          : Extended: 100G AOC or 25GAUI C2M AOC with worst BER of 5x10^(-5)
        Encoding                                  : 0x01 (8B/10B)
        BR, Nominal                               : 1300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 0m
        Length (62.5um)                           : 0m
        Length (Copper)                           : 100m
        Length (OM3)                              : 0m
        Laser wavelength                          : 0nm
        Vendor name                               : QSFPTEK
        Vendor OUI                                : 00:00:00
        Vendor PN                                 : QT-SFP-T
        Vendor rev                                : A
        Option values                             : 0x00 0x10
        Option                                    : TX_DISABLE implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : REDACTED
        Date code                                 : 220321
        Optical diagnostics support               : No
mii-tool -vvv output:
Code:
Using SIOCGMIIPHY=0x8947
enp2s0f1: negotiated 1000baseT-FD flow-control, link ok
  registers for MII PHY 6:
    1140 796d 0141 0cc2 0de1 cde1 000f 2801
    4306 0e00 3800 0000 0000 0000 0000 f000
    0878 ac0c 0000 1c40 0878 0000 0000 0000
    4100 0000 0002 8084 0000 0000 0000 0000
  product info: Yukon-EC 88E1111 rev 2
  basic mode:   autonegotiation enabled
  basic status: autonegotiation complete, link ok
  capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
  advertising:  1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
  link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
FS SFP-10G-T 10GBASE-T-SFP+ :
Code:
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 10G Ethernet: 10G Base-SR
        Encoding                                  : 0x06 (64B/66B)
        BR, Nominal                               : 10300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 30m
        Length (62.5um)                           : 10m
        Length (Copper)                           : 0m
        Length (OM3)                              : 0m
        Laser wavelength                          : 850nm
        Vendor name                               : FS
        Vendor OUI                                : 00:00:00
        Vendor PN                                 : SFP-10G-T
        Vendor rev                                : 10
        Option values                             : 0x00 0x1a
        Option                                    : RX_LOS implemented
        Option                                    : TX_FAULT implemented
        Option                                    : TX_DISABLE implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : REDACTED
        Date code                                 : 191114
        Optical diagnostics support               : Yes
        Laser bias current                        : 0.000 mA
        Laser output power                        : 0.0000 mW / -inf dBm
        Receiver signal average optical power     : 0.0000 mW / -inf dBm
        Module temperature                        : 9.75 degrees C / 49.56 degrees F
        Module voltage                            : 3.3032 V
        Alarm/warning flags implemented           : Yes
        Laser bias current high alarm             : Off
        Laser bias current low alarm              : Off
        Laser bias current high warning           : Off
        Laser bias current low warning            : Off
        Laser output power high alarm             : Off
        Laser output power low alarm              : Off
        Laser output power high warning           : Off
        Laser output power low warning            : Off
        Module temperature high alarm             : Off
        Module temperature low alarm              : Off
        Module temperature high warning           : Off
        Module temperature low warning            : Off
        Module voltage high alarm                 : Off
        Module voltage low alarm                  : Off
        Module voltage high warning               : Off
        Module voltage low warning                : Off
        Laser rx power high alarm                 : Off
        Laser rx power low alarm                  : Off
        Laser rx power high warning               : Off
        Laser rx power low warning                : Off
        Laser bias current high alarm threshold   : 0.000 mA
        Laser bias current low alarm threshold    : 0.000 mA
        Laser bias current high warning threshold : 0.000 mA
        Laser bias current low warning threshold  : 0.000 mA
        Laser output power high alarm threshold   : 0.0000 mW / -inf dBm
        Laser output power low alarm threshold    : 0.0000 mW / -inf dBm
        Laser output power high warning threshold : 0.0000 mW / -inf dBm
        Laser output power low warning threshold  : 0.0000 mW / -inf dBm
        Module temperature high alarm threshold   : 75.00 degrees C / 167.00 degrees F
        Module temperature low alarm threshold    : -5.00 degrees C / 23.00 degrees F
        Module temperature high warning threshold : 70.00 degrees C / 158.00 degrees F
        Module temperature low warning threshold  : 0.00 degrees C / 32.00 degrees F
        Module voltage high alarm threshold       : 3.6300 V
        Module voltage low alarm threshold        : 2.9700 V
        Module voltage high warning threshold     : 3.4600 V
        Module voltage low warning threshold      : 3.1300 V
        Laser rx power high alarm threshold       : 0.0000 mW / -inf dBm
        Laser rx power low alarm threshold        : 0.0000 mW / -inf dBm
        Laser rx power high warning threshold     : 0.0000 mW / -inf dBm
        Laser rx power low warning threshold      : 0.0000 mW / -inf dBm
Cisco MGBT1 RJ45 1G:
Code:
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x80 (reserved or unknown)
        Transceiver codes                         : 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x20
        Transceiver type                          : Ethernet: 1000BASE-T
        Transceiver type                          : Extended: 100G SWDM4
        Encoding                                  : 0x01 (8B/10B)
        BR, Nominal                               : 1000MBd
        Rate identifier                           : 0x20 (reserved or unknown)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 0m
        Length (62.5um)                           : 0m
        Length (Copper)                           : 100m
        Length (OM3)                              : 320m
        Laser wavelength                          : 8224nm
        Vendor name                               : CISCO
        Vendor OUI                                : 00:01:6c
        Vendor PN                                 : MGBT1
        Vendor rev                                : C
        Option values                             : 0x20 0x10
        Option                                    : TX_DISABLE implemented
        Option                                    : Power level 3 requirement
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : REDACTED
        Date code                                 : 121118__
        Optical diagnostics support               : No
HiFiber CAB-10GSFP-P 10Gbase-CU DAC:
Code:
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x21 (Copper pigtail)
        Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00
        Transceiver type                          : Passive Cable
        Encoding                                  : 0x00 (unspecified)
        BR, Nominal                               : 10300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 0m
        Length (62.5um)                           : 0m
        Length (Copper)                           : 1m
        Length (OM3)                              : 0m
        Passive Cu cmplnce.                       : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]
        Vendor name                               : OEM
        Vendor OUI                                : 00:40:20
        Vendor PN                                 : SFP-H10GB-CU1M
        Vendor rev                                : R
        Option values                             : 0x00 0x00
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : REDACTED
        Date code                                 : 220406
        Optical diagnostics support               : No
10Gtek CAB-1GSFP-P 1GBase-CU DAC:
Code:
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : Ethernet: 1000BASE-SX
        Encoding                                  : 0x01 (8B/10B)
        BR, Nominal                               : 1300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 550m
        Length (62.5um)                           : 270m
        Length (Copper)                           : 0m
        Length (OM3)                              : 0m
        Laser wavelength                          : 850nm
        Vendor name                               : OEM
        Vendor OUI                                : 00:90:65
        Vendor PN                                 : GLC-SX-MMD     _
        Vendor rev                                : _e
        Option values                             : 0x00 0x12
        Option                                    : RX_LOS implemented
        Option                                    : TX_DISABLE implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : REDACTED
        Date code                                 : 211103
        Optical diagnostics support               : No
Note: You might ask yourself why the 1G DAC cable reports itself as "1000-BX" type instead of "passive cable" like the 10G DAC cable does. This is because DAC cables do not exist in the digital diagnostic monitoring specification for 1G SFP modules (see SFF-805 Table D.4).
It's only in the SFF-8472 specification for Management Interface for SFP+ Rev 12.4.
I guess they just entered some random values for compatibility.

I'm not so sure if this was correct. I don't actually know why this is.
 

artorias

New Member
May 8, 2022
12
9
3
My next goal is to try and find out, how to disable two of the four ports to maybe safe some power.
Stay tuned....
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
The amount you save might very well be within the margin of error of your measuring method. Most of the power consumption will be the ASIC when it's slinging packets or any plugged in optics/DACs.
 

artorias

New Member
May 8, 2022
12
9
3
I'm back!
Ok, so disabling ports of the i350-F4 via EEPROM does not make any difference whatsoever reagarding power consumption.
However, I got some nice deals on a bunch of other cards, which means the quest for the lowest power consumption DIY router continues.

Here are my power measurements of the i210AS, i210AT, i350-F4, i350-T2 and x520-DA2/SR2.

The measurements are taken with a Shelly Plug S. The measurements are a little inconsistent/inaccurate. I also can't really untangle the additional system power draw during my testing from these measurements. Additional CPU load, changing of power states of the CPU and PCIe devices may occur and can't be isolated and substracted. Sometimes the CPU kicks in and adds either 0.5W or 1W randomly and some time later doesn't anymore. I also had to estimate some values since I don't have the necessary equipment. The measurements should be seen more as estimates than anything else. However, I think we can still see some patterns and draw a conclusion.

tl;dr:
Medium: DAC < Fiber < "native RJ-45" < RJ-45 SFP
Controller: i210 < i350AM2 < i350AM4 < 82599 (x520)

Two i210AS cards:

"card only" / total system power2x i210AS with DAC2x i210AS with Fiber2x i210AS with RJ-45 SFP
base system, no card0.0W / 3.9W0.0W / 3.9W0.0W / 3.9W
idle, no link1.2W / 5.1W1.2W / 5.1W1.2W / 5.1W
idle, 1 link, no traffic1.2W / 5.1W1.7W / 5.6W2.3W / 6.2w
idle, 2 links, no traffic1.3W / 5.2W2.4W / 6.3W4.1W - 4.7W / 8.0W - 8.6W
iperf, 1 port, full speed unidirectional (1Gb/s)3.4W / 7.3W4.0W / 7.9w5.0W / 8.9W
iperf, 1 port, full speed bidirectional (2Gb/s)3.9W / 7.8w4.5W / 8.4W5.0W / 8.9W
iperf, 2 ports, full speed unidirectional (2Gb/s)5,7W / 9.6W (estimated)7.0W / 10.9W (estimated)6.7W / 10.6W
iperf, 2 ports, full speed bidirectional (4Gb/s)6.5W / 10.4W (estimated)8.0W / 11.9W (estimated)7.9W - 8.3W / 11.8W - 12.2W

EDIT: Added i210-T1

Two i210-T1 cards:

"card only" / total system power2x i210AT (RJ-45)
base system, no card0.0W / 3.9W
idle, no link0.9W / 4.8W
idle, 1 link, no traffic1,2W / 5.1W
idle, 2 links, no traffic1,9W / 5.8W
iperf, 1 port, full speed unidirectional (1Gb/s)3.9W / 7.8W
iperf, 1 port, full speed bidirectional (2Gb/s)4.6W / 8.5W
iperf, 2 ports, full speed unidirectional (2Gb/s)4.7W / 8.6W
iperf, 2 ports, full speed bidirectional (4Gb/s)6.0W / 9.9W

i350-T2 (i350AM2):

"card only" / total system poweri350-T2 (RJ-45)
base system, no card0.0W / 3.9W
idle, no link1.5W / 5.4W
idle, 1 link, no traffic2.3W / 6.2W
idle, 2 links, no traffic3.2W / 7.1W
iperf, 1 port, full speed unidirectional (1Gb/s)4.1W / 8.0W
iperf, 1 port, full speed bidirectional (2Gb/s)5.1W / 9.0w
iperf, 2 ports, full speed unidirectional (2Gb/s)6.0W / 9.9W
iperf, 2 ports, full speed bidirectional (4Gb/s)7.3W / 11.2W

EDIT: Note: I don't know what was wrong with this card, but I tested again and the same thing happened. 1 port bidirectional works no problem (940+/940+) but when using both ports the card drops to around 770 on both ports in unidirectional test and to 700/650 in both directions in bidirectional tests on both ports.

i350-F2 (i350-AM2):

"card only" / total system poweri350-F2 with DACi350-F2 with Fiberi350-F2 with RJ-45 SFP
base system, no card0.0W / 5.6W0.0W / 5.6W0.0W / 5.6W
idle, no link1.5W / 7.1W1.9W / 7.4W1.9W / 7.4W
idle, 1 link, no traffic1.9W / 7.4W2.5W / 8.1W3.5W / 9.1W
idle, 2 links, no traffic1.9W / 7.4W3.2W / 8.8W5W / 10.6W
iperf, 1 port, full speed unidirectional (1Gb/s)3.8W / 9.4W4.8W / 10.4W6.0W / 11.6W
iperf, 1 port, full speed bidirectional (2Gb/s)4.5W / 10.1W5.3W / 10.9W6.1W / 11.7W
iperf, 2 ports, full speed unidirectional (2Gb/s)5.8W / 11.4W (estimated)7.1W / 12.7W (estimated)8.6W / 13.9W
iperf, 2 ports, full speed bidirectional (4Gb/s)7.2W / 12.8W (estimated)8.8W / 14.4W (estimated)9.4W / 15.0W

i350-F4 (i350AM4):

"card only" / total system poweri350-F4 with DACi350-F4 with Fiberi350-F4 with RJ-45 SFP
base system, no card0.0W / 3.9W0.0W / 3.9W0.0W / 3.9W
idle, no link3.3W / 7.2W3.3W / 7.2W3.3W / 7.2W
idle, 1 link, no traffic3.3W / 7.2W3.7W / 7.6W4.6W / 8.5W
idle, 2 links, no traffic3.3W / 7.2W4.3W / 8.2W6.0W / 9.9W
iperf, 1 port, full speed unidirectional (1Gb/s)4.6W - 5.3W / 8.5W - 9.2W5.5W / 9.4W6.6W - 7.1W / 10.5W - 11.0W
iperf, 1 port, full speed bidirectional (2Gb/s)5.2W - 6.4W / 9.1W - 10.3W6.8W / 10.7W7.3W / 11.2W
iperf, 2 ports, full speed unidirectional (2Gb/s)5.9W - 7.3W / 9.8W - 11.2W (estimated, questionable)7.9W / 11.8W (estimated)8.5W / 12.4W
iperf, 2 ports, full speed bidirectional (4Gb/s)7.1W - 9.5W/ 11.0W - 13.4W (estimated, questionable)9.3W / 13.2W (estimated)10W / 13,9W
3 and 4 portsno equipmentno equipmentno equipment

EDIT: Did one additional test to try to max out the card with the limited gear I have:
"card only" / total system poweri350-F4 with 1x DAC + 1x Fiber + 2x RJ-45 SFP
base system, no card0.0W /5.6W
idle, no link3.2W / 8.8W
idle, 4 links6.6W / 12.2W
iperf, 4 ports, full speed unidirectional (4Gb/s)*9.9W / 15.5W
iperf, 4 ports, full speed bidirectional (8Gb/s)*12.1W / 17.7W

* note on the bidirectional tests: I think I was maxing out the PCIe 2.0x1 lane, the card was connected to. The speeds during the tests dropped to 680 on the unidirectional test and to 600/800 in the bidirectional tests.

x520 (82599) (1G mode):

This one is interesting. When in idle after a reboot with no link, the cards seems to be in 10G mode. When you then establish a 1G link on both ports and remove the medium, the card stays in 1G mode and draws way less power in idle. It's almost the same as the i350-F4, in some cases even better. It also can only drop down to 1G with fiber and RJ-45 SFPs but not with DAC.

EDIT: The card can do 1G DAC...post will follow

"card only" / total system powerx520 with 1G DAC*x520 with Fiberx520 with RJ-45 SFP
base system, no card0.0W / 5.4W0.0W / 3.9W0.0W / 3.9W
idle, no link3.0W (1G) - 4.7W (10G) / 8.4W (1G) - 10.1W (10G)2.9W (1G) - 4.4W (10G) / 6.8W (1G) - 8.3W (10G)2.9W (1G) - 4.4W (10G) / 6.8W (1G) - 8.3W (10G)
idle, 1 link, no traffic3.0W / 8.4W4.1W / 8.0W4.2W / 8.1W
idle, 2 links, no traffic3.0W / 8.4W4.5W / 8.4W5.3W / 9.2W
iperf, 1 port, full speed unidirectional (1Gb/s)4.9W / 10.3W5.6W / 9.5W6.8W / 10.6W
iperf, 1 port, full speed bidirectional (2Gb/s)5.7W / 11.1W6.3W / 10.2W6.8W / 10.7W
iperf, 2 ports, full speed unidirectional (2Gb/s)6.8W / 12.2W (estimated)7.5W / 11.4W (estimated)7.6W / 11.5W
iperf, 2 ports, full speed bidirectional (4Gb/s)8.4W / 13.8W (estimated but makes no sense)8.1W / 12W (estimated)9.4W / 13.3W

* These measurements were on another day. For some reason the system idle consumption was higher this time. Same system, no changes.


For more accurate stats about the power consumption, you might look into the datasheets sections of the i210 (11.5), i350 (11.5) and 82599 (11.4.1.1). However, these seem to take only the chip into consideration and not the rest of the PCIe Card.

Also, one thing I noticed is that the FS SFP-10G-T struggled during the bidirectional tests. Instead of 940/940 I was getting 940/220. Not the case with the other RJ45-SFPs. Maybe it's an issue with the 1G mode of the module.

Verdict

The differences are not as big as expected.

For my specific scenario, I will most likely go with the double i210AS for my FTTH connection with a fiber ONU/ONT SFP module in one card and a DAC to my switch in the other one. In the end, the connection will idle like 80% of the day and the double i210ASs have the lowest consumption in that case. I also don't need the additional functions like SR-IOV or the additional queues (I only have a quad core) of the i350 and 82599 (see feature comparison here (section 1.5)and here).

The power savings are not that big, so I guess one should rather go by availability and price. These i210ASs and I350-F2/4s are most of the time more expensive than x520s and i350-T2/4s. It would take a long time to amortize.

But hey, for the low power enthusiasts, I suggest go for the double i210AS for a scenario like mine or double i210-T1 or i350-T2 if you have DSL or DOCSIS and no SFP on your switch.
 
Last edited:
  • Like
Reactions: istamov and arnbju

artorias

New Member
May 8, 2022
12
9
3
I updated the last post with the missing measurements. I couldn't get this done earlier because my DAC cable broke somehow during my testing. I don't know if it was a power issue or something like that but somehow the first two bytes of one of the DAC cable connector's eeprom were erased, which caused it to not be recognized by any NIC anymore. I thought I'd need to buy a new one but I also didn't want to throw away the cable. I discovered this blogpost, which describes how to hook up an SFP module's eeprom to a Raspberry Pi and reprogramm it. With the Cisco TwinGig adapter and some soldering this was way easier than I thought. I also looked around for SFP breakout boards or programmers and holy moly are they expensive. The TwinGig adapter for 5-10 bucks was waaayy cheaper. Instead of a RPi you could also use a cheap USB-to-I2C adapter.
After connecting to the eeprom and reprogramming it, the DAC cable was working again like it was new. Thank God for 3rd party SFPs with non-writeprotected eeprom and no need for vendor passwords and checksums.

Anyway, I attached a zip of a collection of eeprom images from various cards, I got my hands on during this saga. I don't care about "leaking" my cards' serial numbers and mac addresses. The images are from these cards:
  • Fujitsu D2755-A11 GS2 (Intel X520-DA2 equivalent) - v30e0.ffff
  • 10gtek i210-GE-1S-X1 (i210-SFP) - v3.20
  • Digitus DN-10160 (i210-SFP) - v3.25
  • 10gtek i350-GE-4S-X4 (i350-4x SFP) - v1.63
  • noname i350-2x-SFP - v1.52
  • Kalea Informatique i350AF2 (i350-2xSFP) - v1.52
  • Silicom PR2G4SFPi35L (i350-2x-SFP) - v1.63
  • Intel i350T2V2 (i350-2xRJ-45) - v1.63
  • Intel i210-T1 (i210-RJ-45) - v3.16
For the i210 1xSFP and i350 2xSFP variants, there are cards with different image versions. You can find the fixes in the respective controller's Specification Update document (i210, i350). From what I can tell, these are not crucial fixes.

If you looked at my last post again, you might have noticed that I added measurements for the x520 card with 1G DAC cables. I got it to work with some workaround and I don't understand why this does not work out of the box. However, I'll write that down tomorrow and finally wrap this whole thing up...
 

Attachments

Last edited:

artorias

New Member
May 8, 2022
12
9
3
Making the x520 Work with 1G DAC

There a two methods to do this:
1. Make the default operation mode 1G instead of 10G by modifying the eeprom
2. Patch the driver to bolt on multispeed DAC support

I'll explain both.

Method 1: EEPROM Modification

I use the first method, since I don't care about 10G operation here. I also don't really know how this changes the general interaction with 10G gear. The card will link at 1G by default, if possible i guess. When using a multispeed 1G/10G fiber SFP, it first links at 1G but you can set the speed to 10G via ethtool with e.g. ethtool -s ifname speed 10000. I tested it by connecting both ends of the cable to the card. So 10G operation is not permanently disabled, you just have to set the speed manually. I don't know which speed will be chosen if you connect to a 10G only port on the other end.
With this method, the card will link with 1G gear via DAC.
Doing this works the same way I did it with the i350-F4. Dump the eeprom, modify it, flash it.

Some background:
The 82599's operation mode is set through the AUTOC.LMS PCIe register. The default state of this register can be set via eeprom. During operation of the card, the state of the register is handled by the ixgbe driver.

To know what we have to edit in the eeprom dump, we take a look at the 82599 controller datasheet. Section 6.1 documents the flash map. We need to edit the MAC 0&1 Module field. However, the map only says that the addresses 0x09 and 0x0A contains a pointer to the field and is not the real address of the wanted field. The value at those addresses is the absolute end address of the the MAC 0 or 1 module field. This addrees might be different for your specific card depending on how the OEM programmed it. In my case the addresses for MAC 0&1 were 0x1C7 and 0x1CF. You can also find them by looking out for your MAC address. There are two lines that begin the MAC address followed by the two lines that we are looking for.

Looking at section 6.3.8 tells us the our word is at offset 0x1. In my case the addresses are 0x1C0 and 0x1C8. The values here were 0x6084 (01100000 10000100) for both.

The modification:
Section 6.3.8.2 bits 15:13 are the ones we need to change. Currently it is set to "011b = 10 Gb/s serial link (no auto-negotiation). Supports SFI without backplane auto-negotiation." We set it to 000b = 1 Gb/s link (no auto-negotiation)."

The words are now 0x0084 (00000000 10000100).

You can now flash the modified eeprom image back to the card via the tools mentioned in my post above and reboot. The card should now be in 1G mode by default.

-----------------------

Method 2: Driver Modification


The second method doesn't need eeprom modification. The card's default operation mode is 10G here. The driver can be patched with two modifications. The first one justs adds 1G capabilities for all types of DAC cables. This alone will not make the card establish a link to 1G gear automatically. You still need to set the speed manually via ethtool -s ifname speed 1000 duplex full.

With the second modification, we make the driver treat DAC cables like multispeed fiber SFPs. This makes the card "auto-negotiate" (not in the IEEE sense) 10G and 1G links, by first trying 10G and then dropping down to 1G in software. You can still set the speed via the ethtool command mentioned above if needed.

What I have to warn about here is that I can't test all the implications this has for all the different types of DAC cables (passive vs. active). By adding this modification, the driver will trigger some functions that control the laser for fiber SFPs. In the case of my passive DAC cable, these functions will essentially do nothing, since there are no lasers or additional electronics on the DAC connectors. I don't know what happens in the case of active DAC cables or if there is even anything bad that could happen.

Note that these mods could be a bit janky. This is the first time I touched C and I'm not too familiar with the ixgbe driver to be able to tell, if this is the proper or clean way to do it.

Anyway, here are the modifications:

Download the latest ixgbe linus driver here.

Modification 1: Just add 1G capabilities for DAC:
In ixgbe_82599.c in function ixgbe_get_link_capabilities_82599() around line 481, add this chunk of code to make it look like this:

C:
...
/**
 * ixgbe_get_link_capabilities_82599 - Determines link capabilities
 * @hw: pointer to hardware structure
 * @speed: pointer to link speed
 * @autoneg: true when autoneg or autotry is enabled
 *
 * Determines the link capabilities by reading the AUTOC register.
 **/
s32 ixgbe_get_link_capabilities_82599(struct ixgbe_hw *hw,
                      ixgbe_link_speed *speed,
                      bool *autoneg)
{
/** added for orientation
 * ...
 *    if (hw->phy.multispeed_fiber) {
 * ...
 *    }
**/
    
    if (hw->phy.sfp_type == ixgbe_sfp_type_da_cu_core0 ||
        hw->phy.sfp_type == ixgbe_sfp_type_da_cu_core1) {
        *speed |= IXGBE_LINK_SPEED_10GB_FULL |
                  IXGBE_LINK_SPEED_1GB_FULL;
        *autoneg = true;
    }

out:
    return status;
}
....
This adds the reporting of 1G support.

In ixgbe_ethtool.c in function ixgbe_set_link_ksettings(), around line 747, add this chunk of code to make it look like this:

C:
...
static int ixgbe_set_link_ksettings(struct net_device *netdev,
                    const struct ethtool_link_ksettings *cmd)
{
    ...

    if (hw->phy.media_type == ixgbe_media_type_copper ||
        hw->phy.multispeed_fiber ||
            hw->phy.sfp_type == ixgbe_sfp_type_da_cu_core0 ||
            hw->phy.sfp_type == ixgbe_sfp_type_da_cu_core1))  {
        /*
         * this function does not support duplex forcing, but can
         * limit the advertising of the adapter to the specified speed
         */
...
This will enable setting the speed via ethtool.

Modification 2: Also add "auto-negotiation"

In ixgbe_phy.c in function ixgbe_identify_sfp_module_generic(), around line 1402, add this chunk of code to make it look like this:

C:
...

/**
 * ixgbe_identify_sfp_module_generic - Identifies SFP modules
 * @hw: pointer to hardware structure
 *
 * Searches for and identifies the SFP module and assigns appropriate PHY type.
 **/
s32 ixgbe_identify_sfp_module_generic(struct ixgbe_hw *hw)
{

...

        /* Determine if the SFP+ PHY is dual speed or not. */
        hw->phy.multispeed_fiber = false;
        if (((comp_codes_1g & IXGBE_SFF_1GBASESX_CAPABLE) &&
           (comp_codes_10g & IXGBE_SFF_10GBASESR_CAPABLE)) ||
           ((comp_codes_1g & IXGBE_SFF_1GBASELX_CAPABLE) &&
           (comp_codes_10g & IXGBE_SFF_10GBASELR_CAPABLE)) ||
           (IXGBE_SFF_DA_PASSIVE_CABLE)))
            hw->phy.multispeed_fiber = true;
...
Now just run make install to compile and install the driver and rmmod ixgbe; modprobe ixgbe to reload it. That's it.

------

Discussion

Now that we proved that this works, the only question remaining is: Why does the ixgbe driver not support 1G DAC operation out of the box?

The datasheet lists 1000base-CX compatibility (datasheet section 1.8 References) and the official Intel DAC cable XDACBL3M-C also lists 1000base-CX compatibility according to this post were an eeprom dump of the cable is posted. The 6th word is 0x04, which indicates 1000base-CX Ethernet compliance, according to the specification SFF-8472 Rev 12.4, Table 5-3 and Table 5-5. The card can do it, as proven here and can also "auto-negotiate" by (ab)using the multispeed fiber functionality. I think proper implementation could be realized relatively hassle-free from my outsider's point of view. I don't even need the fancy "auto-negotiation". Setting the speed in ethtool would already be enough. For the "auto-negotiation" of link speed, the method used in the function for multispeed fiber SFPs works. One could modify it or add a proper function for DAC without the laser stuff. I'm not an expert, I don't know if I got all of this correct.

I would propose this in the Intel forum but I don't have much hope for any changes. The problem is that I use a Fujitsu x520 variant as well as a third party DAC cable. Even if this doesn't actually matter, from what I have seen there they won't give any support if you can't prove that you have 100% original Intel gear. Getting a card is not the issue but I have not seen the original Intel XDACBL1M/3M/5M cable for a reasonable price anywhere. Also, I can't tell what implications such a change can have and can imagine that they would prefer not to change a battle-tested driver like this one for an edge-case and a feature nobody seems to have cried about before in over 10 years.
 
  • Like
Reactions: istamov and itronin