I've designed an adapter to use HPE FlexibleLOM NICs in normal full height PCIe slots. The advantage of HPE FlexibleLOM NICs is that they are very inexpensive since they can usually only be used with HP servers.
JoshDi did already post a short thread on this earlier, but I thought you might enjoy a more complete writeup.
First of all: Who is this guide for?
This guide is intended for enthusiasts that take interest in fast home networking. I've only tested this guide on Linux, thus I'd advice to have some level of familiarity with Linux and be comfortable using a shell. Additionally soldering skills may be required if you do not get your adapter premade by someone.
FlexibleLOM
HPE FlexibleLOM is both an electrical and a physical form factor specification. The height of a FlexibleLOM card is somewhere inbetween a half-height and a full-height PCIe card. Physically FlexibleLOM cards use a PCIe x8 connector to interface with the host system. However, electrically the interface is incompatible, due to a modified pinout. However, on the protocol layer the card is still using PCIe. Thus the interface can be adapted to PCIe through a passive adapter.
The Adapter
The adapter itself is a simple PCB that adapts the FlexibleLOM pinout to PCIe x8. The design is fully open source and the design files are freely available on my GitHub.
Supported NICs
The adapter should support all FlexibleLOM NICs. I've personally tested them with HP 649283-B21 (dual-port QSFP+) and HP 682625-001 (single-port SFP+). Those can be had cheaply on ebay. However, all NICs that are available cheaply are usually configured for Infiniband only. Fortunately they seem to be absolutely identical to their Ethernet counterparts and can thus be flashed with Ethernet NIC firmware, converting them to Ethernet NICs.
Firmware
First of all check if your FlexibleLOM card does actually need to be flashed with other firmware. Sometimes even cards sold as Infiniband are actually Ethernet cards. If your do already see an Ethernet interface you do not need to flash your card.
If your card is indeed Infiniband firmware required depends on the adapter used. Ideally your FlexibleLOM Infiniband card has a sister model that is an Ethernet or VPI version of this card. Then you can simply download the firmware from the HPE knowledge base/service center directly. Just search for the partnumber of the sister card there.
To flash a card with new firmware install the Mellanox firmware tools first. Do this now.
Custom firmware
This section is only relevant for your card if there is no sister model available. Skip this section if you have already found firmware for your card.
The firmware customization procedure involves a few steps:
To build custom firmware for your card you will need a special, customizable firmware package from Mellanox. Mellanox used to release this until a few versions ago, the file you are looking for is called
Create a new directory,
2. Install OEM Mellanox firmware tools
The Mellanox firmware tools contain a special installation package called
3. Dump and backup current firmware/config
Make sure the FlexibleLOM card is installed in your system and verify you can see it in
Verify your card was actually picked up:
If so you can now dump the current firmware and firmware config:
I'd suggest backing up the .bin file in case you ever want to restore the card to original configuration.
4. Modify firmware config
Next up is enabling Ethernet support.
Copy the firmware configuration to a new configuration file
Now open
The config file is in ini format and contains multiple sections. There are two options we would like to modify:
As the name suggests the
There is a chance this option is not even specified in your config yet. If so add it to the
The second option controls wether SR-IOV is enabled. Unless you know you need SR-IOV I'd suggest disabling this feature, since some mainboards do not support SR-IOV and have trouble with the resulting giant PCI BAR sizes. Set
or add it to the
Now save the config.
5. Build firmware image
The
6. Verify firmware image
To make sure your firmware image is actually bootable you should quickly veriy it with flint:
Flashing
At this point you should have a firmware image for your card. If you have not started mst yet, do it now:
Now you can flash the firmware image to your card:
Replace
If flashing fails you might have to add
Now reboot your system. It should come with the network adapter detected.
Troubleshooting
Card seems bricked
Whenver you can not see the card in
Network interfaces are not showing up
When you do not see any network interfaces but can still see the card in
If your card is not showing up it is more than likely just not configured correctly. Some cards are Mellanox VPI cards. That means they can be configured to support different protocols on the fly without reflashing the firmware. To configure your card for Ethernet do the following:
Start mst:
Configure the cards ports for Ethernet:
If your card has two ports configure the second port for Ethernet, too:
Now reboot and the system should come up with the expected Ethernet interfaces.
2. PCI BAR is too large
Depending on your host system the PCI BAR required for the card might be too large to be supported. This is an issue in particular when SR-IOV is still enabled. Have a look through
3. Card has Infiniband firmware flashed
If none of the above helps your card might still simply have the Infiniband firmware flashed. Just carefully redo the configuration and flashing procedure.
I Hope you found this writeup useful or at least somewhat interesting
Tobias
JoshDi did already post a short thread on this earlier, but I thought you might enjoy a more complete writeup.
First of all: Who is this guide for?
This guide is intended for enthusiasts that take interest in fast home networking. I've only tested this guide on Linux, thus I'd advice to have some level of familiarity with Linux and be comfortable using a shell. Additionally soldering skills may be required if you do not get your adapter premade by someone.
FlexibleLOM
HPE FlexibleLOM is both an electrical and a physical form factor specification. The height of a FlexibleLOM card is somewhere inbetween a half-height and a full-height PCIe card. Physically FlexibleLOM cards use a PCIe x8 connector to interface with the host system. However, electrically the interface is incompatible, due to a modified pinout. However, on the protocol layer the card is still using PCIe. Thus the interface can be adapted to PCIe through a passive adapter.
The Adapter
The adapter itself is a simple PCB that adapts the FlexibleLOM pinout to PCIe x8. The design is fully open source and the design files are freely available on my GitHub.
Supported NICs
The adapter should support all FlexibleLOM NICs. I've personally tested them with HP 649283-B21 (dual-port QSFP+) and HP 682625-001 (single-port SFP+). Those can be had cheaply on ebay. However, all NICs that are available cheaply are usually configured for Infiniband only. Fortunately they seem to be absolutely identical to their Ethernet counterparts and can thus be flashed with Ethernet NIC firmware, converting them to Ethernet NICs.
Firmware
First of all check if your FlexibleLOM card does actually need to be flashed with other firmware. Sometimes even cards sold as Infiniband are actually Ethernet cards. If your do already see an Ethernet interface you do not need to flash your card.
If your card is indeed Infiniband firmware required depends on the adapter used. Ideally your FlexibleLOM Infiniband card has a sister model that is an Ethernet or VPI version of this card. Then you can simply download the firmware from the HPE knowledge base/service center directly. Just search for the partnumber of the sister card there.
To flash a card with new firmware install the Mellanox firmware tools first. Do this now.
Custom firmware
This section is only relevant for your card if there is no sister model available. Skip this section if you have already found firmware for your card.
The firmware customization procedure involves a few steps:
- Obtain customizable firmware package
- Install OEM Mellanox firmware tools
- Dump and backup current firmware/config
- Modify firmware config
- Build firmware image
- Verify firmware image
To build custom firmware for your card you will need a special, customizable firmware package from Mellanox. Mellanox used to release this until a few versions ago, the file you are looking for is called
ConnectX3-rel-2_36_5000-web.tgz
and can still be found on the internet.Create a new directory,
cd
to it and extract the archive there.2. Install OEM Mellanox firmware tools
The Mellanox firmware tools contain a special installation package called
mft-oem
. To build customized firmware images you will need to install this package in addition to the normal mft
package.3. Dump and backup current firmware/config
Make sure the FlexibleLOM card is installed in your system and verify you can see it in
lspci
. Next start mst
through
Bash:
[user@host ~]$ sudo mst start
Code:
[user@host ~]$ sudo mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4099_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:05:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 01
/dev/mst/mt4099_pci_cr0 - PCI direct access.
domain:bus:dev.fn=0000:05:00.0 bar=0xf7900000 size=0x100000
Chip revision is: 01
Bash:
[user@host ~]$ sudo flint -d /dev/mst/mt4099_pciconf0 ri original_ib_firmware.bin
[user@host ~]$ sudo flint -d /dev/mst/mt4099_pciconf0 dc original_ib_config.ini
4. Modify firmware config
Next up is enabling Ethernet support.
Copy the firmware configuration to a new configuration file
Bash:
[user@host ~]$ cp original_ib_config.ini modified_eth_config.ini
modified_eth_config.ini
in your favourite editor.The config file is in ini format and contains multiple sections. There are two options we would like to modify:
eth_xfi_en
and sriov_en
both of them live in the [HCA]
section.As the name suggests the
eth_xfi_en
option controls wether Ethernet is enabled. Set it to true:
Code:
eth_xfi_en = true
[HCA]
section.The second option controls wether SR-IOV is enabled. Unless you know you need SR-IOV I'd suggest disabling this feature, since some mainboards do not support SR-IOV and have trouble with the resulting giant PCI BAR sizes. Set
Code:
sriov_en = false
[HCA]
section in case that config option is not present yet.Now save the config.
5. Build firmware image
The
mft-oem
package installed earlier contains a tool called mlxburn
. Use the firmware package you extracted and the modified config to build an Ethernet firmware image:
Bash:
[user@host ~]$ mlxburn -fw fw-ConnectX3-rel.mlx -conf modified_eth_config.ini-wrimage modified_eth_firmware.bin
modified_eth_firmware.bin
will now contain your firmware image6. Verify firmware image
To make sure your firmware image is actually bootable you should quickly veriy it with flint:
Bash:
[user@host ~]$ flint -i modified_eth_firmware.bin verify
Flashing
At this point you should have a firmware image for your card. If you have not started mst yet, do it now:
Bash:
[user@host ~]$ sudo mst start
Bash:
[user@host ~]$ sudo flint -i <fimware file> -d /dev/mst/mt4099_pciconf0 -allow_psid_change burn
<fimware file>
with your firmware filename.If flashing fails you might have to add
-nofs
to the flash command (depends on currently running and to be flashed firmware image):
Bash:
[user@host ~]$ sudo flint -i <fimware file> -d /dev/mst/mt4099_pciconf0 -nofs -allow_psid_change burn
Troubleshooting
Card seems bricked
Whenver you can not see the card in
lspci
anymore you can force the card into a bootloader mode that will allow you to flash a new firmware file as described in Flashing. To do so you have to find the recovery jumper on your card. On some HPE cards that jumper is labeled FLASH PRSNT
. Remove the card from your system, short the two pins of that jumper, reinstall the card and boot the system with the two pins shorted. Then you will be able to flash a new firmware image. Note that the jumper might not be fitted, meanin you will need to find something conductive to jam into the throughholes. The following photo shows the jumper on a single-port SFP+ FlexibleLOM NIC.Network interfaces are not showing up
When you do not see any network interfaces but can still see the card in
lspci
this could be due to one of three reasons:- Card is not configured correctly
- PCI BAR is too large
- Card has Infiniband firmware flashed
If your card is not showing up it is more than likely just not configured correctly. Some cards are Mellanox VPI cards. That means they can be configured to support different protocols on the fly without reflashing the firmware. To configure your card for Ethernet do the following:
Start mst:
Bash:
[user@host ~]$ sudo mst start
Bash:
[user@host ~]$ sudo mlxconfig -d /dev/mst/mt4099_pciconf0 s LINK_TYPE_P1=2
Bash:
[user@host ~]$ sudo mlxconfig -d /dev/mst/mt4099_pciconf0 s LINK_TYPE_P2=2
2. PCI BAR is too large
Depending on your host system the PCI BAR required for the card might be too large to be supported. This is an issue in particular when SR-IOV is still enabled. Have a look through
dmesg
to identify such issues. If you see any PCI BAR allocation failures or the mlx4 driver complains about missing UAR you are most definetly having a PCI BAR size issue. If you do not want to disable SR-IOV search your BIOS/UEFI setup for an option to either configure PCI(e) BAR size or something like "above 4G IO memory". If that does not help reflash your card with a firmware that has SR-IOV disabled.3. Card has Infiniband firmware flashed
If none of the above helps your card might still simply have the Infiniband firmware flashed. Just carefully redo the configuration and flashing procedure.
I Hope you found this writeup useful or at least somewhat interesting
Tobias
Last edited: