10 and 40 Gb networking on the cheap ($5 - $10)

tsys

New Member
Oct 5, 2020
4
23
3
I've designed an adapter to use HPE FlexibleLOM NICs in normal full height PCIe slots. The advantage of HPE FlexibleLOM NICs is that they are very inexpensive since they can usually only be used with HP servers.

JoshDi did already post a short thread on this earlier, but I thought you might enjoy a more complete writeup.

First of all: Who is this guide for?
This guide is intended for enthusiasts that take interest in fast home networking. I've only tested this guide on Linux, thus I'd advice to have some level of familiarity with Linux and be comfortable using a shell. Additionally soldering skills may be required if you do not get your adapter premade by someone.

FlexibleLOM
HPE FlexibleLOM is both an electrical and a physical form factor specification. The height of a FlexibleLOM card is somewhere inbetween a half-height and a full-height PCIe card. Physically FlexibleLOM cards use a PCIe x8 connector to interface with the host system. However, electrically the interface is incompatible, due to a modified pinout. However, on the protocol layer the card is still using PCIe. Thus the interface can be adapted to PCIe through a passive adapter.

The Adapter
The adapter itself is a simple PCB that adapts the FlexibleLOM pinout to PCIe x8. The design is fully open source and the design files are freely available on my GitHub.

FlexibleLOM_adapter_small.jpg

Supported NICs
The adapter should support all FlexibleLOM NICs. I've personally tested them with HP 649283-B21 (dual-port QSFP+) and HP 682625-001 (single-port SFP+). Those can be had cheaply on ebay. However, all NICs that are available cheaply are usually configured for Infiniband only. Fortunately they seem to be absolutely identical to their Ethernet counterparts and can thus be flashed with Ethernet NIC firmware, converting them to Ethernet NICs.

FlexibleLOM_NIC_SFP+_small.jpg FlexibleLOM_NIC_2QSFP+_small.jpg

Firmware
First of all check if your FlexibleLOM card does actually need to be flashed with other firmware. Sometimes even cards sold as Infiniband are actually Ethernet cards. If your do already see an Ethernet interface you do not need to flash your card.

If your card is indeed Infiniband firmware required depends on the adapter used. Ideally your FlexibleLOM Infiniband card has a sister model that is an Ethernet or VPI version of this card. Then you can simply download the firmware from the HPE knowledge base/service center directly. Just search for the partnumber of the sister card there.

To flash a card with new firmware install the Mellanox firmware tools first. Do this now.

Custom firmware
This section is only relevant for your card if there is no sister model available. Skip this section if you have already found firmware for your card.

The firmware customization procedure involves a few steps:

  1. Obtain customizable firmware package
  2. Install OEM Mellanox firmware tools
  3. Dump and backup current firmware/config
  4. Modify firmware config
  5. Build firmware image
  6. Verify firmware image
1. Obtaining customizable firmware package
To build custom firmware for your card you will need a special, customizable firmware package from Mellanox. Mellanox used to release this until a few versions ago, the file you are looking for is called ConnectX3-rel-2_36_5000-web.tgz and can still be found on the internet.
Create a new directory, cd to it and extract the archive there.

2. Install OEM Mellanox firmware tools
The Mellanox firmware tools contain a special installation package called mft-oem. To build customized firmware images you will need to install this package in addition to the normal mft package.

3. Dump and backup current firmware/config
Make sure the FlexibleLOM card is installed in your system and verify you can see it in lspci. Next start mst through
Bash:
[user@host ~]$ sudo mst start
Verify your card was actually picked up:
Code:
[user@host ~]$ sudo mst status

MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4099_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:05:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 01
/dev/mst/mt4099_pci_cr0          - PCI direct access.
                                   domain:bus:dev.fn=0000:05:00.0 bar=0xf7900000 size=0x100000
                                   Chip revision is: 01
If so you can now dump the current firmware and firmware config:
Bash:
[user@host ~]$ sudo flint -d /dev/mst/mt4099_pciconf0 ri original_ib_firmware.bin
[user@host ~]$ sudo flint -d /dev/mst/mt4099_pciconf0 dc original_ib_config.ini
I'd suggest backing up the .bin file in case you ever want to restore the card to original configuration.

4. Modify firmware config
Next up is enabling Ethernet support.
Copy the firmware configuration to a new configuration file
Bash:
[user@host ~]$ cp original_ib_config.ini modified_eth_config.ini
Now open modified_eth_config.ini in your favourite editor.
The config file is in ini format and contains multiple sections. There are two options we would like to modify: eth_xfi_en and sriov_en both of them live in the [HCA] section.

As the name suggests the eth_xfi_en option controls wether Ethernet is enabled. Set it to true:
Code:
eth_xfi_en = true
There is a chance this option is not even specified in your config yet. If so add it to the [HCA] section.

The second option controls wether SR-IOV is enabled. Unless you know you need SR-IOV I'd suggest disabling this feature, since some mainboards do not support SR-IOV and have trouble with the resulting giant PCI BAR sizes. Set
Code:
sriov_en = false
or add it to the [HCA] section in case that config option is not present yet.

Now save the config.

5. Build firmware image
The mft-oem package installed earlier contains a tool called mlxburn. Use the firmware package you extracted and the modified config to build an Ethernet firmware image:
Bash:
[user@host ~]$ mlxburn -fw fw-ConnectX3-rel.mlx -conf modified_eth_config.ini-wrimage modified_eth_firmware.bin
modified_eth_firmware.bin will now contain your firmware image

6. Verify firmware image
To make sure your firmware image is actually bootable you should quickly veriy it with flint:
Bash:
[user@host ~]$ flint -i modified_eth_firmware.bin verify

Flashing
At this point you should have a firmware image for your card. If you have not started mst yet, do it now:
Bash:
[user@host ~]$ sudo mst start
Now you can flash the firmware image to your card:
Bash:
[user@host ~]$ sudo flint -i <fimware file> -d /dev/mst/mt4099_pciconf0 -allow_psid_change burn
Replace <fimware file> with your firmware filename.

If flashing fails you might have to add -nofs to the flash command (depends on currently running and to be flashed firmware image):
Bash:
[user@host ~]$ sudo flint -i <fimware file> -d /dev/mst/mt4099_pciconf0 -nofs -allow_psid_change burn
Now reboot your system. It should come with the network adapter detected.


Troubleshooting
Card seems bricked
Whenver you can not see the card in lspci anymore you can force the card into a bootloader mode that will allow you to flash a new firmware file as described in Flashing. To do so you have to find the recovery jumper on your card. On some HPE cards that jumper is labeled FLASH PRSNT. Remove the card from your system, short the two pins of that jumper, reinstall the card and boot the system with the two pins shorted. Then you will be able to flash a new firmware image. Note that the jumper might not be fitted, meanin you will need to find something conductive to jam into the throughholes. The following photo shows the jumper on a single-port SFP+ FlexibleLOM NIC.

FlexibleLOM_bootrom_jumper_small.jpg

Network interfaces are not showing up
When you do not see any network interfaces but can still see the card in lspci this could be due to one of three reasons:
  1. Card is not configured correctly
  2. PCI BAR is too large
  3. Card has Infiniband firmware flashed
1. Card is not configured correctly
If your card is not showing up it is more than likely just not configured correctly. Some cards are Mellanox VPI cards. That means they can be configured to support different protocols on the fly without reflashing the firmware. To configure your card for Ethernet do the following:
Start mst:
Bash:
[user@host ~]$ sudo mst start
Configure the cards ports for Ethernet:
Bash:
[user@host ~]$ sudo mlxconfig -d /dev/mst/mt4099_pciconf0 s LINK_TYPE_P1=2
If your card has two ports configure the second port for Ethernet, too:
Bash:
[user@host ~]$ sudo mlxconfig -d /dev/mst/mt4099_pciconf0 s LINK_TYPE_P2=2
Now reboot and the system should come up with the expected Ethernet interfaces.

2. PCI BAR is too large
Depending on your host system the PCI BAR required for the card might be too large to be supported. This is an issue in particular when SR-IOV is still enabled. Have a look through dmesg to identify such issues. If you see any PCI BAR allocation failures or the mlx4 driver complains about missing UAR you are most definetly having a PCI BAR size issue. If you do not want to disable SR-IOV search your BIOS/UEFI setup for an option to either configure PCI(e) BAR size or something like "above 4G IO memory". If that does not help reflash your card with a firmware that has SR-IOV disabled.

3. Card has Infiniband firmware flashed
If none of the above helps your card might still simply have the Infiniband firmware flashed. Just carefully redo the configuration and flashing procedure.



I Hope you found this writeup useful or at least somewhat interesting

Tobias
 
Last edited:

klui

Active Member
Feb 3, 2019
295
124
43
How difficult is it to securely mount the LOMs on the extender to the chassis?
 

tsys

New Member
Oct 5, 2020
4
23
3
@klui Depends on how securely you want it to be mounted. I've designed the adapter in a way that brings the top hole in the FlexibleLOM bracket very close to the PCIe bracket retention screw. Thus it should be pretty simple to just use a small metal angle piece or similar.
I'll likely get some custom slot covers laser-cut and bent by a friend for my own use. Maybe he can share the design files or something.
 

tsys

New Member
Oct 5, 2020
4
23
3
@omavel If you are based in Germany I might be able to send you one or two adapters, but apart from that no. To make that worthwhile I would have to charge like $20 per adapter since it is a pretty niche product. I might consider doing it when I can get the PCIe slot cover made in decent amounts for a reasonable price.
 

psc

Member
Jun 30, 2019
34
6
8
@omavel If you are based in Germany I might be able to send you one or two adapters, but apart from that no. To make that worthwhile I would have to charge like $20 per adapter since it is a pretty niche product. I might consider doing it when I can get the PCIe slot cover made in decent amounts for a reasonable price.
I've just picked up a 764285-B21 to try this with. I'm going to be wanting an adapter, but I'll probably go the 3d printer route to generate a physical support; I'll work that out when the module arrives. Are you interested in popping one of a PCB in the post, or is anyone else interested in getting a few made up? Otherwise I'll end up with a few space PCBs.

Edit: I'm in the UK
 
Last edited:

psc

Member
Jun 30, 2019
34
6
8
Just to confirm, I've put in an order for PCBs (with JLC) and Amphenol G630H9812248EU. Once I've got them and soldered up what I need I'll be happy to sell spares on the basis they'll be supplied as parts for you to solder and I'm not responsible if it blows up your machine, burns down your house or takes your family hostage. Might be a few weeks until everything's here.
 

mlc130104

New Member
Feb 26, 2017
6
0
1
26
Grabbed 3 561FLR-T on chinese second hand market for $10 each (I'm HK-based)
Cant wait to try them!
 

psc

Member
Jun 30, 2019
34
6
8
Managed to get at least one adapter soldered up OK, and have a Connect-X 3 Pro showing up now. I think it's in IB-mode, but hopefully that's easy to fix.

The boards were delivered without a chamfer on the PCIe connector, sadly, so I need to progress my complaint with JLC. I really wish I'd got my reflow toaster oven sorted, too; it's a faff by hand with the huge-tipped iron I've got lying around! I need to see how many more write-offs I have to get a 2nd good adapter to know what I've got spare; I'll post back here once I have a tally.

I might also see if it's possible to 3d-print a solder mask, and subsequently a bracket to hold card+adapter to the chassis opening. Hopefully I can extract the mask and board outline from the gerbers.
 

mlc130104

New Member
Feb 26, 2017
6
0
1
26
Managed to get at least one adapter soldered up OK, and have a Connect-X 3 Pro showing up now. I think it's in IB-mode, but hopefully that's easy to fix.

The boards were delivered without a chamfer on the PCIe connector, sadly, so I need to progress my complaint with JLC. I really wish I'd got my reflow toaster oven sorted, too; it's a faff by hand with the huge-tipped iron I've got lying around! I need to see how many more write-offs I have to get a 2nd good adapter to know what I've got spare; I'll post back here once I have a tally.

I might also see if it's possible to 3d-print a solder mask, and subsequently a bracket to hold card+adapter to the chassis opening. Hopefully I can extract the mask and board outline from the gerbers.

May this repository shed some lights on your path in search of a bracket :D

I just received my PCBs from JLC. I waited for 2 weeks as the first batch of 10 were all failed boards according to their CS. They sent me 11 pieces eventually. I also grabbed 682150-001 single port cards from eBay and I finally fired them up tonight. Now I have to find the mystery OEM FW packages to upgrade them...
 

ftl_c_ian

New Member
Nov 3, 2020
5
1
3
I got my batch of adapters and edge connectors in and none are chamfered, but it's not a big deal at all. I haven't been having much luck getting cards to work yet though. I've got 4 530FLR cards (647581-B21) I've tried 3 of them so far with 2 different adapters. Have yet to see anything in lspci, a few times the machine boots up very slowly like its mad about something. Got a few too many variables I figure, I've got some 560 FLR (665243-B21) boards coming in that I'll test today.

Speaking of which how is everyone soldering or testing these things? I've tried drag soldering, smothered in flux then cleaned off with acetone, last few cards I did with solder paste and hot air, still ended up with a decent number of bridges though. Have yet to figure out an easy and good way to solder these. The two I have that seem ok took quite a bit of time. For testing I've just checked that the power and ground pins aren't shorted and spot checking some of the other pins, it's hard enough to spot bridges, going with less solder and you might end up with cold joints.
I don't have the worlds best eye sight so I've been using my phone camera to inspect when I start getting cross eyed. Maybe I just need some manner of microscope type setup, anyone recommend a decent budget setup?
 

omavel

New Member
Oct 6, 2020
2
1
3
ftl_c_ian
I just get two of 682148-b21 from postoffice and going to purchase some lom-pci bridges. So I'm interesting in your issue too.
Could you tell me, where did you order these bridges?
 

ftl_c_ian

New Member
Nov 3, 2020
5
1
3
ftl_c_ian
I just get two of 682148-b21 from postoffice and going to purchase some lom-pci bridges. So I'm interesting in your issue too.
Could you tell me, where did you order these bridges?
I had them made at jlcpcb following the instructions on the github linked in OPs post. Not having any luck with these cards yet, but still a ton of variables. Have one of the 682625-001 mellanox cards coming tho, hoping for more luck with that.
 
  • Like
Reactions: omavel

mlc130104

New Member
Feb 26, 2017
6
0
1
26
I got my batch of adapters and edge connectors in and none are chamfered, but it's not a big deal at all. I haven't been having much luck getting cards to work yet though. I've got 4 530FLR cards (647581-B21) I've tried 3 of them so far with 2 different adapters. Have yet to see anything in lspci, a few times the machine boots up very slowly like its mad about something. Got a few too many variables I figure, I've got some 560 FLR (665243-B21) boards coming in that I'll test today.

Speaking of which how is everyone soldering or testing these things? I've tried drag soldering, smothered in flux then cleaned off with acetone, last few cards I did with solder paste and hot air, still ended up with a decent number of bridges though. Have yet to figure out an easy and good way to solder these. The two I have that seem ok took quite a bit of time. For testing I've just checked that the power and ground pins aren't shorted and spot checking some of the other pins, it's hard enough to spot bridges, going with less solder and you might end up with cold joints.
I don't have the worlds best eye sight so I've been using my phone camera to inspect when I start getting cross eyed. Maybe I just need some manner of microscope type setup, anyone recommend a decent budget setup?
I happened to have some soldering paste (technically Sn63/Pb36.5/Ag0.5) around me, and I dragged the paste on the pcb with a soldering iron. In my experience, the paste will nicely form on the pad. Just make sure that won't get too thick will be fine. You will then need to insert the socket into the board, solder a few pins on the left on one side, then some pins at the same position on the opposite side. Then some pins back another side, and so forth. Not soldering all pins on a single side first prevents the socket get overheated and thus have some sort of inclination/banking. I dont have a oven but only with an iron to solder 10 boards and I think first 3 board look less decent haha.

(sorry english is not my first language :p )


IMG_755324E2BE60-1.jpeg
 

ftl_c_ian

New Member
Nov 3, 2020
5
1
3
I happened to have some soldering paste (technically Sn63/Pb36.5/Ag0.5) around me, and I dragged the paste on the pcb with a soldering iron. In my experience, the paste will nicely form on the pad. Just make sure that won't get too thick will be fine. You will then need to insert the socket into the board, solder a few pins on the left on one side, then some pins at the same position on the opposite side. Then some pins back another side, and so forth. Not soldering all pins on a single side first prevents the socket get overheated and thus have some sort of inclination/banking. I dont have a oven but only with an iron to solder 10 boards and I think first 3 board look less decent haha.

(sorry english is not my first language :p )


View attachment 16310
Nice, those look a lot better than mine so far, I'll give that a go :)
 

ftl_c_ian

New Member
Nov 3, 2020
5
1
3
I had them made at jlcpcb following the instructions on the github linked in OPs post. Not having any luck with these cards yet, but still a ton of variables. Have one of the 682625-001 mellanox cards coming tho, hoping for more luck with that.
That did the trick, the 682625-001 from ebay just worked right out of the box... Testing with the other cards, still haven't got a 530FLR (qlogic chip) to work, but I've got two 560FLR cards that work in ubuntu 20 and win 10 w/ the 3v3, Vdet and PRST jumpers soldered, reports as intel 10gb ethernet adapters. Not sure what to try with the 530 cards, maybe a different combination of jumpers, kinda sucks since I've got 4 of those... Oh well, definite progress and I've got 3 working cards which is already as many as I need at any rate.

Edit: got another 560FLR Intel card working in my main desktop box, only jumpered the PRST one for A17 I believe and it's cooking right along, don't need 3v3 or Vdet. The mellanox single port card didn't wanna work in win 10, guess maybe it needs drivers?
 
Last edited:

ftl_c_ian

New Member
Nov 3, 2020
5
1
3

May this repository shed some lights on your path in search of a bracket :D

I just received my PCBs from JLC. I waited for 2 weeks as the first batch of 10 were all failed boards according to their CS. They sent me 11 pieces eventually. I also grabbed 682150-001 single port cards from eBay and I finally fired them up tonight. Now I have to find the mystery OEM FW packages to upgrade them...
Printed a couple of those out and the bumps are just like 3 or 4 mm from being in the right spot, those adapters from the chinese forum are bit taller it seems
 

hmartin

Active Member
Sep 20, 2017
250
176
43
34
Has anyone tested the 533FLR-T or 561FLR-T yet?

HPE quickspecs for the 533FLR-T state 9W maximum power consumption, while the 561FLR-T is missing power consumption in the quickspecs.

Both cards are available for approximately the same price ($27-$29 USD from eBay sellers in China with free shipping). Given that both the 57810S (533FLR-T) and X540-T2 (561FLR-T) have mainline kernel support, my interest is which one has the lower power consumption.

Intel's own documentation states that the X540-T2 should have a maximum power consumption of 17.4W:
FireShot Capture 196 -  - www.intel.com.png

Between these two sources, it sounds like the 533FLR-T has lower power consumption.