10 and 40 Gb networking on the cheap ($5 - $10)

Babbsdrebbler

New Member
Nov 17, 2020
7
0
1
Yes, with the adapter I got TrueNAS to recognize the 561FLR-T and collect an IP via DHCP.
Unfortunately I'm still in the process of buying a 10GbE switch and new motherboards with 2 8x PCIe Slots to setup networking.
Hence I could not test 10GB speeds and stability. Also the quality of the soldering on the adapter is not the greatest, I need to practice more...

The adapter popped up on a German shop which saved me some waiting time.
 
Last edited:

hmartin

Active Member
Sep 20, 2017
250
176
43
34
Yes, with the adapter I got TrueNAS to recognize the 561FLR-T and collect an IP via DHCP.
Unfortunately I'm still in the process of buying a 10GbE switch and new motherboards with 2 8x PCIe Slots to setup networking.
Hence I could not test 10GB speeds and stability. Also the quality of the soldering on the adapter is not the greatest, I need to practice more...

The adapter popped up on a German shop which saved me some waiting time.
Okay, good to hear. I think I will pick up some 533FLR-T then. Was the shop sysmocom? That's where I ordered my adapters from.
 

Babbsdrebbler

New Member
Nov 17, 2020
7
0
1
Yes, I also got them from there.

The problem with these adapters and cards is securing them in place and getting a good connection.
In one of my cases the rj45 sockets need to be jammed into the open slot to be externally accessible and that keeps the card from sagging down. In my other case I need to feed the cable to the inside of the case and I need something to support the card so it doesn’t fall down.

On the other hand getting a x540 for 40€ in Germany is quite rare, so I’m willing to put some time into it.
 

hmartin

Active Member
Sep 20, 2017
250
176
43
34
No luck getting the 533FLR-T to work. I've tried two different cards on different adapters, and with different jumper combinations (3V3, then 3V3+Vdet, then 3V3+VDET+PERST), no success yet.

The motherboard doesn't detect anything present in the slot. The card itself seems happy, link lights come up and it's consuming around 6W, just no device in lspci or BIOS.
 

Babbsdrebbler

New Member
Nov 17, 2020
7
0
1
Have you tested it in a proper server motherboard?
I’m asking because my 561 is only detected after a reboot on my x570 desktop.
The supermicro x10 server board doesn’t need that trick...
 

RageBone

Active Member
Jul 11, 2017
364
97
28
i have a few cards on the way from the US, adapters are nearly finished too, got the last few from sysmocom.
some 544+ and a 561, will report once i have everything here and running.

@hmartin did you solder on those pcie connectors?


Another thing bothering me is that i have a report of one of the KCores risers killing an SM H11SSL.
Shot BMC in that case apparently, me and him have no idea what went wrong, which is worrying.
 
Last edited:

psc

Member
Jun 30, 2019
34
6
8
No luck getting the 533FLR-T to work. I've tried two different cards on different adapters, and with different jumper combinations (3V3, then 3V3+Vdet, then 3V3+VDET+PERST), no success yet.

The motherboard doesn't detect anything present in the slot. The card itself seems happy, link lights come up and it's consuming around 6W, just no device in lspci or BIOS.
I'm currently in the opposite situation; the cards I got (764285-B21, dual Connect-X 3 Pro) are picked up fine but refuse to connect to my switch with any of my cables; the driver reports either unsupported cable or unsupported module. I doubt that's a problem with the adapter, but when I have no other 40GbE cards that's hard to narrow down.

However, it sounds similar to my normal 10GbE card, which wasn't picked up by my EPYC system until I updated the firmware even though it was fine elsewhere. See if you can try it in a different mobo.
 

hmartin

Active Member
Sep 20, 2017
250
176
43
34
Have you tested it in a proper server motherboard?
I’m asking because my 561 is only detected after a reboot on my x570 desktop.
The supermicro x10 server board doesn’t need that trick...
I've tried the card in an eGPU adapter on a Dell Latitude and didn't have any success, so I thought perhaps it was a PCIe BAR issue. My second attempt has been with a Fujitsu TX140 S2, which has an Intel C224 chipset. BIOS and Linux both report the slots as empty, despite the cards being there, powered, and establishing a gigabit link to my switch.

@hmartin did you solder on those pcie connectors?
Yes, using a TS110 and the BC2 tip, some 1,0mm 60/38/2 solder and a lot of patience.
photo5075558442655131902.jpg

However, it sounds similar to my normal 10GbE card, which wasn't picked up by my EPYC system until I updated the firmware even though it was fine elsewhere. See if you can try it in a different mobo.
I'll try it in my Z420. Maybe the card has an HP motherboard whitelist ;)
 

Babbsdrebbler

New Member
Nov 17, 2020
7
0
1
I've tried the card in an eGPU adapter on a Dell Latitude and didn't have any success, so I thought perhaps it was a PCIe BAR issue. My second attempt has been with a Fujitsu TX140 S2, which has an Intel C224 chipset. BIOS and Linux both report the slots as empty, despite the cards being there, powered, and establishing a gigabit link to my switch.

I'll try it in my Z420. Maybe the card has an HP motherboard whitelist ;)
I think I found my problem and yours might be related since your cards seem to have power and establish a link:
The card needs some time to initialize before the BIOS can find it.

After setting the boot delay to 10 seconds the card is available even on a cold boot.
That would also explain why it worked once I rebooted or stayed in the BIOS for a while (followed by the obligatory reset).
My Supermicro board is slow enough, especially because the HBA needs ages to initialize, so I never ran into that problem on my server.
 

hmartin

Active Member
Sep 20, 2017
250
176
43
34
No difference for me between a cold boot and a warm reboot, they're not visible in lspci. I wonder if the adapter is simply not functional with the 533FLR...

Edit: I have also tried it in my OCP Windmill motherboard and none of the cards are detected from a cold boot or a reboot.
 
Last edited:

Babbsdrebbler

New Member
Nov 17, 2020
7
0
1
Great that it works for you, I’m still fighting my mainboard trying to get it working on first boot and after suspend. On which board are you using the card?

I‘ve now bridged the 3.3V aux connection but that didn‘t change anything.
Well sometimes you win and sometimes you loose.

As the other 10gbe cards are only between 18—30€ it might be worth testing them.
 

turbochris

New Member
Dec 20, 2020
1
0
1
I have some toblerminer adapters, i got a 560FLR working, anyone have any luck with the 530FLR? KKORES seems to have found the answer w their 1.1 board.
 

Babbsdrebbler

New Member
Nov 17, 2020
7
0
1
He most likely referred to this adapter which was already posted earlier in the thread:

As there's no gerber zip I have no idea how to order it from a manufacturer.
 

hmartin

Active Member
Sep 20, 2017
250
176
43
34
According to the KCORES GitHub, v1.0 is not compatible with the 530FLR-SFP+ but v1.1 is. The change between v1.0 and v1.1 is that A10 and A11 are tied to +3.3V in v1.1

I tried to modify the adapter from Tobias to have +3.3V onto A10/A11 (also A4/A6/A7 and B9/B11 to GND) but no difference with my 533FLR-T. :(
 

RageBone

Active Member
Jul 11, 2017
364
97
28
so, my cards and adapters arrived, nearly finished soldering them all too, so far 3 pairs have been put together for testing and tinkering.
First specimen came with 2.42.5056 HP FW and had link-lights.
Went ahead and flashed a few different cx3 Pro FWs from mellanox because i wasn't really getting the card to work, partially due to stupidity.

I then stumbled upon the cx345A fcCt which appears to be the Pro version to the cx345a fcBt non pros i already have, which would also should be the same Mellanox version of the HP 544+ non flexlom card, so i went ahead and flashed that.

They appear in lspci, mstflint and config find it, i have two ethernet ports.
Everything so good with that so far, except that the ports stay down permanently.

went back and flashed the HP Flexlom FW version 2.35.5100 and the card appears in lspci, mstconfig is not able to read its config "driver might be down or busy" and no eth or ib devices, but a link light without activity.

Looking at /sys/bus/pci/devices/..../ shows no port mode options.


The release-notes of the newest HP 2.42.5700 FW say that having SRIOV enabled on the card and not enabled in the bios keeps the card from working. Maybe tahts the issue here?

EDIT:
My WS stops booting into my Manjaro after flashing the 2.42.5700.
gona try the 2_42_5056 for the hp 544+ non flr again to see if that does anything.

Final: Haven't found any other cards FW to be compatible, though you should test that yourself.
Technically the 544+FLR should work out of the box, i have settled on version 3.42.5700 which has that SRIOV boot failure bug, so right after flashing, you want to use mstconfig to disable sriov.
If the card fails too boot into the OS on your system, well, probably has that issue.
 
Last edited:

hmartin

Active Member
Sep 20, 2017
250
176
43
34
After looking at the KCORES GitHub repo (v1.1), I've successfully modified my TobleMiner adapters to work with the 533FLR-T.

Here are the modifications required:
  1. Cut the trace between Vdet (to make pin B1 N/C)
  2. Cut the trace connecting PERST to pin A1 (between the center/right solder pads)
  3. Bridge the PERST left/center solder pads so PERST is connected to A14
  4. Bridge 3.3V pad to pin A1
  5. Connect 3.3V pad to A10/A11
  6. Bridge GND between pins A4/A5/A6/A7/A8
In picture form (annotated, before modifying):


The modifications above bring the pinout of the FlexLOM to the KCORES v1.1 pinout (apart from nWAKE and B9/B11 to GND, which don't seem necessary):


Here is the lspci output of the 533FLR-T.
 

Babbsdrebbler

New Member
Nov 17, 2020
7
0
1
After looking at the KCORES GitHub repo (v1.1), I've successfully modified my TobleMiner adapters to work with the 533FLR-T.

Here are the modifications required:
Thanks for that detailed description of the needed changes! :)

@tsys Could you update your github project?

As for myself:
After forgetting to turn of the PC and therefore pulling the card while there was power I seem to have let the magic escape (although there was no smoke to be seen). So now I have two cards that are paperweights and a third one which happily works in my NAS.

I bought a used X550 for ~60 Euro and that works but it only wants to link at 5GbE whereas the X540 did 10GbE without a problem.
10GbE on copper seems to be haunted and I currently can't move to fiber to get around that.

I might give away my remaining 4 adapters at the Chaos Communication Congress this year.
I could buy the CX3 FLOM card for fun as they are ~5 Euro on ebay but I'd need a ~50 Euro transceiver which would make that unattractive again.