10 and 40 Gb networking on the cheap ($5 - $10)

[psc]

why not get Mellanox CX3+ and a DAC? Cheaper and faster.

I'll second this - especially now the HP prices have gone up a bit. I've tried and failed to get the 40Gb cards working with an early revision, and ended up getting a 40Gb CX3 card instead (well, 2 of them - one Fujitsu branded). I'm not sure it's cheaper, but the difference isn't that big vs card+PCB+connector if you can bide your time, and it's a lot less risk and faff vs channel bonding 2 10Gb ports on a DIY adapter and mounting plate. If I'd got a toaster reflow oven to hand, and I had more enthusiasm for working out kinks than money to avoid them, then it might be different, but...

If anyone wants a couple of 40Gb HP cards drop me a message!

 

[RageBone]

actually, i would go for flexlom CX3+
554+flr is their hp name.
they were about 7$ a few month ago.
+ 10$ pcb and 25$ DAC

 

[Peter Blanchard]

why not get Mellanox CX3+ and a DAC? Cheaper and faster.

I'm in the UK and pricing is different from the US. Almost always more expensive but very variable. ConnectX-3 about £80 if you can find one whereas 561FLR-T, I've seen for as little at £12. Shipping costs from the US are prohibitive.

 

[RageBone]

I'm in the UK and pricing is different from the US. Almost always more expensive but very variable. ConnectX-3 about £80 if you can find one whereas 561FLR-T, I've seen for as little at £12. Shipping costs from the US are prohibitive.

I still have some from the 7$ era imported from the US to germany.
could spare two.
Or send you my old plain CX3 non +

 

[sdiaconu]

Hello,
Small question. I have a 561FLR-T rev. 0C. Can this card work in a PC without adapter ? I have a TrueNas that doesn't find the card ...
If no where can I buy one ?

Thanks in advance.

 

[hmartin]

Can this card work in a PC without adapter ?

All FlexLOM cards require an adapter to be used in a PC. Yes, it may look like it fits, but the pin layout is different between a FlexLOM slot and a PCIe slot. Do not attempt to plug the card directly, you will damage your motherboard.

There are a number of eBay sellers who carry the adapter:

• PCIe X8 Riser card for FlexibleLOM GbE 331FLR 366FLR 544FLR 561FLR US seller | eBay
• PCIe X8 Riser card for HP FlexibleLOM 2Port GbE 331FLR 366FLR 544FLR 561FLR | eBay
• PCIE X1 X4 to HP FlexibleLOM Quad GbE Ethernet 331FLR 366 530 544 561 FLR CX3 | eBay
• LOT of 5/10 PCIE X8 to HP FlexibleLOM Quad GbE Ethernet 331 366 530 544 561 FLR | eBay

I have no experience with these vendors or adapter designs, YMMV.

 

[tathers]

Printed a couple of those out and the bumps are just like 3 or 4 mm from being in the right spot, those adapters from the chinese forum are bit taller it seems

Didn't see anyone in the thread make one for the other pcie bracket, so I hastily made one in f360 based on the one in the github. Anyone got recommendations as to where I should upload the stl?

EDIT:
Lost the source file but here's the thingiverse link: PCIE Bracket for Tobias Schramm's FlexLOM adapter card by Tathers

 

[psc]

Didn't see anyone in the thread make one for the other pcie bracket, so I hastily made one in f360 based on the one in the github. Anyone got recommendations as to where I should upload the stl?View attachment 30700

I tend to look on Thingiverse for stuff like that. Ideally, upload the source file as well as the exported STL.

 

[Vit K]

Don't want to hijack the thread, but they selling this simple adapter on aliexpress now along with card for 30-40 usd.

 

[Paladin]

Hi! First of all thanks for the wonderfull work. I have around 80 544flr and have found this topic. can I buy the adapter somewhere in the eu? If not, where do you recommend me to buy the parts? Thanks!

 

[bogardon]

I am very late to this, but I bought one of those PCIE X8 to FlexibleLOM on eBay to pair with my 562FLR-T. Everything works...except Wake-on-LAN does not. I wonder if it is because the one I got is missing the 3v3 aux mentioned in the github readme?

I tried buying one from sysmocom but it's like $40 to ship to California! Anyone know where else I can get one that supports WoL? Thanks.

 

[hmartin]

I am very late to this, but I bought one of those PCIE X8 to FlexibleLOM on eBay to pair with my 562FLR-T. Everything works...except Wake-on-LAN does not. I wonder if it is because the one I got is missing the 3v3 aux mentioned in the github readme?

I tried buying one from sysmocom but it's like $40 to ship to California! Anyone know where else I can get one that supports WoL? Thanks.

I am not sure WoL works for FlexLOM cards? 3.3V aux is already connected in Tobleminer's adapter (which AFAIK is what sysmocom is selling):

Modification required for Broadcom based FlexLOM cards (530/533FLR) · Issue #4 · TobleMiner/HPE-FlexibleLOM-adapter

The HPE 530FLR-SFP+ and 533FLR-T cards are based on the Broadcom BCM57810. These cards do not work with the adapter without modification. The KCORES FlexibleLOM Adapter appears to have a more accur...

github.com

You can clearly see the 3.3V aux trace to B10 in my photo.

And WoL doesn't work. That might have something to do with #WAKE not being routes, but as you can read in the above comment, I've tried soldering that and I don't recall WoL working.

 

[bogardon]

The 562FLR-T is supposed to have WoL according to HPE's own specifications. Here are some pictures of what I have:





I don't know much about PCI-E or hardware for that matter, but it I did see etching around the B10 pin (3.3v aux), which AFAIU is used for standby power for WoL. Seems no etching around B11 (WAKE#) though...is it easy to add this myself?

I tried this combination in two PCs and double checked bios/windows settings, but still can't get WoL working :/ Shame, becuase this 562FLR-T was really cheap for a X550-T2 based card.

 

[hmartin]

I can see that HP claims WoL is supported, though only for the 562FLR-T (FlexLOM) and not the normal 562T PCIe card:

https://support.hpe.com/hpesc/public/docDisplay?docId=a00027151en_us&page=GUID-4C9DCC71-88C2-45F3-8489-4310637B9DD6.html&docLocale=en_US

You can try adding the #WAKE connection yourself. I would mask all but the very top of A9 with Kapton tape to prevent solder from flowing over the entire pad, and then solder the thinnest wire you can find from the PCIe side to the FlexLOM side.

Unfortunately the 562FLR-T is a bit too expensive for me to justify buying one to try this myself.

 

[bogardon]

I can see that HP claims WoL is supported, though only for the 562FLR-T (FlexLOM) and not the normal 562T PCIe card:

https://support.hpe.com/hpesc/public/docDisplay?docId=a00027151en_us&page=GUID-4C9DCC71-88C2-45F3-8489-4310637B9DD6.html&docLocale=en_US

You can try adding the #WAKE connection yourself. I would mask all but the very top of A9 with Kapton tape to prevent solder from flowing over the entire pad, and then solder the thinnest wire you can find from the PCIe side to the FlexLOM side.

Unfortunately the 562FLR-T is a bit too expensive for me to justify buying one to try this myself.

If you are US based I would just send you mine

 

[hmartin]

I do have a 533FLR-T, which claims to support WoL:

$ ethtool enp1s0f0 | grep Wake
    Supports Wake-on: g
    Wake-on: g
$ ethtool -s enp1s0f0 wol g

However, when I suspend the system, the link lights (1G link) go dark. I have no issues to WoL the system through the built-in NIC on the motherboard.

I checked with a multi-meter and the 3.3V standby rail is providing power to the card in S3. So, unless your 562FLR-T has the link lights on with a 1G link while the system is suspended, then I don't think WoL will work. Either the HPE datasheets are wrong, or more voodoo is required to make WoL work.

I just checked the KCORES adapter, and it has #WAKE connected already (PCIe B11 -> FlexLOM A9).

 

[bogardon]

My link lights on 562FLR-T are both off when in S3 sleep. Like you, onboard LAN WoL works fine.

Are you sure 533FLR-T supports WoL? I don't see it in the specs.

Would be surprising if HPE datasheets are wrong, as they specifically call out this functionality for 562FLR-T and not the regular 562T. In my BIOS I could actually enable/disable the WoL feature for the 562FLR-T ports, but on the 562T WoL was greyed out as N/A.

My bet is that FlexibleLOM's WoL related pins just aren't mapped right on the currently available adapters.

Btw, I bought my 562FLR-T on eBay for $22. Surely that isn't too expensive for you?

 

[hmartin]

Are you sure 533FLR-T supports WoL? I don't see it in the specs.

ethtool output above says it does.

My bet is that FlexibleLOM's WoL related pins just aren't mapped right on the currently available adapters.

The only relevant pins for WoL should be 3.3V AUX and #WAKE, which are both connected in the KCORES adapter.

My link lights on 562FLR-T are both off when in S3 sleep.

WoL can't work without the physical layer being up. Maybe someone with a FlexLOM installed in an actual HP server (which probably doesn't support S3) can come and tell us if WoL works or not.

Btw, I bought my 562FLR-T on eBay for $22. Surely that isn't too expensive for you?

$22 is not expensive. But it is too expensive when I factor in the shipping + VAT from US to EU. A $22 card becomes $40-50 easily and when I have no use for it later, because Mellanox adapters with a DAC consume much less power and pass bits just as fast, it's not worth it.