Will SFP FTTH transceiver work in SFP+/SFP28 NIC?

[homeserver78]

I'm loosely thinking about upgrading my home network. Current situation:

• Cat5e cables in the walls.
• 100/100 Mbit/s internet via fiber: SFP transciever marked SKYLANE SBU35010GE00000 in "broadband switch" Inteno XG6846 (ISP infrastructure by Zitius, for the Swedes out there). ISP supports up to 1000/1000 Mbit/s.
• 1 GbE RJ45 everywhere else, except my workstation has a 2.5 Gbit/s NIC.
• WiFi 5/ac via TP-Link EAP225v1 powered using PoE. This thing works flawlessly (but no FW updates since 2018!).
• OPNsense firewall (gets WAN IP via DHCP).

All this currently works quite well, I'm not overly annoyed by transfer times even of video files to/from my NAS. The only real limitation that I experience is that my switch is dumb and I'd like to be able to use VLAN segmenting. And hey, upgrading is fun.


Loose upgrade plan:

• Get a 2.5 (or 5) Gbit/s NIC for my NAS.
• Get a L2-manageable switch with, say, 2x SFP+ and 8x RJ45 2.5 Gbit/s ports with PoE. TP-Link SG2210XMP-M2 looks ideal but it's way too expensive at the moment. I'm thinking <200 EUR is reasonable. (I'm not buying anything mains connected from AliExpress or Amazon.) So I guess I'm in for some waiting.
• Upgrade the firewall HW to something that can take a dual SFP+/SFP28 card (and get such a card).
• Hook my ISP's SFP transciever into the firewall directly, and hook the other firewall SFP+/SFP28 port to the switch.

If this works it would let me get rid of the ISP "broadband switch" and play with SFP a bit, it would give plenty of bandwidth for OPNsense to handle routing between VLANs, and would give me 2.5 Gbit/s between workstation and NAS.

Is it possible to say anything about how likely it is that the ISP's transciever will work in something else than the provided switch (and SFP+ or SFP28 at that)? Or is it simply "try it and see"?

Any thoughts about my upgrade plan? Am I missing something obvious/doing anything stupid?

 

[nexox]

I don't think I would bother to mess with the ISP equipment for 100M or 1G, seems not worth saving a couple watts and the space a little switch takes up.

I would also not look too hard at a PoE switch for a single PoE device, just get an injector and then you have significantly more options for a switch, like that new 2.5G model Mikrotik makes.

I like the ConnectX 4 Lx for 10/25G NICs these days, but I wouldn't be surprised if the price in Europe isn't as appealing as the US, if your fileserver is near enough to the switch to run a cable then I would skip the 2.5G NIC there and go straight to 10G, it's often cheaper and the hardware/drivers tend to be more reliable. On the other hand, for a maximum of 1G WAN I think I would stick with 1G NICs on the firewall, since you're not going to see any return on the increased price and energy consumption of a 10G NIC and router that can run it.

 

[homeserver78]

Thanks, lots of interesting angles there! I especially like the PoE injector idea. (I guess I could use my existing dumb 1 GbE PoE switch as an injector!)

Unfortunately the apartment patch panel (and the fiber entry point) is in a really bad spot: in the hallway, in a rather small box close to the ceiling. The box can hold a couple of small switches, and I think I can fit in a small form factor firewall on a shelf nearby, but that's about it. Everything else has to go elsewhere.

Hmm, maybe I should dig around a bit and see if it's possible to add a fiber cable among the existing cables to my NAS. But I doubt it. They have a rather limited bend radius, don't they?

The ConnectX-4 Lx goes for about 95 EUR incl shipping on Ebay it seems. Certainly not peanuts, but reasonable compared to a 1x 2.5 Gbit/s NIC for like 60 EUR (incl shipping) perhaps? ¯\_(ツ)_/¯

There's also the "fun" factor and the opportunity to get some experience with the SFP "ecosystem". I don't particularly like the higher power draw of the 10+G NICs, but I'm thinking that if I could get rid of the ISP switch then maybe that'll offset it, more or less.

Ah, I don't know. Maybe I'll try it for the experience and then go back to the 1G equipment.

 

[nexox]

Fiber is actually quite flexible, but it requires expensive equipment to terminate so you mostly have to pull it with the connectors on, which makes it tricky (unless you hire a pro, no idea what that costs.) It's probably very not supported by your ISP but if you could pull a single fiber (rather than the duplex pair used for typical LAN runs,) you could just use a passive fiber coupler in the box and move the ISP switch to somewhere more convenient.

If the ConnectX 4 Lx is that expensive then it doesn't make as much sense (I don't do a lot of international shipping but it seems like postage and tax on a $25 NIC from the US would still come out under 60 EUR,) but you can also look at the Intel X520, ConnectX 3, and various Solarflare NICs to see if they're less expensive.

The ISP switch specs suggest it probably draws 10W at the wall, likely less when its idle, and 1-2W of that will be for the fiber module which still draws the same power even if you move it.

In your situation I would be tempted to put just a single managed 2.5G switch in the fiber box, connect the ISP switch to the managed switch, and then use VLAN trunking to put the router wherever is convenient - I did similar for a long time because my cable terminated in a coat closet, but I kept my router in my server closet two switch hops away (eventually I got fiber and the installer was happy to make the much easier run directly to the server closet, but my fiber ONT still connects to a switch, not my router.)

 

[homeserver78]

There are actually two Cat5e Cat6 cables to each room, so I have the ISP fiber-to-copper switch in the patch panel box and a connection from that to my "server closet" (computer corner in my case ) directly to the firewall/router. The LAN side then connects to a switch in the computer corner (to which also my NAS connects) and then back through the walls to the patch box, to another (PoE) switch that connects everything else.

I've read about (cheap) managed switches that forget their settings when they are few years old and lose power, so I'm a bit loathe to use VLAN trunking for the WAN side. Maybe I'm a bit on the paranoid side, lol.

---

The ISP fiber enters to a socket (SC I believe), then there is a short SC-to-LC patch cable to the ISP switch. I see similar cables are sold e.g. by FS for not a lot of money so it would certainly be possible to use a longer one. The linked cable is "bend tolerant" with a minimum bend radius of only 10 mm too! (I didn't know such fiber cables existed.)

... but I dug around a bit and there is no way I can get a pre-terminated cable through the existing holes. They are just the right size for the existing cables and no more. (And the walls are concrete.) So that's a bummer. On the positive side, the existing cables turn out to be Cat6 and not Cat5e as I had been told!

it seems like postage and tax on a $25 NIC from the US

You get the ConnectX-4 Lx for 25 USD? The cheapest I can find on Ebay is Dell CX4121C from China for 47 EUR incl shipping, ex VAT. So about 70 EUR incl VAT and custom fees, I guess. Or did you mean that the 2.5 Gbit/s cards are $25? (If so, similarly priced, used, no-name ones can be found here as well.)

Used Solarflare dual SFP28 NICs seem to go for 175-699(!) EUR.

 

[jehu85]

The ISP's SFP transceiver might work in other switches, but it can be hit or miss, so you might just have to try it out. Just ensure the new switch supports the same protocols. Overall, you're on the right track

 

[nexox]

I've read about (cheap) managed switches that forget their settings when they are few years old and lose power, so I'm a bit loathe to use VLAN trunking for the WAN side.

There are different levels of cheap, but I've run years and years on quite affordable Netgear managed switches before switching to Mikrotik when it was time to go to 10G, never had any of them lose configuration settings, but if you have two wires then no need for VLANs.

On the positive side, the existing cables turn out to be Cat6 and not Cat5e as I had been told!

If it has "550MHz" printed on then it should run 10G, that opens up some fun. Not efficient or low-cost, but it works - due to the way my place was wired I have two runs of CAT6 and a switch between my server closet and workstation, and I have had no problems with reliability or speed.

You get the ConnectX-4 Lx for 25 USD?

Mellanox CX4121A Dual-Port 25GB SFP28 PCI-E Ethernet Card [Low Profile] | eBay only $24 today, $25 for an HP-branded card with a high profile bracket: HPE 840140-001 Ethernet 10/25Gb 2-Port 640SFP28 Network Adapter High Profile | eBay.

Used Solarflare dual SFP28 NICs seem to go for 175-699(!) EUR.

Oh yeah I meant if you couldn't get the ConnectX-4 Lx then you should probably just look at sfp+ NICs, since 25G switches are so expensive there's not a lot to be gained with sfp28.

 

[homeserver78]

Mellanox CX4121A Dual-Port 25GB SFP28 PCI-E Ethernet Card [Low Profile] | eBay only $24 today, $25 for an HP-branded card with a high profile bracket: HPE 840140-001 Ethernet 10/25Gb 2-Port 640SFP28 Network Adapter High Profile | eBay.

Ha, that's amazing! I'm envious!

 

[blunden]

Yeah, the fact that the 25% VAT is calculated on the price including shipping makes it quite a bit more expensive when buying from outside the EU. It can still sometimes be worth it though since I've found that prices on the local market for used enterprise equipment can be quite a bit higher. For instance, when I bought my used Intel X710-DA2 NIC from the US I ended up paying almost twice as much as the list price on eBay but my total cost was still roughly half of what I could find them for locally.

There's really nothing wrong with those Inteno switches in terms of reliability. Still, it's likely that the SFP transceiver you have will work in other devices too as long as the NIC in your firewall doesn't use vendor locking and assuming the transceiver's EEPROM doesn't contain something the NIC doesn't like.

The few fiber transceivers I've used have mostly just worked, but the FlexOptix 10GBASE-T copper transceiver I need to use for one run only worked with 2 out of the 5 or 6 EEPROM configurations it could be programmed with at the time. One of the working ones was the factory configuration that wasn't available to flash it with if you ever decided to overwrite it, and the other working one was only added after I asked their support to do so. Only one of them has the vendor OUI of Intel so it's not a vendor lock, it's something else.

 

[homeserver78]

Mellanox CX4121A Dual-Port 25GB SFP28 PCI-E Ethernet Card [Low Profile] | eBay

I ordered two of those. Now to wait up to a month for delivery... Turns out that store uses Ebay's International Shipping so VAT was charged by Ebay. Hopefully that means no further custom fees and such - it's the first time I use the European IOSS system with an IM number, or whatever it's called, so that'll be interesting. The total for two cards, international shipping, and VAT: 88.06 USD. So less than half the price of the cheapest card on Ebay in EU. (Well, if you buy two.) Let's hope they work!

I've also ordered (and received) a SFF computer: a "Fujitsu Esprimo D738/E94+" (157.89 EUR incl shipping). i5-8400, 16 GiB DDR4, and a 256 GiB SSD. (Listed as M.2 but they shipped a 2.5" Samsung PM871b SATA SSD - some OEM drive which supports OPAL encryption, it seems. ¯\_(ッ)_/¯) I selected this SFF in particular since it's got both a x16 slot and an x4 x2 slot (open x8 physical), and it's supposed to be very efficient at idle. And sure enough, when booted to a debian live distro, with an additional bcm5720 2x GbE NIC installed and with some powertop tweaking, it idles at 5-6W!

It passed memtest86 and there's even an UEFI update from Fujitsu dated October 2024! That applied without issues. It's not silent though, far from it: it's got three(!) fans and I think at least the CPU fan is on its last legs. Idles like a diesel engine. Year of manufacture seems to be 2019 so it's probably spun for a while...

The plan now, once the ConnectX-4 Lx cards arrive, is to flash the latest firmware, and then check what happens to the power consumption. The card itself should draw about 6 W (with no transceivers)? The question is what happens to the C-states and the power draw of the rest of the system. We'll see.

 

[homeserver78]

What's a reasonable network cable to use for testing the cards when they arrive? I'm thinking ~1 m long "loopback" cable between the two ports on a single card.

fs.com has 1 m SFP28 DAC for 32.50 EUR (and I probably need to place an order there anyway for an SC-to-LC patch cable if I want to try the ISP SFP transceiver in my new, used, NICs). The pickings on ebay.de are slim for SFP28 DACs. If I get the FS one, what vendor compatibility do I choose? (IIUC the Mellanox cards are not vendor locked, so going with Mellanox branding is kinda wasted?) Is "generic" branding the most widely compatible one?

What about SFP+ DACs - can a short 10 Gbit/s DAC work at 25 Gbit/s?

Is there a good source for used optical transceivers that I can use with fibre cables instead?

So many questions.

I'm in the EU.

 

[homeserver78]

I ordered two of those.

I haven't really had the time/energy to start messing with these until now. First findings:

I got one Rev AC card and one Rev AF. Both could be updated to the latest FW without issues. Tests below done with latest FW and no transceivers.

• The Fujitsu's second (chipset) PCIe slot is x2, not x4 as is printed on the PCB. The PCH root complex also does not support ASPM.
• The computer idles at 5 W without anything connected apart from the Live boot USB stick.
• With a ConnectX-4 Lx in the CPU slot idle power is 15 W.
• With a ConnectX-4 Lx in the chipset slot idle power is 13 W.
• pcie_aspm=force, powertop --autotune, 'echo powersupersave > /sys/modules/pcie_aspm/parameters/policy' all makes absolutely no difference.
• Regardless of where I put the NIC and other settings, powertop reports Pkg power ~9 % in C2 and ~90 % in C3. Without the NIC Pkg power is mostly in a much lower state (C7 if I remember correctly).
• No difference in idle power consumption between AC and AF HW revisions.

I haven't had the time to actually try out if my FTTH transceiver works with the NICs yet. ¯\_(ッ)_/¯

 

[homeserver78]

I can finally answer the question in the thread title: Yes!

The ISP's fiber transceiver works with my ConnectX-4 Lx card @ 1 Gbit/s link speed. Auto-negotiation fails though, I had to set the speed manually.

With the NIC in the chipset slot in the Fujitsu and the link up idle power draw is around 19 W. This is under OPNsense so probably not entirely comparable to the Linux power figures above. (And I have no idea how to tweak OPNsense power draw...) This is still a little bit lower than having the NIC sitting idle (no transceivers plugged in) plus running the ISP's switch. And it does give me that ability to use a 10 Gbit/s link to a switch later. And it's fun to have a little bit of fiber network.

 

[WhiteNoise]

I am surprised that it worked. I thought you couldn't just plug the ISP SFP transceiver into a network card, you also need an ONT.

There are some special "SFP-shaped" cards that do both functions (they have an internal cpu and run a small OS), like this

https://www.fs.com/products/133619.html

A Brief Introduction to the GPON ONU Stick Module

The GPON ONU Stick module seamlessly integrates into the GPON network, serving as a compact and versatile component that enhances communication capabilities within the network architecture.

www.fs.com

 

[blunden]

I am surprised that it worked. I thought you couldn't just plug the ISP SFP transceiver into a network card, you also need an ONT.

There is nothing surprising about it. So far, Swedish ISPs mostly use AON, not PON. I think that's partially a result of having offered fiber internet service to consumers for 20+ years already. I seem to recall 1 Gbps was available to me back in at least 2006 for a reasonable price.

Sadly, I've heard that at least two ISPs are switching to XGS-PON for some new deployments.

It's only with PON that you need those special ONTs. With AON, the customer equipment the ISP provides is usually just a media converter to convert from fiber to RJ45. There is usually nothing stopping you from taking the SFP (or SFP+) transceiver that is often found plugged into the media converter and plugging it into your own SFP/SFP+ gear.

 

[WhiteNoise]

Thanks for the explanation!

 

[homeserver78]

With the NIC in the chipset slot in the Fujitsu and the link up idle power draw is around 19 W.

I've managed to get this down to around 13 watts now. The biggest change was from simply placing the computer on its face so that convection can do its thing! This reduced temperatures by some 4-5 degrees which made the fans slow down which reduced their power draw.

The other thing was enabling Cstates in OPNsense. This had a very marginal direct effect (maybe 0.5 W) but it lowered CPU temperature by another couple of degrees which in turn made the fans spin even slower which cut another couple of watts from the total power consumption.

This means that this system, with its 6-core CPU, 32 GiB of DDR4, and dual 25 Gbit/s NIC actually draws a tiny bit less power than my old fanless OPNsense router based on a Via C7 with 1 GiB of RAM and an add-on 100 Mbit/s 3Com PCI NIC for WAN interface. (That system has been incredibly stable over the years though -- after I replaced the 12 V brick PSU with a MeanWell HRP-100-12 mounted in an aluminium box, that is. Bought in 2007 it has certainly served me well. Hat off!)

---

Repeating some of the info about Cstates here:

To check current settings and stats:

root@OPNsense:~ # sysctl dev.cpu | grep cx
dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.3.cx_usage_counters: 304429829 0 0
dev.cpu.3.cx_usage: 100.00% 0.00% 0.00% last 312us
dev.cpu.3.cx_lowest: C1
dev.cpu.3.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.2.cx_usage_counters: 352002831 0 0
dev.cpu.2.cx_usage: 100.00% 0.00% 0.00% last 47us
dev.cpu.2.cx_lowest: C1
dev.cpu.2.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.1.cx_usage_counters: 288856368 0 0
dev.cpu.1.cx_usage: 100.00% 0.00% 0.00% last 305us
dev.cpu.1.cx_lowest: C1
dev.cpu.1.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 268697840 0 0
dev.cpu.0.cx_usage: 100.00% 0.00% 0.00% last 529us
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1000

To modify temporarily (until next boot):

# sysctl dev.cpu.0.cx_lowest=C3
# sysctl dev.cpu.1.cx_lowest=C3
# sysctl dev.cpu.2.cx_lowest=C3
# sysctl dev.cpu.3.cx_lowest=C3

(Repeat for each CPU.)

There is a Tunables page in OPNsense GUI where the settings can be added permanently.