10Gb Ethernet Linux router thermals

[gpjt]

Hi all!

I'm upgrading my Internet connection from 1Gb to 10Gb, and I'm wondering about which machine to choose for a router, and how to manage the heat. My home is wired for Ethernet -- standard RJ45 sockets in most rooms to a patch panel at the front door -- and I've confirmed that the important links work at 10Gb using a point-to-point network between machines, so I guess it must be CAT-6a or short runs of CAT-5e in the walls.

Right now I'm running an internal 2.5Gb network using some Trendnet switches. I'll be replacing the switches with MikroTik (CRS304-4XG-IN at the patch panel, CRS305-1G-4S+IN in my study where I can use DACs to connect the machines), but I need to work out what to use for the router, and I'm worried about thermals.

The ISP's router will be 10GBASE-T -- so, another RJ45 connector. That means I'll need to have two RJ45 10Gb networking ports, one from the ISP and the other to the wall socket. The SFP+ modules for 10Gb run pretty hot.

My router setup right now is a Protectli VP2420 running Arch, with a custom setup (iptables, Bind, dhcpd, half a dozen VLANs, etc -- no VPNs), and I'm planning to essentially port that over to a 10Gb-based machine. The ones I'm considering are the Protectli Vault VP2440, the VP6630, and the Minisforum MS-01.

Does anyone have experience with using any of those machines for 10GBASE-T? In particular, would two SFP+ 10Gb RJ45 modules in one of them make it run so hot it started throttling?

If the worst comes to the worst I can move the RJ45 stuff out to a "sacrificial" switch (maybe another CRS305) and then just use DACs to the router, but it starts getting a tad complicated, so I'd like to avoid that if I can.

(Also: would the VP2440 be fast enough to do this kind of routing?)

TIA for any thoughts!


Giles

 

[louie1961]

Using a DAC in an SFP+ will generate very little heat. Using fiber in an SFP+ will generate a bit more heat. But using an RJ-45 tranceiver in an SFP+ is where the heat really comes into the picture. two RJ-45 transceivers in one of those protectly boxes could pontentially cause heat issues.

I would eitherfind a firewall device without SFP+ ports (native RJ-45) or build one. Something like an Intel X550-T2 card would make a bang up NIC for a firewall device.The one I found for my Synology actually autonegotiates down to 5gbe and 2.5gbe as well. Kind of neat. Your other option is a switch with both SFP+ and RJ-45 ports.Instead of having your WAN connection go direct to the firewall device, you could have it go to the switch, then through the magic of VLANs, connect your firewall device to the switch with SFP+. I have a QNAP 16-Port 10GbE managed switch (QSW-M3216R-8S8T-US) that has 8 RJ-45ports, and 8 SFP+ ports. It runs very cool and quiet. I am sure Microtik has something similar, but I don't own any Microtikso I can't advise you there. There's lots of tutorials on line on how to run your WAN direct to your switch instead of to the firewall.

Don't discount building your ownfirewall device. I built mine using an Asrock Industrial IMB-V2000M embedded ITX board, and Mellanox ConnectX 3 NIC. Works great, and I actually run Proxmox on it, and virtualize my pfSense. I also run my PiHole on that box as an LXC container. Same for Crowdsec.

 

[gpjt]

Thanks, louie1961! Yes, my fallback if the PC itself can't handle two RJ45 modules is to use a MikroTik CRS305 SFP+ switch. Put RJ45 modules in ports 1 and 4, then set it up to connect ports 1 and 2, and 3 and 4 separately. Then the PC can go via DAC to port 2 for WAN and 3 for LAN -- essentially your VLAN magic but with hardware rather than software. That would move the heat out of the PC and into the switch, which is more designed for it (and also less of an issue if the heat kills it).

(Do correct my if I'm wrong on this, but I've got the impression that RJ45 SFP+ modules run hot but so do "native" RJ45 ports on switches/PCs -- I had an RJ45 10Gb card for my PC a while back and its heatsink is pretty massive. It seems like the advantage of using something with native 10GBASE-T built in comes from the fact that the designer of the machine -- the manufacturer or in a custom job like yours, you -- will have taken that into account.)

 

[louie1961]

I personally would avoid RJ-45 modules at all costs. They really do get very warm. The native RJ45 ports in my switch are no where near as hot as RJ45 modules. Maybe its the built in heat sink, I don't know. But I had them here in my environment for a while and I won't use them unless it is urgent and temporary. In my current setup the links from my WAN are RJ45 (but I don't have 10gbe internet), using the onboard NICs on my Asrock motherboard. My link to my switch and the link from my switch to my Proxmox server and my TrueNAS storage server are all SFP+ DACs. My only 10g RJ45 connection is to my Synology NAS with its X550 card. I went RJ45 on that device intentionally because it is the only piece of hardware my wife would keep if I ever kick the bucket. Otherwise SFP+ is better in all ways in my opinion. Cooler, lower power draw, and lower latency.

 

[gpjt]

Interesting! I wonder if that's because there's something inherently high-wattage in converting SPF+ signals to 10GBASE-T (so they're hotter just because of that), or if avoiding the SPF+ stage just makes it easier to thermally couple the RJ45 plugs to the rest of the chassis (so they're not hotter just because the heat is spread better). Either way, it's super-useful information

Any thoughts on what kind of CPU power you need to route a symmetric 10Gb ISP uplink using low-level stuff like iptables (which I imagine pfSense is using under the hood)? Intel N150 vs i3 vs i5?

 

[louie1961]

It may just be the form factor. In a switch, every PHY has a heat sink normally. In a sfp+ transceiver, there really is no heat sink and you are packing the electronics into a small package.

Any thoughts on what kind of CPU power? Well, yes and no. I would say go look at the CPUs used in the Protectli or Netgate 10gbe devices and check out their benchmarks. Then aim for a CPU with similar or better bench marks. My Asrock Industrial IMB-V2000M has a AMD embedded Ryzen V2718 CPU. That is a 15 watt part and Zen 2 architecture. Pretty old stuff. it runs a base frequency of 1.7 GHz and turbos up to 4.2 GHz. 8 cores/16 threads. That's roughly equivalent to a Ryzen 7 2700 or a Intel Core i9-9900.

That CPU runs an entire hypervisor (Proxmox) and pfSense in a VM with only 6 cores and 4mb of memory assigned to it. and it routes 10gbe internally for me all day long, without breaking a sweat, at line speed. That same CPU is also running Crowdsec (LXC) PiHole (LXC), cloudflared tunnel software (LXC), and a debian VM as a docker host (for only a few key network/edge related containers like vaultwarden, NPM, Librespeed, etc.). So that's a long winded way of saying I don't think it takes too much CPU power. BUT to run a 10gbe NIC at full speed in both ports, you need a minimum of 8 PCI lanes. Lots of the lower speed parts like the N100 only have 9 PCI lanes total. That will handicap your 10gbe NIC. That's why I decided to build my own device. A number of the fanless firewall devices out there with two or more sfp+ ports use a CPU with only 9 total PCI lanes. That's a trap you don't want to fall into.

 

[TrevorH]

As a data point, I ran iperf3 --bidir from my N100 opnsense box to an internal machine and got 2.22Gbps in each direction and overall cpu was ~25% idle, 75% used.

 

[gpjt]

Makes sense re the thermals, louie1961. Either way, the solution seems clear.

Re: CPU -- thanks both to you and to TrevorH! Useful data points. I've found that on my desktop I can get solid 10Gb using iperf3 on a 4x PCIe card (running a point-to-point link to another PC). So with two 10Gb ports, I think I'll be safe with 8 available lanes -- which seems unlikely in a N100 with only 9 in total...

 

[louie1961]

So with two 10Gb ports, I think I'll be safe with 8 available lanes -- which seems unlikely in a N100 with only 9 in total...

The problem is that many of the 10gbe NICs, like the Intel X520 NIC that is in all the fanless firewall units, are engineered for PCI 2.0X8 lanes. Four lanes of PCI 3.0 ought to be enough bandwidth, but those NICs can't use it. There are newer NICs that support PCI 3 and maybe PCI 4. If you get one of those, you can theoretically get by with a CPU that only provides 9 lanes. BUT this assumes the device isn't using the PCI lanes up for NVME drives, SATA controllers, 2.5gbe NICs, WiFi, USB controllers, etc. Even with a modern NIC chipset on PCI 3.0, you still need four PCI lanes of the nine total, to be available. That's kind of a tall order, unless the device is designed for a dual port 10gbe NIC. Most of those are built on Atom CPUs.

Anyway, buyer beware. There's devices out there that are not what they seem.

 

[Scott Laird]

Doing 10Gbase-T requires a lot of processing with a hefty DSP, and this historically used 2-4W of power. The newest modules are a bit better, but there's still a lot of work involved, and the newest/lowest power/coolest ones are usually the most expensive.

I'm using a MS-01, but I'm using a dual-port ConnectX-5 in it with 2x40G links. I threw a cheap USB fan on top of it, and that's been enough to keep the MS-01 and CX5 cool enough to be happy for most of a year. Without the fan the NIC will heat up to 120C and shut down within a few minutes. With fiber SFP+s or a DAC cable, I wouldn't worry about heat at all. With high-power copper modules, I'd probably consider adding a fan. I literally just bought a 120mm USB model off of Amazon and plugged it into the USB port in front.

I wrote this whole thing up last year at Routing with VyOS on a Minisforum MS-01, part 1: Background

 

[gpjt]

Thanks all! I'll get going on this and will report back when I've got something up, running and (importantly) tested over a few days.

 

[gpjt]

I now have 19 days of data on this setup, and it looks like it's pretty solid! I have a Protectli Vault VP2440, with Protectli-branded 10GBASE-T modules in both of the SFP+ cages. One is connected to my ISP's box, the other to a Mikrotik CRS304-4XG-IN 10GBASE-T switch. The ISP box, the Vault and the switch are inside a sideboard, with limited airflow -- I wanted to see it it could handle that before trying to do anything to address it.

I have Telegraf reporting the CPU and SFP+ 1 and 2 modules on the Vault to Influx, and have a Grafana dashboard to monitor the results. Here are the charts:


Late on the 16th, I put a Unifi U6 Enterprise in the same sideboard cupboard, so this is about as pathologically bad a setup as you can have -- the AP is moving elsewhere later -- and it's still a perfectly manageable temperature. The upward trend is just the weather, I think -- it's been unseasonably warm lately.

I think that the Protectli must have excellent thermal conductivity between the SFP+ modules and the case.

I'm a little more concerned about the temps inside the CRS304-4XG-IN switch, which is in the same sideboard:


That's above the 70C official operating range for the switch. That's a result of the location; another one I have elsewhere is averaging about 57C. Still, no issues with networking, so it seems OK.

Finally, just to round things off, to support what others have said in this thread about SFP+ 10GBASE-T modules running hot in general; I have a separate CRS305-1G-4S+IN in my study, excellent ventilation. It has a single 10GBASE-T SFP+ module going to the house's structured cabling (everything else using DACs), and you can see that that is running super-hot:


I think I'll get one of those mini-heatsinks that people use on Raspberry Pi, and see if that helps.

Thanks to everyone for the help in this thread, and I hope the charts were interesting

 

[blunden]

I have a Protectli Vault VP2440, with Protectli-branded 10GBASE-T modules in both of the SFP+ cages.

[...]

Finally, just to round things off, to support what others have said in this thread about SFP+ 10GBASE-T modules running hot in general

The Protectli 10GBASE-T transceivers appear to be the kind with the old and power hungry PHYs, seeing as they are rated for only 30 meters and the high temperatures you're seeing. You'll need to replace those with modern ones rated for 80 or 100 meters if you want to meaningfully reduce the temperature.

There are a few threads about them on here, including the one below:

https://forums.servethehome.com/index.php?threads/10gb-sfp-rj45-100m.45817/

 

[gpjt]

I'm actually much more concerned about the temp on the MiktoTik one on the last chart -- 93C is definitely hotter than I'd like it to be. Let's see what one of those Raspberry Pi heatsinks does.

For the Protectli, I think that I'll move the WiFi AP elsewhere and improve ventilation before I try anything else. If I can get them to a reasonably stable 62C or so, I'll be happy enough.

 

[gpjt]

Well, the Raspberry Pi heatsinks helped... but unfortunately, they didn't help very much! I installed two of them right at the mid-point of this chart of the last 48 hours, and while you can see that there was a drop, it was pretty small:



Don't be fooled by the Y-axis, that's just a 3.5 C improvement in temperature :-(

Sounds like you were right, @blunden ! I'll keep things as they are for now, but if things start breaking as we move into summer, I'll switch to newer SFP+ modules -- specifically, ones that are rated for >30 metres.