Setting up network/pfsense machine w/ 2.5gbe home connection

[semicole]

I recently got 2gig internet from Frontier (I am getting around 2.3gbe at the router) and want to revamp my home network to take advantage of the new speed and had some questions getting started. Before this I was stuck at 25-50mbps, so didn't ever really worry about any of my equipment. The 2gig connection is coming out of the fiber box and into my house on RJ45.

The devices that I have in my home that I am mostly concerned about from a LAN perspective are as follows below. Everything outside of the below devices will be fine operating on wifi or 1gbe LAN.

• Main PC (SFF, so no add on cards) which has an onboard 2.5gbe LAN port
• Unraid NAS (SFF, have room for one add on card) which has an onboard 2.5gbe LAN port

Currently I am setting up a PFsense box using a Lenovo M720q. I have ordered a 2.5gbe RJ45 M.2 adapter that will add a 2.5gbe port to the PFsense box, which I will use as my WAN port. From there I would like to add a managed switch and a WAP. I have everything small and SFF form factor, so would like to keep it all that way. I was considering the Mikrotik CSS610-8G/P for the switch. I was planning on getting an SFP+ NIC for the M720q to connect to the SFP+ on the switch and then connect all of the rest of my LAN through the Mikrotik.

My main questions that I need some assistance on are the following:

• What NIC should I get for the M720q to go from the PFsense box to the switch?
• With 2.5gbe coming into the PFsense box, once it hits the switch do I still need to worry about 1/2.5/10gbe? For example, my Unraid machine has a 2.5gbe onboard lan port, if I connect it to one of the 10gbe ports on the Mikrotik switch, will it be able to operate at 2.5gbe?

Thanks in advance for any help you guys can give.

 

[reasonsandreasons]

The second-generation Supermicro AOC-STGN-I2S is the usual recommendation for TMM, as it's a short card with good SFP compatibility (it's an OEM Intel X520).

I don't know if I quite understand your topology, though. I'm seeing modem into M720Q, which will use the 10G NIC to get into the Mikrotik over SFP+. You'll use the other SFP+ port for either the PC or the NAS, but either way one of them's getting left out in the cold. I think you need at least one more multi-gig port to make things work, unless I'm missing something? I've also heard some not-great things about SwitchOS lite, so it might be worth getting something else either way.

In any case, make sure you get a transceiver and switch that has support for 2.5GBase-T and 5GBase-T over SFP+. Mikrotik does, thankfully.

 

[semicole]

The second-generation Supermicro AOC-STGN-I2S is the usual recommendation for TMM, as it's a short card with good SFP compatibility (it's an OEM Intel X520).

I don't know if I quite understand your topology, though. I'm seeing modem into M720Q, which will use the 10G NIC to get into the Mikrotik over SFP+. You'll use the other SFP+ port for either the PC or the NAS, but either way one of them's getting left out in the cold. I think you need at least one more multi-gig port to make things work, unless I'm missing something? I've also heard some not-great things about SwitchOS lite, so it might be worth getting something else either way.

In any case, make sure you get a transceiver and switch that has support for 2.5GBase-T and 5GBase-T over SFP+. Mikrotik does, thankfully.

You are correct, I was mostly concerned about getting the NAS on 2.5gbe since I used it to download files and things, so I don’t mind connecting the PC to 1gbe.

The issue I have been struggling with is making sure that whatever switch/port I use can handle multi gig for my 2.5gbe devices. How do you determine that? For example on Mikrotik’s site for that switch I don’t see where it explicitly says it has multi gig support, just that it has SFP+. Do SFP+ ports natively support multi gig as long as you’re using a transceiver that is the speed/baseT you’re needing?

 

[DavidWJohnston]

I found this link inside the SVH article on the M720q; a riser card to plug in a 10G NIC, if you don't already have a riser:

English Community-Lenovo Community

For a 10G card, the Connectx-3 low-profile should work, you may need to add a small fan to keep it cool, and maybe double-check (measure) the space available in your case to make sure it'll fit: Mellanox MCX311A-XCAT ConnectX-3 EN 10G Ethernet 10GbE SFP+ PCIe NIC w/2 Bracket | eBay

Now for the switch - That model appears to not support 2.5G. The SFP ports are listed as 10G. See this page, that switch is not in the list that support 2.5G: MikroTik wired interface compatibility - MikroTik Wiki

I think you are also short a port - Going with your design, you need:

• One 2.5G port for your NAS
• One 2.5G port for your main PC
• One 10G port for your pfSense box
• Enough 1G ports for your other devices

I will mention there is another way to design this network; that is have only 1 NIC in the pfSense box, then use VLANs for LAN/WAN separation. All of your devices, including ISP modem would plug directly into the switch. So technically you could save a NIC - Of course then you'll need an additional 2.5G port on the switch for your modem.

For your last question, for 2 local devices communicating with each other, as long as they are on the same VLAN (ex. your single LAN) the packets will not go through pfSense, they stay local to the switch. The switch will take care of providing the max speed possible depending on the capability of both devices that are talking.

If I understand your situation correctly, try looking around for a different switch. If you have trouble I can do some research for you.

 

[semicole]

I found this link inside the SVH article on the M720q; a riser card to plug in a 10G NIC, if you don't already have a riser:

English Community-Lenovo Community

I do have the riser already fortunately.

For your last question, for 2 local devices communicating with each other, as long as they are on the same VLAN (ex. your single LAN) the packets will not go through pfSense, they stay local to the switch. The switch will take care of providing the max speed possible depending on the capability of both devices that are talking.

So if I have a switch that has a 10gbe port and it is communicating with a 1gbe or 5gbe device for example, it will auto negotiate to the speed of the lowest speed device? So you only have to worry about the port speed matching between two devices when it's externally facing i.e. WAN or when do you have to worry about the port speeds matching? This is the part in all of this that I have had the hardest time wrapping my brain around. For the longest time I always thought that basically all networking devices would just raise or lower their speed depending on the bottleneck in the network and that a 10gbe device for example could do anything from 1-10gbe.

If I understand your situation correctly, try looking around for a different switch. If you have trouble I can do some research for you.

I could definitely use the help, just looking for something relatively cheap, within a similar price range as that Mikrotik and a similar form factor.

 

[DavidWJohnston]

In all cases, whether it's a NIC or switch port, the port needs to support the speed of the device to run at that speed. There are some networking devices (NICs, switches, etc) that are "multi-gig" and can do any of 10M/100M/1G/2.5G/5G/10G all on the same port. If you plug in a device that's 2.5-capable, it'll run at a max of 2.5.

If you plug a device that supports 2.5G into a port that does not have 2.5G support (whether a switch or another NIC), it will only ever run at 1G. Just because a port supports 10G, that does not mean it will do 2.5G & 5G. (But usually they will do 1G)

The switch will forward packets as fast as it can per conversation - Which you are correct this means the speed will be that of the slower device in the conversation. If there are other devices having a conversation, the speed of that conversation may be different; again the fastest possible speed both the source and dest support.

So port speeds don't necessarily need to "match", but the speed needs to be supported by that port in order to negotiate. For example if you bought 1G/2.5G/5G/10G M.2 NIC for your Lenovo, when you plug in your 2.5G ISP modem it'll negotiate at 2.5. Same for a switch port.

To give another example, in my homelab I run a Celestica Seastone DX010. Each port can run at 1G, 10G, 25G, 40G, 50G, or 100G. So if I plug a 2.5G device into my switch, that port will only run at 1G because 2.5 isn't supported. But at the same time, there could be 100G devices talking on other ports, at 100G.

I hope that makes sense, I probably could summarize it better.

For the switch, are you open to used equipment? What about power and fan noise?

 

[DavidWJohnston]

I believe this switch would work for you, and it has L3 support which could be useful in the future: MikroTik

Since the switch has only SFP+ ports, you need transceivers. You could consider these: (For reference only)

Edit: The Mikrotik RJ10 modules I think will work in-place of all of the RJ-45 below options, so this may be preferable - But they may run hot.

For 2.5G: 2.5GBase-T 2.5G SFP-T to RJ45 Copper Transceiver Module For Cisco,Ubiquiti UniFi | eBay

For 1/10G: 10G SFP+ to RJ45 Copper Module 10gb SFP RJ45 Module SFP 30M For Cisco Mikrotik | eBay

For 1G Only: Finisar FCLF8521P2BTL SFP-1G-T 1000BASE-T SFP Copper RJ-45 transceiver | eBay

For SFP-to-SFP: SFP+ DAC Twinax Direct attached Copper Cable 30 AWG For Cisco SFP-H10GB-CU0.3M | eBay

For 10G fiber, they are much cheaper than RJ45: 10GBase-SR SFP+ Transceiver For Cisco SFP-10G-SR multimode SFP 850nm up to 300 M | eBay

To plug all your 1G equipment in, you may need another inexpensive gig switch to daisy-chain into an SFP port.

I did a bit of research, Mikrotik appears to not be brand-picky about SFPs, but getting a second opinion from someone on the forum would be best.

According to this page: MikroTik wired interface compatibility - MikroTik Wiki the CRS3xx switches are all 2.5G-capable.

 

[reasonsandreasons]

The CRS305 is tricky because it can only really accommodate two SFP+ to RJ45 adapters without risking thermal issues, so you'll need to use SFP+ for the remaining connections. If you're genuinely okay with 1G for everything but the NAS, a MikroTik CRS326-24G-2S+in might be good.

If you do want the extra high-speed connectivity, you can easily use the switch's RJ45 port for meaningful networking. It's connected up to the switch chip, so you could daisy-chain your existing gigabit switch.. The one caveat is that you'll be sharing a single 1G link to all the ports on any downstream switch, so you'll be effectively creating a section of the network that's limited to 1G bandwidth to the NAS and internet.

In either case, I'd consider pricing out what it would cost to just upgrade the NAS to 10G; SFP+ to RJ45 hardware isn't cheap and, in the case of the CRS305, connecting two devices with DACs would let you use the remaining two points for 2.5G networking.

 

[DavidWJohnston]

Interesting! I can see Reddit posts about it running hot. That's exactly the kind of second opinion I was hoping for.

2.5G SFPs run a lot cooler than 10G, especially for short cable runs. Adding a little fan to blow air across/into the case would work well - But yeah DACs would be better, where possible. Space around the unit will help too.

It's amazing how effective a small amount of airflow is at cooling equipment. I have a bank of PCIe cards that get too hot to touch, but just a super-slow fan above them totally solves the problem.