Setting up a 10gig interbuilding network at home

[Pallee]

Hi all!

I am very green in enterprise gear networking with VLANs and all but figured it is going to be a fun project with much learning involved. My plan is to wire two buildings (garage and main living) on my property together with 10gig networking over single mode fiber (for futureproofing and possibility to go BIDI if need arises) to keep the noise and heat in the garage

I just pulled the trigger on 2x brocade icx6450-24 (4x 10G SFP+ and 24x 1GBaseT, non PoE) for €190 delivered to Sweden including taxes. (Waiting for delivery)

Current networking equipment:

• NAS based on a Supermicro X10SDV-8C-TLN4F (integrated x540-T2 NIC) + IPMI
• pfSense router in an older Dell Optiplex with 1 intel I350 (4x 1GbaseT) + 1 intel V-pro enabled NIC
• Workstation: intel x540-T2
• SO's WS: integrated I217-V, but thinking of installing an x540-T2
• VM-host: integrated whatever 1GbaseT it works, apparently... but I would like to run 10gig
• 1x intel X710-T4 (4x 10GBaseT) network card, unused at the moment.

Do you have other hardware recommendations? I know that I will need SFP+ optics and 10GBaseT adapters. Do I need attenuators for the optics since the fiber runs will be in the 50-meter range compared to multi-Km range the optics are usually specified for?

What is the best practice to setup my network?

• I would, if possible, like a 20G link between my switches. Should I do this with the stacking functionality or some other LAG?
• How should the ISP connection be forwarded to the pFsense-box? IP-address is provided from ISP via DHCP.
• I would like a management VLAN for my switches, IPMI devices etc.
• A separate VLAN for IoT/smart home stuff would be nice.
• Have I missed something really obvious to an experienced network guy? (of course I have )

Please see the attached sketch for my proposed hardware layout and future expansions if needed:

 

[altmind]

>I would, if possible, like a 20G link between my switches. Should I do this with the stacking functionality or some other LAG?

yes, go with LAG, LACP (unless you are stacking - then it may be unnecessary)

> How should the ISP connection be forwarded to the pFsense-box? IP-address is provided from ISP via DHCP.

sorry for that no answering you question, but i would rather change the setup and move the pfsense to be closer to the ISP handoff - to be between your living room and isp. router-on-a-stick is far from optimal.

> I would like a management VLAN for my switches, IPMI devices etc.

good idea.

> A separate VLAN for IoT/smart home stuff would be nice.

good idea

> Have I missed something really obvious to an experienced network guy? (of course I have )

i would make 2x icx6450 a stack. having a single address space and switch management interface is a big advantage.




you have a decent setup, good luck!

 

[Pallee]

sorry for that no answering you question, but i would rather change the setup and move the pfsense to be closer to the ISP handoff - to be between your living room and isp. router-on-a-stick is far from optimal.

Is there a particular reason for this?
Could I not just keep the WAN traffic on an isolated VLAN through the switches ending in on my "WAN" interface on the pfsense box, pfsense doing pfsense things and then forward the traffic to the "LAN"-side interface?
I understand this adds a little bit of complexity, but I (and my SO ) really like the idea to keep as much networking as possible in the garage.... I could, of course, add a fifth fiber to the garage and move my media converter too like this:

you have a decent setup, good luck!

Thank you!

 

[mattventura]

> single mode fiber

Good choice. Don't fall for the multi-mode trap.

> NAS based on a Supermicro X10SDV-8C-TLN4F (integrated x540-T2 NIC) + IPMI
> pfSense router in an older Dell Optiplex with 1 intel I350 (4x 1GbaseT) + 1 intel V-pro enabled NIC

So, I actually run a virtualized router on an X10SDV-8C-TLN4F along with other stuff. You could potentially forget the old Dell and do a lot more on the X10. This is especially true if you want to do multiple VLANs, since the router will need to be able to handle inter-VLAN routing as well. Mine can easily max out the 10GbE link (can theoretically do 20, tested with virtual interfaces) using OpenWRT. Since it has SR-IOV support, you don't even need dedicated host/guest ports - every host and guest can share every port, without the overhead of software switching. Only downside is that VM-to-VM or host-to-VM traffic has a 10GbE full-duplex bottleneck on the internal links from the CPU to the NICs (i.e. if two hosts are transferring 10Gb/s bidirectionally at the same time, it will only be able to do 5Gb/s).

> Workstation: intel x540-T2
> SO's WS: integrated I217-V, but thinking of installing an x540-T2
> VM-host: integrated whatever 1GbaseT it works, apparently... but I would like to run 10gig
> 1x intel X710-T4 (4x 10GBaseT) network card, unused at the moment.

Why 10GBaseT? You'll just have to pony up for 10GBaseT transceivers which can cost more than just buying an SFP+ card and using direct attach cables.

> Do I need attenuators for the optics since the fiber runs will be in the 50-meter range compared to multi-Km range the optics are usually specified for?

Nope, see page 77.

> I would like a management VLAN for my switches, IPMI devices etc.
> A separate VLAN for IoT/smart home stuff would be nice.

Both good ideas, just keep in mind what I wrote above - inter-VLAN routing will be bottlenecked on your router.

> Have I missed something really obvious to an experienced network guy? (of course I have )

Depending on what the ISP is giving you, you might not need a media converter in the first place. If it's FTTH, you might be able to buy an appropriate SFP transceiver and skip the converter entirely.

 

[Pallee]

So, I actually run a virtualized router on an X10SDV-8C-TLN4F along with other stuff. You could potentially forget the old Dell and do a lot more on the X10. This is especially true if you want to do multiple VLANs, since the router will need to be able to handle inter-VLAN routing as well. Mine can easily max out the 10GbE link (can theoretically do 20, tested with virtual interfaces) using OpenWRT. Since it has SR-IOV support, you don't even need dedicated host/guest ports - every host and guest can share every port, without the overhead of software switching. Only downside is that VM-to-VM or host-to-VM traffic has a 10GbE full-duplex bottleneck on the internal links from the CPU to the NICs (i.e. if two hosts are transferring 10Gb/s bidirectionally at the same time, it will only be able to do 5Gb/s).

Cool! I will think about it, but I prefer to have my router/firewall as a standalone box.

Why 10GBaseT? You'll just have to pony up for 10GBaseT transceivers which can cost more than just buying an SFP+ card and using direct attach cables.

Several reasons.
1) I have the suff, and it is much cheaper to reuse what I already have, compared to get new nics, DACs etc. (Getting a X10SDV-8C or similar with SFP+ instead is not cheap)
2) Because BaseT copper is much more flexible with consumer devices when you wire up the entire house.
That said, if I need more 10gig stuff on the back end than I have today I will surely look for SFP+ gear

Both good ideas, just keep in mind what I wrote above - inter-VLAN routing will be bottlenecked on your router.

I may be severely mistaken, but I was under the impression the switches should be capable of inter-VLAN routing... but I am willing to stand very much corrected on that point.

Depending on what the ISP is giving you, you might not need a media converter in the first place. If it's FTTH, you might be able to buy an appropriate SFP transceiver and skip the converter entirely.

Well, I could pop the SFP module from the converter, and the switches should be very agnostic when it comes to the modules... but I will be wasting a 10gig port on the 1gig connection. But it is surely an option.


Thank you very much for the input!

 

[mattventura]

> I may be severely mistaken, but I was under the impression the switches should be capable of inter-VLAN routing... but I am willing to stand very much corrected on that point.

Oh, you certainly can do that as well.

> Well, I could pop the SFP module from the converter, and the switches should be very agnostic when it comes to the modules... but I will be wasting a 10gig port on the 1gig connection. But it is surely an option.

Another way to do that would be to add a card to whichever box you end up using as the router.

 

[kpfleming]

I have a pair of SMF running between our house and garage (total length is around 30m), using 10Gtek BiDi SFP+ modules, for a 20G link. Works very well, I'd recommend it

 

[Pallee]

Another way to do that would be to add a card to whichever box you end up using as the router.

Do you have a good NIC to recommend? I am very new to all SFP stuff, but from what I gather compatibility between brands and modules can be a PITA. I was looking at the intel NICs due to stable drivers, but they seems to be quite picky... tbh I am at a loss atm...

 

[nexox]

I'm not sure how much luck you'll have using the ISP SFP module in a router, some ISPs make that pretty difficult, but assuming you're using Linux then SolarFlare S7120 cards are pretty good, cheap, and at least so far as I have seen, will accept any SFP modules.

 

[mattventura]

Do you have a good NIC to recommend? I am very new to all SFP stuff, but from what I gather compatibility between brands and modules can be a PITA. I was looking at the intel NICs due to stable drivers, but they seems to be quite picky... tbh I am at a loss atm...

I'm not sure which ones support 2.5g SFP+ (which may be important depending on the ISP), but X520 is fine for Linux usage as it can be told to ignore SFP restrictions using a kernel boot option. Mellanox is also fine, SFN7120 as the poster above said, and many more. Just depends on feature set, really. I tend to prefer Intel because I've found their SR-IOV support to be pretty good for router operations.

 

[Pallee]

[...] but assuming you're using Linux then SolarFlare S7120 cards are pretty good, cheap, and at least so far as I have seen, will accept any SFP modules.

How about FreeBSD? The router is running pfsense. Found conflicting info when I tried to look around for an answer.

I'm not sure which ones support 2.5g SFP+ (which may be important depending on the ISP), but X520 is fine for Linux usage as it can be told to ignore SFP restrictions using a kernel boot option. Mellanox is also fine, SFN7120 as the poster above said, and many more. Just depends on feature set, really. I tend to prefer Intel because I've found their SR-IOV support to be pretty good for router operations.

I'am quite sure that the x550 and x710 based cards support multi-gig speeds while x520 and x540 does not. However, my ISP only provides 1gig, so anything else is just to "futureproof" (tm).
I read on the forum that it may be possible to modify the NVM on the intel cards to default to accept all modules, regardless of drivers/OS.

I really appreciate the friendliness of this community, Cheers

 

[mattventura]

I'am quite sure that the x550 and x710 based cards support multi-gig speeds while x520 and x540 does not. However, my ISP only provides 1gig, so anything else is just to "futureproof" (tm).

Yes and no. The X550 does, but it comes with RJ-45 ports rather than SFP+, whereas the X520 can have either, but doesn't support multi-gig. The spec sheet for the X710 copper version supports it, but not the SFP version as far as I can tell. There's supposedly a way to do it with certain Broadcom NICs but is a complicated process. But if you only need 1G (and it's not using a faster tech but throttled down), then there's tons of options.

I really appreciate the friendliness of this community, Cheers

No problem!

 

[blunden]

If you need SFP+ modules that connect to the device side with 10 Gbit/s and to the other side with 2.5 or 5 Gbit/s speeds, there are Aquantia/Marvell modules that can do that much better than the Broadcom modules based on other posts on this forum. 1 Gbit/s modules are much simpler though if that's all you need.

 

[Dave Corder]

Yes and no. The X550 does, but it comes with RJ-45 ports rather than SFP+, whereas the X520 can have either, but doesn't support multi-gig. The spec sheet for the X710 copper version supports it, but not the SFP version as far as I can tell. There's supposedly a way to do it with certain Broadcom NICs but is a complicated process. But if you only need 1G (and it's not using a faster tech but throttled down), then there's tons of options.

I run a Broadcom card with a 2.5 Gbps NBase-T SFP+ module in OPNsense to connect to my Xfinity XB7 in bridge mode (based entirely on the work in that thread for the 2.5 Gbps GPON modules). Works great. It's not actually that hard - the kernel module you need to use is available pre-compiled from that thread, so you just need to copy it to the firewall and drop it in the right directory. The only hitch is that the module will get removed/replaced whenever you update OPNsense, so after every update I won't have WAN access and will have to ssh into the box and re-copy the kernel module to the directory (I keep a copy of it in /conf on the firewall itself, so it's always there when I need it). But it'll persist on reboots, so you only need to do this when updating OPNsense itself (and so I end up being lazy and don't update it very often).