Help me rationalise my network please

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

crembz

Member
May 21, 2023
35
0
6
Hey everyone, I've got a home lab environment which has grown in recent weeks and I'm thinking my network is holding me back.

My entire network is currently 1gb unifi.

My network looks like this:

Udm se - usw8, flex mini, main desktop - unifi ap

Usw8 (Poe)- 5 proxmox hosts (one running virtual truenas and in lacp), usw8 #2

Usw8 #2 (Poe) - 3 proxmox hosts, unifi ap

Flex mini - tv, Xbox, unifi ap

The proxmox hosts are vlan aware and run a number of nested hypervisors on different vlans as well as a few jump boxes and media streaming services.

I'm getting very poor performance to the Nas especially with 4k tests. Truenas is running 5 HDD SATA mirrors on a SAS card and a zil mirror.

Given I'm not routing at the switches, any inter vlan traffic to the Nas needs to hit the udm over a shared 1gb link and come back down the same link. Sounds like bottleneck central to me.

My walls are called with cat6a.

I ran a 4k fio test locally on the Nas box and then on another VM on the same subnet. Even though they're on the same subnet, I got about 30% of the iops and throughout on the VM.

I'm trying to decide the best way forward to go 10gb from the proxmox hosts to the Nas.

In each of these options I'd be connecting the Nas host with 10gb

Option one is run a flat network with a basic l2 managed switch maybe an old juniper ex 3300 or something (not sure if a new mikrotik would be wiser)

Option 2 Buy a unifi pro l3 switch and keep the vlans

Option 3 keep the vlans with a basic l2 10gb switch and replace the udm se with a proper 10gb capable router having a single 10gb uplink between the switch and the router. (Maybe the same ex3300 plus one of those little r86s box with pfsense)

I'm not sure there are any other viable options. I dont really want to recable the walls so I'll probably use rj45 transcievers if I go option 3.

Any feedback or ideas?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
You're missing the most relevant aspects re recommendations:
Budget, size (desktop/rack), noise/heat, location

You can get old enterprise switches with 10G uplinks for 20 bucks or so (maybe not, havent checked, but those 10 year+ old hp/dell switches are sure not worth much more).
Most enterprise switches are L3 capable (although sometimes you need a license that may or may not be available)

A more sensible approach is to
1. Identify how many 10G ports you need - Fibre or Base-T, and how many regular Base-T you need. Do you need PoE? Remember Fibre to BaseT adapaters are not cheap and produce heat too.
2. Identify Budget, noise and space constraints
3. Decide whether you want to stay on Unifi or switch to something like Ruckus or another brand. Note that Unifi does not get much love here (from me either) for various reasons;) Typical recommendation here would probably a Ruckus switch...

p.s. There is no discussion *if* you need to move to 10G here at all, I think most here deem that a mandatory minimum for a home network;)
 

crembz

Member
May 21, 2023
35
0
6
You're missing the most relevant aspects re recommendations:
Budget, size (desktop/rack), noise/heat, location

You can get old enterprise switches with 10G uplinks for 20 bucks or so (maybe not, havent checked, but those 10 year+ old hp/dell switches are sure not worth much more).
Most enterprise switches are L3 capable (although sometimes you need a license that may or may not be available)

A more sensible approach is to
1. Identify how many 10G ports you need - Fibre or Base-T, and how many regular Base-T you need. Do you need PoE? Remember Fibre to BaseT adapaters are not cheap and produce heat too.
2. Identify Budget, noise and space constraints
3. Decide whether you want to stay on Unifi or switch to something like Ruckus or another brand. Note that Unifi does not get much love here (from me either) for various reasons;) Typical recommendation here would probably a Ruckus switch...

p.s. There is no discussion *if* you need to move to 10G here at all, I think most here deem that a mandatory minimum for a home network;)
Thanks for the feedback.

Ok let's break it down a little more.

I'm thinking a budget of about usd1200. Happy to ditch ubiquiti and back to a more open ecosystem. I'll probably stick with their access points though.

System will be in the house in a spare room. Noise is a consideration which is why I haven't just bought an old juniper switch or something for the minute. Doesn't need to be whisper quite but I don't run rack servers for a reason (as tempting as a cheap dual CPU Xeon with 256gb of memory is on the used market)

Keeping an open mind about desktop vs rack. I have 4 cabled points throughout the house so uplinks I'd like to stick with rj45. Anything else I'm happy to go finer.

Drawing a quick diagram based on option 3, I'm looking at a router with at least 2x SFP+ and 4 x gbe capable of 10gb switching if using a l2 switch. The switch would need a minimum of 2x SFP+ however would prefer more for future proofing, and 9x gbe. The third switch would be a simple 4x gbe at a minimum.

Poe would be great but happy to work around that for the right product.

I keep reading about mikrotik but I'm not sure that's the right direction to go with so many complaints about software I keep reading especially with the CCS range. Let me take a look at rukus.
 

crembz

Member
May 21, 2023
35
0
6
Those icx units look like they could do the trick! I have some reading to do.

Now for a 10gb router!
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
The ruckus should do l3 too. You might need to enable on newer models or ask @fohdeesha for a license on the older ones. Read up first:)
 

crembz

Member
May 21, 2023
35
0
6
Thanks for that, I've found a 7250-48p on eBay which should come in much cheaper than an enter level mikrotik and about 30% the cost of a unifi aggregation switch.

Looks like they do in fact do l3 but I'm a little confused about the licensing. Seems like you need optional licenses to enable l3 and 10gb ports Reading through the thread you posted it sounds like maybe licensing is now free since they're eol?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Well on the 6xxx series you can get a license for free from the user i mentioned since he has some to spare and they are EOL.

On the 7xxx series the license is honour based so basically you can activate for free. Now if you're a company and you earn money with them you o/c should not (you'll want support anyway), but for home this is something that everybody can decide for themselves,
 

crembz

Member
May 21, 2023
35
0
6
Just found the guide, looks pretty good to get started. Hopefully I can work out vlan routing without too many issues. If I can get that working I might not need a new router.

If not, what sort of router would be a good match for the icx switch?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I have no idea, I've been routing in my FW for years, but prolly others can chime in :)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I've been running Sophos UTM (old one, not XG) for years, but OPNSense/pfsense will do the job just as fine.
I run Sophos since it provided a simple way to join multiple locations together with RED devices (all virtualized o/c, with 2 Sophos @Home 50 Ip based licenses)
 

crembz

Member
May 21, 2023
35
0
6
Got it, so the setup would end up looking like

Wan->pfsens->icx->VM hosts & Nas & AP
| |
| V
| Workstation & AP
V
Managed switch
|
V
TV, Xbox & AP

So question ... Would it be a better idea to operate the icx in l3 and route the VM vlans there or pass everything through to the pfsense box (likely to be the r86s at this point). I'm thinking it might be overly complex to have both the icx and pfsense box routing in the network.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
You want only one box doing routing for a small setup.
Both will work, pick whats easier for you.
I like fw cause that gives me more control what actually can pass one way or the other (blocking cloud-oriented iot devices for example)
 
  • Like
Reactions: Amrhn

crembz

Member
May 21, 2023
35
0
6
You want only one box doing routing for a small setup.
Both will work, pick whats easier for you.
I like fw cause that gives me more control what actually can pass one way or the other (blocking cloud-oriented iot devices for example)
Got it so I'm going to need a fw that will handle 10gb routing... Not sure I know what to get there
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
If your inter-VLAN rules aren’t terribly complex, do it as ACLs in the switch. Then you do a transit VLAN to your firewall. Your firewall rules and NAT should all work just fine. No need for a firewall that can do 10g unless you have 10g WAN in that case.

Or do multiple physical links to the firewall so you don’t wind up with an over saturated link again.
 

crembz

Member
May 21, 2023
35
0
6
If your inter-VLAN rules aren’t terribly complex, do it as ACLs in the switch. Then you do a transit VLAN to your firewall. Your firewall rules and NAT should all work just fine. No need for a firewall that can do 10g unless you have 10g WAN in that case.

Or do multiple physical links to the firewall so you don’t wind up with an over saturated link again.
Yes I was thinking about multilink to the firewall but I keep reading that it is difficult to get pfsense routing intervlan at 10Gb speeds. I'd go down that route if I knew a HW config that would do it ... still researching
 

crembz

Member
May 21, 2023
35
0
6
So I've spent some time thinking about this and decided to get it down on paper. I'm thinking things are going to look something like this:

1684857522907.png

The L3 switch in room 2 will be an icx7250 from Brocade.
I'd like to get 10G between the main workstation and the virtual nas
I'm throwing a 10G on the router just in case I need it to route intervlan traffic
Hanging off the APs I'll have a bunch of smart devices, laptops and phones. Probably about 15 devices all up.
VLAN1: Management
VLAN2: PVE cluster, nested hypervisors, workstations, laptops
VLAN3: Smart devices and phones
VLAN4: Demo/lab VMs

I still need to decide on:

A router
2 x L2 switches
APs

Any advice on the above plan and recommended hardware? I'm really stuck on deciding on the router.

Notes:
  • I was thinking of either a used optiplex 7070 SFF wth i5-9600 & 8GB memory or a self built i5-12400 with pfsense. I'm not sure an off the shelf router will really be able to handle 10G routing between networks (although I may not really need it). I can't seem to find a guide on the minimum CPU requirements for pfsense to route 10GB between VLANs.
  • I was looking at the mikrotik range for the switches which I believe will need to be managed to trunk VLANs
  • I'd like to drop the need to run unifi for the APs but not too sure what's comparable and would also do wireless meshing for outdoor coverage.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Where do you see a need for true 10G Inter-VLAN traffic?
Lab VMs to storage? aren't those running on the PVE cluster?
 

crembz

Member
May 21, 2023
35
0
6
Where do you see a need for true 10G Inter-VLAN traffic?
Lab VMs to storage? aren't those running on the PVE cluster?
Correct, I'm not sure I'm going to end up needing to. I was just thinking I might account for it in case I need to do it at some point since this will be all new hardware.

I was also thinking I might be able to shave some cost and virtualise pfsense on the pve cluster, and pass the internet connection though on a seperate vlan, so internet-room 1 switch- room 2 switch- pve cluster-pfsense. I don't think that would introduce Intel vlan routing.