Hey everyone, I've got a home lab environment which has grown in recent weeks and I'm thinking my network is holding me back.
My entire network is currently 1gb unifi.
My network looks like this:
Udm se - usw8, flex mini, main desktop - unifi ap
Usw8 (Poe)- 5 proxmox hosts (one running virtual truenas and in lacp), usw8 #2
Usw8 #2 (Poe) - 3 proxmox hosts, unifi ap
Flex mini - tv, Xbox, unifi ap
The proxmox hosts are vlan aware and run a number of nested hypervisors on different vlans as well as a few jump boxes and media streaming services.
I'm getting very poor performance to the Nas especially with 4k tests. Truenas is running 5 HDD SATA mirrors on a SAS card and a zil mirror.
Given I'm not routing at the switches, any inter vlan traffic to the Nas needs to hit the udm over a shared 1gb link and come back down the same link. Sounds like bottleneck central to me.
My walls are called with cat6a.
I ran a 4k fio test locally on the Nas box and then on another VM on the same subnet. Even though they're on the same subnet, I got about 30% of the iops and throughout on the VM.
I'm trying to decide the best way forward to go 10gb from the proxmox hosts to the Nas.
In each of these options I'd be connecting the Nas host with 10gb
Option one is run a flat network with a basic l2 managed switch maybe an old juniper ex 3300 or something (not sure if a new mikrotik would be wiser)
Option 2 Buy a unifi pro l3 switch and keep the vlans
Option 3 keep the vlans with a basic l2 10gb switch and replace the udm se with a proper 10gb capable router having a single 10gb uplink between the switch and the router. (Maybe the same ex3300 plus one of those little r86s box with pfsense)
I'm not sure there are any other viable options. I dont really want to recable the walls so I'll probably use rj45 transcievers if I go option 3.
Any feedback or ideas?
My entire network is currently 1gb unifi.
My network looks like this:
Udm se - usw8, flex mini, main desktop - unifi ap
Usw8 (Poe)- 5 proxmox hosts (one running virtual truenas and in lacp), usw8 #2
Usw8 #2 (Poe) - 3 proxmox hosts, unifi ap
Flex mini - tv, Xbox, unifi ap
The proxmox hosts are vlan aware and run a number of nested hypervisors on different vlans as well as a few jump boxes and media streaming services.
I'm getting very poor performance to the Nas especially with 4k tests. Truenas is running 5 HDD SATA mirrors on a SAS card and a zil mirror.
Given I'm not routing at the switches, any inter vlan traffic to the Nas needs to hit the udm over a shared 1gb link and come back down the same link. Sounds like bottleneck central to me.
My walls are called with cat6a.
I ran a 4k fio test locally on the Nas box and then on another VM on the same subnet. Even though they're on the same subnet, I got about 30% of the iops and throughout on the VM.
I'm trying to decide the best way forward to go 10gb from the proxmox hosts to the Nas.
In each of these options I'd be connecting the Nas host with 10gb
Option one is run a flat network with a basic l2 managed switch maybe an old juniper ex 3300 or something (not sure if a new mikrotik would be wiser)
Option 2 Buy a unifi pro l3 switch and keep the vlans
Option 3 keep the vlans with a basic l2 10gb switch and replace the udm se with a proper 10gb capable router having a single 10gb uplink between the switch and the router. (Maybe the same ex3300 plus one of those little r86s box with pfsense)
I'm not sure there are any other viable options. I dont really want to recable the walls so I'll probably use rj45 transcievers if I go option 3.
Any feedback or ideas?