Where to Start with Switching Home Network from Consumer Grade to Ent Grade?

[netager]

Hello, everybody
First post here...

We moved to our first home last year and now it's time to think about networking while doing light remodel.
Currently we run everything of Asus RTN-66 with Tomato. Everything is wireless, except for nVidia Shield.

I started reading and shopping few months ago... And I know it's going to be a steep learning curve...

So far, I got this:
Protectli box with J3160/8Gb/32GB with 4 Intel 210 ports.
Brocade ICX6450 nonPoE with Layer 2 and 10G license (Thank you STH )
ASUS RS300-E9-RS4 Server e3-1245v5/16GB ECC/128GB NVMe/256 SSD/12TB WD/10Gb Mellanox Dual port + 4 Intel 210

My internet connection is only 250/50, so 1GBe is enough for Protectli box with Sophos XG Home, but I want to run 10Gb between server and couple workstations, all LAN routing through Brocade switch. Server will run Proxmox with OpenMediaVault and some other stuff.

Now the question... Where to start? How to make transition? What would you do? Setup switch first? Setup firewall first?
Network diagram still in progress, just trying to start somewhere...

 

[Jason Antes]

I don't think it matters whether you start at the firewall or the switch first as long as you have the connectivity at either side to do the other. If your new firewall was all 10GbE and your current switch did not have connectivity for it, then you'd start with the switch and work your way around. For myself I started with the switch and then did the servers/PC's/AP's (I was going to a PoE+ switch and getting rid of injectors). This allowed me to also merge 2 switches into the icx6610 and set up VLAN's on the switch to segregate ports out by physical firewall interface. My next task is to do the firewall to 10GbE internally. Not sure if I'll wind up doing VLAN at the firewall or not, just depends on if I segment off another network for the wireless side interface or not.

 

[netager]

Thank you. I think I'll start with figuring out the switch since I want to do all the routing for LAN on it. Definitely will need some VLANs, but still planning.

 

[RobstarUSA]

I went the Mikrotik + Mellanox route. Been working rock solid for me. YMMV of course.

 

[Lost-Benji]

For home or business use, a good firewall/UTM is always a good way to start but where things go "HOME" is you don't need expensive, noisy and power hungry gear for it. Asus are just crap at anything to do with networking, plain and simple. Mikrotik is fine for switching and routing (term used very lightly) and if you are half decent with it, good way to go. Sadly, it is way over optioned and complicated with too many bells & whistles and in many cases, simply does not have the hardware to process all of what you are given as options. For home/business, the UniFi gear will do just fine but again, you need to do your homework first. UBNT are known for being hard to get help out of and their forums are full of fustrations and questions without answers.

P.S: Home does not need 10GbE FFS.

 

[SRussell]

Congratulations on the Brocade. It is a solid switch.

I started my journey at the firewall. I went with pfSense. For me, it made more sense to start at the firewall because everything else connected to it: DNS, DHCP...

I quit recommending Unifi equipment. The heaping pile of trash that was the 16 port POE. The lack of integration between Unifi and Edge gear was/is frustrating. I hated the stupid Cloud Key configuration; so many other people continue to have hardware and software issues. The thing that bugged me the most is the damn dashboard in Unifi. If you are not running a USG it gives the nag screen about your network not being 100% and you lose access to a good bit of metrics. There is no reason the Cloud Key could not function as syslog and pull stats from your switches and access point.

If the power consumption and noise does not bother you I would stick with used Enterprise gear.

 

[tsteine]

My case is probably in the realm of insanity, but my upgrade path went something like this:

I started out with getting an Ubiquiti Edgerouter 8 Pro and a Ubiquiti edgeswitch lite 48 port.
Then I got an Edgeswitch 16 XG for 10gbit for rack/servers.

So far so good, now comes the madness.

I got an Arista DCS-7050Q-16 and upgraded to 40gbit with mellanox connectx3 adapters.

Then I got a TNSR license, installed an intel XL710-QDA2 into a TNSR Box for 40gbit local vlan routing (which it's awesome at)
And I got myself a Mellanox SN2700 32Port 100gbit switch for my servers, and PCIe4.0 16X dual port Connectx-5 adapters.

So currently, 1000/1000 fiber internet attaches to TNSR router, into the Arista 40gbit switch, which supplies my local network with 10/40gbit and is connected to an edgeswitch 24 250W poe switch for 1gbe devices, and has a trunk port to the Mellanox switch for all my servers in the homelab rack.

So I started off reasonably, then went completely off into the deep end.

@Lost-Benji
I agree home doesn't *need* 10gbit, but damn if it's not fun with 10gbit or greater to play around with.

 

[Rand__]

And I got myself a Mellanox SN2700 32Port 100gbit switch for my servers, and PCIe4.0 16X dual port Connectx-5 adapters.

Nice - whats the power draw on it for how many used ports? Still toying with the idea even if totally overkill, but too much power draw does kill it [which is why I dont run the Celestica 100Gs I have - well that and missing x16 slots]

 

[tsteine]

@Rand__
Surprisingly low, it pulls about 90~ watts (230volts x 0.4 amps) with 10 ports populated, 6 of which are 100gbe, rest run at 40gbe

 

[Rand__]

hm 20%+ less than the celestica's id guess, but still plenty - especially if I am not using the extra speed vs the 6036. You getting close to utilizing 100G?

 

[tsteine]

Edit: to not mislead readers, this is only hitting the ram cache and zil, and is intended to show network utilization, but the actual throughput to the underlying drives is much lower (about 3GB/sec sequentially at absolute peak).

Actually, yes.
I'm running a storage box with 256gb ram, 10core cpu with 24x 2tb drives in a 12x two way mirror with ubuntu and ZFS.

Over iscsi with iser/rdma enabled, i got this result on one of my windows VMs:

20200327_131411.jpg

Nextcloud - a safe home for all your data

nextcloud.tsteine.com

 

[Rand__]

Very nice - wish FreeNas would do RDMA... also that ESXi would do NFS over RDMA, dont like iSCSI

 

[tsteine]

@Rand__

I have entertained the thought of switching from vmware to KVM and trying NFS over RDMA for storage, I'll make sure to let you know how that works if I pull the trigger on that.

I would also love RDMA on FreeNAS, the iscsi target on freebsd just plays nicer with ZFS and iSCSI vs linux iscsi targets.

SCST is the only target that would give me that level of performance with ZFS, the rest were slow as molasses in comparison.