Hi all,
I host a couple of servers as a side business and it has become apparent I need to stop kicking the can down the road and really get the networking done right. I'm looking for recommendations on hardware for OPNsense, switch recommendations, and general network design for redundancy.
tl;dr: I've attached a diagram of my current failing configuration and what I'm considering for an upgraded config. My goal is to have as much redundancy as possible without spending too much money. Budget might be $2000 for all routers and switches, and even that feels high because I'm a terrible cheapskate. Does anyone have any recommendations for cost-effective and reliable hardware for the OPNsense routers?
OPNsense Router Wish list:
I'd also appreciate recommendations on the switches. Wish list:
#end tl;dr
Expanding on the above, currently I just have all of the servers connected to a Cisco C3560E switch with the IPMI connections on a separate VLAN. The switch then connects to the OPNsense firewall/router which then connects to the ISP. Lately the OPNsense router has been acting up and freezing intermittently, and a few weeks ago the Cisco switch failed leaving me in a bind. Needless to say there is room for improvement.
I'm willing to consider almost any solution at this point. I've penciled in using OPNsense in a redundant configuration for the router/firewall. I'm open to alternatives but I'm not sure there is anything better in my price range. I'm also fairly competent with OPNsense/pfSense, whereas I'd have to relearn anything else.
It sounds like the more "enterprise" way to design the network would be to have a separate router and firewall, and then move all of the other services like DHCP, DNS, and Wireguard VPN to some other system. I'm not sure I'd see much benefit to this though and it'd be more expensive. Here's a theoretical question: When is an all-in-one solution like OPNsense not enough? When you need to push over 10Gbps through it? When you have more than X number of devices?
Router hardware options I've come up with:
Currently the ISP handoff is 1000BASE-T but could be upgraded to 2.5GBASE-T or 10GBASE-T in the near future so I'd like to be able to support all of those. This wouldn't be too hard by just using an 10GBASE-T NIC on the OPNsense server. Since there is only a single connection from the ISP, I think I'm also going to need a switch to split the connection to both routers. I haven't put much thought in to that, other than it'd be nice if it was unmanaged to reduce attack surface, NBASE-T/10GBASE-T to support the needed connections, and had redundant PSUs because I'm paranoid. I'm not sure such as thing exists (or is affordable) though. This could be anything from the cheapest consumer 10GBASE-T switch I can find (with no redundant PSU), or even a 2.5GBASE-T switch would be enough (I could even hack in redundant PSUs), or I could use an old Cisco switch that has a few 10G ports on it to get redundant PSUs, but with hardware that old, would it be less reliable that the consumer switch?
As for the switches going to the servers, this is where it gets messy, particularly with the redundancy requirement.
Options I've thought of:
To wrap all this up, I'm currently thinking it might make the most sense to go with OPNsense on the SuperMicro X9SRH-7F hardware option and accept the higher power usage, then go with the ICX6610 for the switches since they aren't that much more expensive than whatever old Cisco thing I could buy is.
I'm really curious to hear any feedback! Thank you! (Especially if you read this entire thing!)
I host a couple of servers as a side business and it has become apparent I need to stop kicking the can down the road and really get the networking done right. I'm looking for recommendations on hardware for OPNsense, switch recommendations, and general network design for redundancy.
tl;dr: I've attached a diagram of my current failing configuration and what I'm considering for an upgraded config. My goal is to have as much redundancy as possible without spending too much money. Budget might be $2000 for all routers and switches, and even that feels high because I'm a terrible cheapskate. Does anyone have any recommendations for cost-effective and reliable hardware for the OPNsense routers?
OPNsense Router Wish list:
- ECC RAM
- Redundant PSUs
- Future expandability for up to 10G uplink (I'm willing to accept that OPNsense itself may be a bottleneck at 5Gbps+ routing)
- Low power consumption.
I'd also appreciate recommendations on the switches. Wish list:
- Ability to work with existing 1G NICs all the way up to future 40G+ NICs
- Redundant PSUs
- MC-LAG capable
- Ability to update firmware on one switch without taking down network (this might rule out stacking)
#end tl;dr
Expanding on the above, currently I just have all of the servers connected to a Cisco C3560E switch with the IPMI connections on a separate VLAN. The switch then connects to the OPNsense firewall/router which then connects to the ISP. Lately the OPNsense router has been acting up and freezing intermittently, and a few weeks ago the Cisco switch failed leaving me in a bind. Needless to say there is room for improvement.
I'm willing to consider almost any solution at this point. I've penciled in using OPNsense in a redundant configuration for the router/firewall. I'm open to alternatives but I'm not sure there is anything better in my price range. I'm also fairly competent with OPNsense/pfSense, whereas I'd have to relearn anything else.
It sounds like the more "enterprise" way to design the network would be to have a separate router and firewall, and then move all of the other services like DHCP, DNS, and Wireguard VPN to some other system. I'm not sure I'd see much benefit to this though and it'd be more expensive. Here's a theoretical question: When is an all-in-one solution like OPNsense not enough? When you need to push over 10Gbps through it? When you have more than X number of devices?
Router hardware options I've come up with:
- Asrock X470D4U motherboard with SC815TQ-R700UBchassis with redundant PSUs
- Ryzen 3 2200G PRO CPU (or similar), DDR4 ECC UDIMM
- Would have to be a custom build, probably wouldn't be very clean, reliability of motherboard is questionable?
- UIO chassis may be a problem
- SuperMicro SYS-5019C-FL- ~$500 barebones
- i3-9100F (or similar), DDR4 ECC UDIMM
- Cheap but doesn't have redundant PSUs
- SuperMicro SYS-5019S-MR- ~$1000 barebones
- Xeon E3-1270 V6 (or similar), DDR4 ECC UDIMM
- Probably too expensive, somewhat dated platform?
- SuperMicro AS-1013S-MTR
- EPYC Naples/Rome (Whatever model I can get cheapest), DDR4 ECC RDIMM
- Expensive, Probably has high idle power consumption
- SuperMicro X9SRH-7F Motherboard with SC815TQ-R700UBchassis (or similar) - $250 barebones
- Xeon E5-2650 v2, DDR3 ECC RDIMM
- Custom build that might not be straightforward.
- I already have the CPUs and RAM so it would be very cheap.
- Probably has high idle power consumption.
Currently the ISP handoff is 1000BASE-T but could be upgraded to 2.5GBASE-T or 10GBASE-T in the near future so I'd like to be able to support all of those. This wouldn't be too hard by just using an 10GBASE-T NIC on the OPNsense server. Since there is only a single connection from the ISP, I think I'm also going to need a switch to split the connection to both routers. I haven't put much thought in to that, other than it'd be nice if it was unmanaged to reduce attack surface, NBASE-T/10GBASE-T to support the needed connections, and had redundant PSUs because I'm paranoid. I'm not sure such as thing exists (or is affordable) though. This could be anything from the cheapest consumer 10GBASE-T switch I can find (with no redundant PSU), or even a 2.5GBASE-T switch would be enough (I could even hack in redundant PSUs), or I could use an old Cisco switch that has a few 10G ports on it to get redundant PSUs, but with hardware that old, would it be less reliable that the consumer switch?
As for the switches going to the servers, this is where it gets messy, particularly with the redundancy requirement.
Options I've thought of:
- 2x old Cisco 1GbE switches - $100-150 each
- Then I could just upgrade to better switches when I actually need them.
- Doesn't support MC-LAG so I'd have to use Linux bonding with load balancing
- Linux bonding with load balancing shouldn't require any switch support. Might not failover as fast as MC-LAG?
- 2x Brocade ICX6610 - $250-300
- Good mix of 1GbE for existing servers and IPMI connections with 16x 10GbE for future expansion
- Doesn't support MC-LAG so I'd have to use Linux bonding or something more exotic? (Wasn't planning on stacking the switches so they could be updated without taking everything down)
- 2x Mellanox SX6036 - $250-300
- Lots of connectivity for 40GbE
- Could be some hidden drawback to using Ethernet-over-Infiniband for everything?
- Would cost a fortune to try to adapt any 1GbE connections
- Not sure if this support MC-LAG
- 2x Brocade ICX6650 - $550-600
- Does support MC-LAG
- Tons of connectivity
- No cost effective way to connect 1GbE devices
To wrap all this up, I'm currently thinking it might make the most sense to go with OPNsense on the SuperMicro X9SRH-7F hardware option and accept the higher power usage, then go with the ICX6610 for the switches since they aren't that much more expensive than whatever old Cisco thing I could buy is.
I'm really curious to hear any feedback! Thank you! (Especially if you read this entire thing!)