What am I doing?

[NerdAshes]

Here is my sad attempt at using LibreOffice Draw for the first time (I'm a Visio guy, but I'm trying to be better).

White = 1gps
Orange = 2.5gps
Blue = 10gps
Purple = 25gps
Red = 100gps

Node 1,2 & 3 do connect to the VM 10gps, Backup 10gps switches I just ran out of annoying arrows.
Node 2 does connect to the Private Ceph storage switch too .. I was to lazy to make more arrows for it.

The Ceph private switch doesn't need to be 100gbs, but PCIe NICs come with dual 100 ports and I can't find a 4 (3) port 25gps switch.

10 port 10gps switch (w/ 25gps uplink for backup NAS) <--- can't find that but I can find 10 or 12 port 10gps switches.
10 port 10gps switch (for VMs & home LAN) <---simple
4 port 2.5 switch (for modem router/FW) <---simple
8 port 2.5 POE (MGMT/WAP) <--- can't find that
4 port 2.5 switch (for Cronosync) <--- simple
4 port 100gps switch (for Ceph Public) <---- costly can't find a 4 (3) port slower switch for less
4 port 100gps switch (for Ceph Private) <---- ditto
6 port 1gps POE + 1 10gps switch (for RPi RKE2 Network) <--- the 6 (8) port POE and 10gps uplink seem to be hen's teeth

Any ideas to make this a less expensive or simpler endeavor?

What dual port 100gps NICs should I slap in the MS-01 (PCIe 4 x8) if I go with MikroTik CRS504-4XQ-IN switches for the Ceph storage networks? RoCEv2 is not on the MikroTik CRS504-4XQ-IN so.... ???

Thoughts? Prayers?

Thanks!

 

[mattventura]

Check if your modem actually supports multiple clients like that. There's a good chance it doesn't, in which case you'll have to set up a much more complex setup. You're already looking at having to set up VRRP, but you'll also need to set up some way of syncing their NAT tables and something to actually do the failover. Easier said than done.

Rather than an explicit active/backup switch, you can use MLAG. That way, you get the increased (under certain conditions) bandwidth of LAG, but also with a bit of redundancy.

What would you be doing with Ceph that would require not only a 100g public switch, but a 100g private switch as well? The nodes only have a 10g upstream connection, so anything happening within that cluster would be severely bottlenecked by whatever is going on upstream. Even if those MS-01s are also running VMs, the VMs generally wouldn't be generating tons of disk activity on their own. One switch for the public and private should be plenty. Same thing with the Cronosync switch - do you really need another switch for that, when those boxes are already connected?

Where are the VMs running in this diagram?

 

[XeonSam]

Mikrotik has a cheap 100G switch which is 4 ports. You really need to simplify this. Instead of 5+ switches just get a single Nexus switch like an N9K that supports 100G. Use breakout cables for the 25G/10G. Your lab will sound like a jet engine but this is just too complicated to enjoy

But honestly, split the networks into seperate vlans and look into Mikrotik. They've got some really affordable "home" friendly enterprise gear.

 

[NerdAshes]

Check if your modem actually supports multiple clients like that. There's a good chance it doesn't, in which case you'll have to set up a much more complex setup. You're already looking at having to set up VRRP, but you'll also need to set up some way of syncing their NAT tables and something to actually do the failover. Easier said than done.

Rather than an explicit active/backup switch, you can use MLAG. That way, you get the increased (under certain conditions) bandwidth of LAG, but also with a bit of redundancy.

What would you be doing with Ceph that would require not only a 100g public switch, but a 100g private switch as well? The nodes only have a 10g upstream connection, so anything happening within that cluster would be severely bottlenecked by whatever is going on upstream. Even if those MS-01s are also running VMs, the VMs generally wouldn't be generating tons of disk activity on their own. One switch for the public and private should be plenty. Same thing with the Cronosync switch - do you really need another switch for that, when those boxes are already connected?

Where are the VMs running in this diagram?

The modem doesn't support anything smart. The switch before the FW is because the modem only has one port. The MS-01 will run OPNSence HA, with MAC cloning, to fool the modem when needed. I haven't tried this before - it may not work. Another member here mentioned it's what they do and it works for them.

I am tempted to use MLAG for the redundancy alone. It's the cost of 2 bigger switches keeping me from going that route, it's at least twice as much. It also limits the available options... I'll spend more time today and see what else I can find (I've been looking for a week however).

You might have missed it in the original post. I pointed out that the 100gps switches are less expensive than slower (40gps+) switches. That's because they are only 4 port switches and the other switches I've found, that are slower, cost more because they all have 8+ ports. The Mikrotik switch I mentioned is only $650. The Ceph private network wants the speed, the public can be as slow as 10gps. The NIC however shows up with dual 100gps, so why not use it? An option I just thought of .. I may be able to save a switch and get a single port NIC instead of the dual. I'd need to setup a thunderblot-net open mesh network using the USB4 ports. I could use that for the public Ceph network... It's about 11gps and the latency is probably high ... but it could work!

I also mentioned I'm looking for 100gps dual port NICs for the MS-01. The other ports are already used. There shouldn't be a bottleneck and if there is, it's because the cheap Mikrotik switches don't have RoCEv2, so the CPU overhead might be a thing?

Cronosync best practice is to have it's own network, it's speed needs are minor however. 4 port 2.5gps switches are dirt cheap and there is an available port for it...

The VMs are running on the MS-01 "Node" 1-2-3 (The other MS-01 are baremetal OPNsence)

 

[Tech Junky]

thunderbolt

This gets you 20gbps but, it's a lot cheaper than all of the switches and NICs. TB4 cads are only ~$60 if you buy the returns off Amazon. Then again it depends on the MOBO headers as well. The Gigabyte cards work with anything except Asus as they decided to use a 19-pin cabling scheme. There's also some Gen4 ASM cards w/ ASM4242 that have been shipping with OEM MOBOs recently. I suspect they might get better speeds than traditional TB as their "data" is showing throughput of 3.8GB/s for drive use vs TB capping out at ~3.2GB/s. The network side seems a bit different though for now.

 

[NerdAshes]

Mikrotik has a cheap 100G switch which is 4 ports. You really need to simplify this. Instead of 5+ switches just get a single Nexus switch like an N9K that supports 100G. Use breakout cables for the 25G/10G. Your lab will sound like a jet engine but this is just too complicated to enjoy

But honestly, split the networks into seperate vlans and look into Mikrotik. They've got some really affordable "home" friendly enterprise gear.

I specifically mentioned that switch in my post

I "personally" would rather a complicated low power, low cost, quiet solution, than a screaming jet engine, high power, high cost Cisco switch. Studying for my CCNA still makes me wake up, in a cold sweat, 20 years later. I bet it's not any less complicated ;-p
Once it's setup - it's setup, no? I guess there are a ton of switches to update/backup.. I should be able to script that though... hmmm

 

[NerdAshes]

This gets you 20gbps but, it's a lot cheaper than all of the switches and NICs. TB4 cads are only ~$60 if you buy the returns off Amazon. Then again it depends on the MOBO headers as well. The Gigabyte cards work with anything except Asus as they decided to use a 19-pin cabling scheme. There's also some Gen4 ASM cards w/ ASM4242 that have been shipping with OEM MOBOs recently. I suspect they might get better speeds than traditional TB as their "data" is showing throughput of 3.8GB/s for drive use vs TB capping out at ~3.2GB/s. The network side seems a bit different though for now.

That 20gbps is in one direction and halved if it's send/receive - at least that's what I've read on other forums.. I've not tried it myself. The MS-01 mentioned, has two USB4 40gps (20 if both are used) ports that support Thunderbolt-Net 3.

I've toyed with the idea to plug in a direct attach storage device to each node.. but I can not think of a good use case for it yet? Maybe image storage, but I have enough space on the OS drive for that.

The U.2 SSD Ceph storage is going to want a lot more storage network speed and much lower latency than what the Thunderbolt-Net appears to offer. Of the few people who have tried it, that I've talked to, they were not pleased with the performance and plan to move to switched networking.

It would be a LOT less expensive however to just use the USB4 ports and SFP+ ports already in the MS-01. I just haven't seen any "happy" data doing it that way.

 

[alaricljs]

@NerdAshes the modem -> switch idea is probably not going to work. I was moving in that direction and had my modem plugged to 1 Proxmox box and bridged to my OPN VM. Regardless of MAC cloning or any other attempts at ensuring the modem was happy it did not work. It bound to the MAC of the NIC port and never let go. Cloning the NIC MAC into the VM did not work. I had to pass the NIC hardware to the VM and I can't fail over without extra steps requiring physical changes. Hoping my pending switch to fiber will help resolve those woes.

 

[Tech Junky]

USB4

When I do it with my laptop / server it links at 20 and moving data between drives / systems seems to indicate it's full duplex as the data hits 1.5-1.7GB/s across the cable.

space on the OS drive

I keep my OS drive for OS stuff and put everything else somewhere else but, I think that's more related to Windows PTSD long ago.

direct attach storage

It's a good option for a repo of data and if one system dies you just move the cable to another system and setup the share again.

It depends on how much $$$ you want to throw at all of these scenarios though. Of course the DAS option also introduces a bottleneck due to the cable being used. Though if you DIY something using cards / cables you can exceed off the shelf speeds. I simplified my network long ago by putting everything into a single box and getting rid of the consumer gear (6-7 devices). My bottleneck at this point could be wifi with a 2.5ge port. I'm however working on a 802.11BE option using a QCA M2 that should bump the speeds upwards with 320mhz vs current 160mhz. Still for me the TB P2P would be faster by 3-6X wifi or Ethernet 5GE.

----------------
The other option would be put the $$$$ into an Epyc setup and virtualize everything though it takes out the troubleshooting issues you run into w/ switches and cables. The upside though would be no bottlenecks as it would all be internal data.

In networking you can virtualize things to not spend a ton on HW for studying. Good for those pesky applets where you have to run commands to find the answers.

 

[mattventura]

Yeah, I would say, you can definitely converge some of the switches.

You might have missed it in the original post. I pointed out that the 100gps switches are less expensive than slower (40gps+) switches. That's because they are only 4 port switches and the other switches I've found, that are slower, cost more because they all have 8+ ports. The Mikrotik switch I mentioned is only $650. The Ceph private network wants the speed, the public can be as slow as 10gps. The NIC however shows up with dual 100gps, so why not use it? An option I just thought of .. I may be able to save a switch and get a single port NIC instead of the dual. I'd need to setup a thunderblot-net open mesh network using the USB4 ports. I could use that for the public Ceph network... It's about 11gps and the latency is probably high ... but it could work!

I also mentioned I'm looking for 100gps dual port NICs for the MS-01. The other ports are already used. There shouldn't be a bottleneck and if there is, it's because the cheap Mikrotik switches don't have RoCEv2, so the CPU overhead might be a thing?

Cronosync best practice is to have it's own network, it's speed needs are minor however. 4 port 2.5gps switches are dirt cheap and there is an available port for it...

The VMs are running on the MS-01 "Node" 1-2-3 (The other MS-01 are baremetal OPNsence)

The "why not use it" is that you'd need to buy a separate 100g switch. You can always set up a private network on the same switch using VLANs, and then break it out into a separate switch later if needed. Think about where the disk writes come from - baseline OS load from logging and such doesn't need 100gb. Load typically comes from somewhere, like if I do a file transfer to one of those VMs. In that case, you're bottlenecked by the 10gb upstream link anyway, so using a 100gb public network and a separate 100gb private network, you're way overkill already.

I'd say the same thing for Cronosync - keep it on the same network if needed, and then split it out later if you've determined that a single 100gb switch is actually getting bottlenecked. There's a good chance that the nodes would get bottlenecked on the CPU or something else anyway.

I'd also wonder - why not run RKE2 on the MS-01s? They absolutely dwarf the Pis.

 

[NerdAshes]

When I do it with my laptop / server it links at 20 and moving data between drives / systems seems to indicate it's full duplex as the data hits 1.5-1.7GB/s across the cable.

1.5GB/s = 12Gbps, seems about right.

I keep my OS drive for OS stuff and put everything else somewhere else but, I think that's more related to Windows PTSD long ago.

I feel the Windows PTSD.. I have 2TB M.2 drives for the OS, just for the higher TDW. VM OS images just sit there and if they are lost, I'd just redownload them. Not even sure I'll mess with it though. I'll probably just leave the OS and nothing else on the drive.

It's a good option for a repo of data and if one system dies you just move the cable to another system and setup the share again.

It depends on how much $$$ you want to throw at all of these scenarios though. Of course the DAS option also introduces a bottleneck due to the cable being used. Though if you DIY something using cards / cables you can exceed off the shelf speeds. I simplified my network long ago by putting everything into a single box and getting rid of the consumer gear (6-7 devices). My bottleneck at this point could be wifi with a 2.5ge port.

I just "want" a Ceph storage network. Honestly all of this is want, for giggles only, just to learn it. I'm retired, no one is going to pay me for my knowledge. I'll never use 20% of the capability. It's rather wasteful - but I'm having fun. The joy of watching the kids, watching Dad burn through their inheritance is enough for me.

I'm however working on a 802.11BE option using a QCA M2 that should bump the speeds upwards with 320mhz vs current 160mhz. Still for me the TB P2P would be faster by 3-6X wifi or Ethernet 5GE.

320mhz!? what is the range on that? 6 feet? Sounds fun, bet it's fast!

The other option would be put the $$$$ into an Epyc setup and virtualize everything though it takes out the troubleshooting issues you run into w/ switches and cables. The upside though would be no bottlenecks as it would all be internal data.

Maybe next winter? I'm not a fan of full depth, loud servers though, so I'll need a Epic-Mini Epyc system.

In networking you can virtualize things to not spend a ton on HW for studying. Good for those pesky applets where you have to run commands to find the answers.

I will still have plenty of virtual servers, switches, routers, etc. in a sandbox on the cluster. ;-)

 

[mattventura]

I'd argue that there's still an advantage to having 3 MS-01s instead of a single big Epyc, in the sense that 3 is a good number for failover purposes. Anything that requires a quorum (Ceph, k8s, etc) wants 3 nodes at a minimum.

 

[NerdAshes]

Yeah, I would say, you can definitely converge some of the switches.

I could. It's the cost of the switches that can handle the convergence that is the issue. I could make it simple and put every on two awesome switches or just one if I wanted to risk it. Just vLAN everything. From the documentation from Ceph and testing from the Proxmox team - it seems that best practice is to not use SDN, and separate all the traffic onto their own physical network. From what I've read, Ceph using SSDs (like the MZ-QL27T600 |PM9A3 NVMe U.2 drives I have) will easily saturate a 25gps network. It was suggested to use up to 400gps! I figure with my setup the old 40gps would be fine. However the 40gps switches and NICs are all old used enterprise equipment. It's not much more and often less to go with 100gps - at least that's what I've found. 100gps have the benefit of being "current" technology too.

The "why not use it" is that you'd need to buy a separate 100g switch.

The price of 2 of the separate 4 port switches is less than a single multi speed or 40gps switch with enough ports to handle the private & public networks.

From Mikrotik (cheapest, acceptable quality switches I've found so far) the option that would handle those two Ceph networks is $1500ish and 25gps. The two 100gps 4 port switches from them would be $1300. The two 100gps 4 port switches are also quite and use less power than the one switch. :-/

You can always set up a private network on the same switch using VLANs, and then break it out into a separate switch later if needed.

True, but I'd have to buy a bigger switch first, then buy another if it didn't work out. I'd still need a dual NIC for each node. Again there is not a 40+gps 6 or 8 port switch (that I have found) that is less expensive then two of the 100gps 4 port switches. I'd LOVE to find one though. If you know of one...

Think about where the disk writes come from - baseline OS load from logging and such doesn't need 100gb. Load typically comes from somewhere, like if I do a file transfer to one of those VMs. In that case, you're bottlenecked by the 10gb upstream link anyway, so using a 100gb public network and a separate 100gb private network, you're way overkill already.

The Ceph OSD runs on the Enterprise U.2 SSDs and the load comes from the Ceph replication across the cluster. That replication is currently bottlenecked by 25gps networks. 100gps is way overkill for the private and public network. It's still less expensive and super simple to setup compared to a single 40gps switch and vLANs/port isolation. Unless I'm not understanding something? Totally possible. I just haven't seen a simpler/less expensive solution yet. That's why I'm asking here I guess! Seems like there should be one?

I'd say the same thing for Cronosync - keep it on the same network if needed, and then split it out later if you've determined that a single 100gb switch is actually getting bottlenecked. There's a good chance that the nodes would get bottlenecked on the CPU or something else anyway.

I'd have to use breakout cables on one of the switches, setup vLAN, etc. For the cost of the cable I could give it it's own unmanaged switch. I just don't see the benefit of using a more complicated solution when a cheap alternative exists.

I'd also wonder - why not run RKE2 on the MS-01s? They absolutely dwarf the Pis.

Very true. I'll probably use the RPi 4s for something else. I was planning on using the RPi 5's as a HA RKE2 Rancher/LB for workers on the MS-01 cluster once it's setup. Not that I need too... just cause. Supposedly it's best practice to have your Rancher cluster separate from your other Kubernetes clusters, so why not - I already have them.

I don't want to come across as stubborn or ungrateful for the help offered. I do appreciate the advice. I just want to understand it better.
Thank you for taking the time to reply

 

[Tech Junky]

stuborn

We all fall into that from time to time as we gather bits and pieces of info from others experiences and focus on one thing.

Since you're "having fun" it might be an option to explore building a "switch" using a PC w/ your higher end NIC needs and using DAC cables to connect your bandwidth. When I built my AIO box initially I had a couple of quad 1GE cards in it to downsize and aggregate things as planned. Worked fine w/o any regression that was noticeable vs using ASICs in a switch.

The problem with going 100/400GE though is the lack of port density due to the enormous port and heat sink.

 

[mattventura]

I could. It's the cost of the switches that can handle the convergence that is the issue. I could make it simple and put every on two awesome switches or just one if I wanted to risk it. Just vLAN everything. From the documentation from Ceph and testing from the Proxmox team - it seems that best practice is to not use SDN, and separate all the traffic onto their own physical network. From what I've read, Ceph using SSDs (like the MZ-QL27T600 |PM9A3 NVMe U.2 drives I have) will easily saturate a 25gps network. It was suggested to use up to 400gps! I figure with my setup the old 40gps would be fine. However the 40gps switches and NICs are all old used enterprise equipment. It's not much more and often less to go with 100gps - at least that's what I've found. 100gps have the benefit of being "current" technology too.

The Ceph OSD runs on the Enterprise U.2 SSDs and the load comes from the Ceph replication across the cluster. That replication is currently bottlenecked by 25gps networks. 100gps is way overkill for the private and public network. It's still less expensive and super simple to setup compared to a single 40gps switch and vLANs/port isolation. Unless I'm not understanding something? Totally possible. I just haven't seen a simpler/less expensive solution yet. That's why I'm asking here I guess! Seems like there should be one?

I think you're overestimating your actual load. Yes, a U.2 SSD can put out some excellent performance. One gen4 drive alone can put out a theoretical 64gb/s. That doesn't mean you need to size your cluster network to be able to handle that full speed. You have to ask yourself: what task would you be doing on a VM that would consume that much write bandwidth, and is it really that performance-sensitive? You don't want to look at just how much bandwidth a have to think about what is actually consuming that bandwidth. All of those best-practices recommendations are set up for mission-critical enterprise-grade workloads, not a homelab.

For example, if I set up a server with 4 Gen5 drives, I'd theoretically need a whopping 500gbps for the network to not be the bottleneck. But in practice, I don't have anything that would actually saturate anything close to that.

In other words - size everything according to your actual needs, not enterprise best practices. Those recommendations are usually made under the assumption that you might actually saturate your current infrastructure.

I'd have to use breakout cables on one of the switches, setup vLAN, etc. For the cost of the cable I could give it it's own unmanaged switch. I just don't see the benifit of using a more complicated solution when a cheap alternative exists.

But aren't those nodes already connected to an existing switch? Why does it need to be separated at all? I suspect this is another case of "a web page somewhere told me it should be on a separate network", but when your network (100gpbs) is likely already massively oversized for the actual load, that's a premature optimization.

Very true. I'll probably use the RPi 4s for something else. I was planning on using the RPi 5's as a HA RKE2 Rancher/LB for workers on the MS-01 cluster once it's setup. Not that I need too... just cause. Supposedly it's best practice to have your Rancher cluster separate from your other Kubernetes clusters, so why not - I already have them.

I would ask - why two different clusters? In true enterprise workloads, separating clusters is most often done for security reasons (i.e. having your internal cluster and your public-facing/DMZ cluster).

I don't want to come across as stuborn or ungrateful for the help offered. I do appreciate the advice. I just want to understand it better.
Thank you for taking the time to reply

No worries, I've definitely been in that position myself.

 

[NerdAshes]

We all fall into that from time to time as we gather bits and pieces of info from others experiences and focus on one thing.

Since you're "having fun" it might be an option to explore building a "switch" using a PC w/ your higher end NIC needs and using DAC cables to connect your bandwidth. When I built my AIO box initially I had a couple of quad 1GE cards in it to downsize and aggregate things as planned. Worked fine w/o any regression that was noticeable vs using ASICs in a switch.

The problem with going 100/400GE though is the lack of port density due to the enormous port and heat sink.

It's also difficult to acquire knowledge in niche topics that normally don't apply to home users, using consumer level equipment, to mimic enterprise grade services. The information comes from teams using up to millions of dollars of enterprise equipment, to create "best practices" that don't translate well into the home world or finances. That, or the information comes from "someone" on a forum that's kind of doing what you are, but not quite the same way. It's hard to know what the correct path is until you try it and fail or maybe even succeed. I'm currently stuck in analysis paralysis, because I want to succeed on the first try and thousands of dollars are on the line. I need to be able to fail and recover, but I'm having a hard time seeing the recovery path if I get it wrong. So here I am - second guessing


Building a switch - honestly that does sound kinda fun! Think I'm going to finish this cluster first though.. maybe take the summer off from technology too.

 

[Tech Junky]

analysis paralysis

It happens. I've rebuilt my AIO setup for the network more times than I can recall at this point since 2015 when I did the first iteration. My motivation is only significant underlying tech advancements though. The original had Gen3 slots and I didn't budge until Gen5 came around. Then didn't bother with jumping to DDR5 until it didn't make sense not to aka switching to AMD.

I look at things a bit differently though when it comes to tech in the event of needing to "upgrade" it's on a rolling cycle to decrease the depreciation of the parts when you go to sell them. Planning and executing are one thing but evolving and pivoting are going to be key.

Finding a good foundation is the best place to start. The must haves are what will make things come together as you dive deeper into the abyss. You could drop $10K on this adventure or build up to it in increments. Coming from a network background and not so much of a concern on the IS side of things it's very easy to spend big on gear just to be able to talk to each other. If you peer things locally since they're all relatively close to each other you can find ways that make it speedy and cheaper like P2P connections with DAC cables instead of messing with $1500 switches. My AIO I end up putting in a single quad 5GE card for $200 instead of buying a switch/router for $500+. Why 5GE? because my laptop would max at 10ge with a dongle for $150 or 5ge for $70 and my drives at the time maxed out at ~400MB/s in raid. I switched to U.3 / NVME though and wanted to bump the speed so I reconsidered 10GE again but then thought deeper about thunderbolt and tested it at 20gbps and thus saved some money on using that instead.

It's easy to go down the rabbit hole or have an ADD moment when it comes to tech.

 

[NerdAshes]

I think you're overestimating your actual load. Yes, a U.2 SSD can put out some excellent performance. One gen4 drive alone can put out a theoretical 64gb/s. That doesn't mean you need to size your cluster network to be able to handle that full speed. You have to ask yourself: what task would you be doing on a VM that would consume that much write bandwidth, and is it really that performance-sensitive? You don't want to look at just how much bandwidth a have to think about what is actually consuming that bandwidth. All of those best-practices recommendations are set up for mission-critical enterprise-grade workloads, not a homelab.

Ohhh - I think I get now... derp!
You're totally correct. I won't have anything causing a storage network a massive "right now!" sync need, at least where I would ever really notice it in the LAN or services the cluster is providing... probably even at 10gps. I bought drives for the power backup and durability. I got stuck trying to optimize for speed I don't need. Foolish & expensive.

For example, if I set up a server with 4 Gen5 drives, I'd theoretically need a whopping 500gbps for the network to not be the bottleneck. But in practice, I don't have anything that would actually saturate anything close to that.
In other words - size everything according to your actual needs, not enterprise best practices. Those recommendations are usually made under the assumption that you might actually saturate your current infrastructure.

I think my issue stems from my old career selling businesses their infrastructure. I always followed best practices and had to provide SLA w/uptime using many 9s. It's hard to turn that part of my brain off at home :-/

But aren't those nodes already connected to an existing switch? Why does it need to be separated at all? I suspect this is another case of "a web page somewhere told me it should be on a separate network", but when your network (100gpbs) is likely already massively oversized for the actual load, that's a premature optimization.

To be fair that webpage is Promox's HA HCI install documentation page.. but you are correct.

I would ask - why two different clusters? In true enterprise workloads, separating clusters is most often done for security reasons (i.e. having your internal cluster and your public-facing/DMZ cluster).

Yes, it's just to setup a simulation of what a true enterprise would do. Like playing "house" as a kid I guess. Best part is.. it's all internal and nothing is pubic facing. Super silly, wildly unnecessary. I bought the Rpi's over the winter as something to do and now that it's up and running - I can't bring myself to bring it down. Still don't know what I'll do with it. lol absurd.

No worries, I've definitely been in that position myself.

Thanks for understanding

 

[NerdAshes]

Not sure my Libreoffice draw is getting better...

So if I gobble together everyone's suggestions - is this "the" bees knees of a plan?

 

[mattventura]

That is a lot simpler. I'd still question whether you actually need the MS-01s hooked up to the 2.5gb switch, rather than using that just for the APs and such.