Homesytems - MS-01 - 7 Questions

[NerdAshes]

I'm looking for advice. I need to finish my home systems redo. It's going to be based off of the new Minisforum MS-01 mini PCs.

I'm going to setup 2 of the MS-01 12i5 (still waiting on both) as a HA OPNsense firewall. I have a Motorola modem that has a single 2.5gbs Ethernet LAN connection.

Q1: The modem connects to what? Should I connect the Modem to a 2.5gbs switch and use port mirroring to the two firewalls?

Q2: Should I go with Mikrotik behind the FW, since they have cheap but fast switches?

The other three MS-01 (I have already) all are the same 13i9 based. All three have 96GB RAM, a Samsung PM9A3 7.6TB NVMe U.2, a Corsair MP600 PRO NH 2TB PCIe in the Gen3 x4 slot. I'm not sure what to put in the open PCIe slot yet...
I'm planning to setup HCI/HA with the three PCs as the cluster nodes.

Q3: Proxmox/Incus/Harvester/XCP-ng all seem to have good HCI/HA options and all cost the same. Anything stand out as a best option or a flat out avoid?

Q4: Ceph/Longhorn all look to play nice with storage on the node and should perform well with the specs available to them. Anything stand out to you as preferred, or something to avoid given the hardware I'm using?

Q5: The storage network looks like it should be a dedicated 25gbs+ switch with nothing else on it. I have fast drives but only 3, would 25gps be enough. 100gps looks rather expensive but doable, if Mikrotik is an acceptable option.

Q6: Ceph needs a fast background sync network and public network - should they both be the same speed?

Q7: What PCIe NIC should I put into the PCs for the storage network? Single or Dual (for private/public)? The MS-01 has PCIe 4 x8 for the NIC.





Thanks for any help you may offer! Cheers!

 

[Kerbys]

Is this for personal? isnt that a little Over powered for a firewall (I cant comment as looking at them for myself, but would be more tempted to run a hypervisor under it to get a few more services on it)

 

[FingerBlaster]

1: I have my WAN connection coming into my switch on a dedicated VLAN that only my router/firewall can access. My router is set up so that only the active one talks to the wan device and the routers clone their mac address when talking to the wan device. Otherwise my instance I would have to reboot my WAN device if I try to access it via a different MAC address.

3: Xcp-ng doesn't play nice with Intel consumer cpu architecture, specifically the big/little CPU design. I am using proxmox. No experience with the other 2 platforms.

4: I'm currently experimenting with ceph, my choice is because proxmox natively supports it out of the box.

5: since I have 3 nodes, each with 2 thunderbolt ports, I'm doing a point to point storage network with thunderbolt, each port maxes out at about 25gbps half duplex. In otherwords 25gbps in 1 direction, or 12.5 gbps bidirectional if there were simultaneous transfers back and forth. But I can also get 25 gbps hd on both ports at the same time. In testing it's good enough for me right now. I may feel differently once I move to production. It depends on your needs and what kind of vms you will be running, most of mine have low disk access needs.


i can't answer 2, 6, or 7.

 

[NerdAshes]

Is this for personal? isnt that a little Over powered for a firewall (I cant comment as looking at them for myself, but would be more tempted to run a hypervisor under it to get a few more services on it)

Yes just a personal house. At $450 shipped they were some of the cheapest "firewalls" I could find and the built in connectivity sealed the deal for me. I do expect them to be far overpowered even if I have an absurd amount of services running.
I'm a bit old school and I like my networking devices to be bare-metal. I want the fewest attack surfaces as possible, especially at the firewall.

 

[NerdAshes]

1: I have my WAN connection coming into my switch on a dedicated VLAN that only my router/firewall can access. My router is set up so that only the active one talks to the wan device and the routers clone their mac address when talking to the wan device. Otherwise my instance I would have to reboot my WAN device if I try to access it via a different MAC address.

3: Xcp-ng doesn't play nice with Intel consumer cpu architecture, specifically the big/little CPU design. I am using proxmox. No experience with the other 2 platforms.

4: I'm currently experimenting with ceph, my choice is because proxmox natively supports it out of the box.

5: since I have 3 nodes, each with 2 thunderbolt ports, I'm doing a point to point storage network with thunderbolt, each port maxes out at about 25gbps half duplex. In otherwords 25gbps in 1 direction, or 12.5 gbps bidirectional if there were simultaneous transfers back and forth. But I can also get 25 gbps hd on both ports at the same time. In testing it's good enough for me right now. I may feel differently once I move to production. It depends on your needs and what kind of vms you will be running, most of mine have low disk access needs.


i can't answer 2, 6, or 7.

The cloned MAC address is a great idea! Thank you for that. I think I'll do a cheap 4 port switch just for these devices. I'm concerned about 0day/LAN hopping.

That is good to know about XCP-NG! I am leaning the Pmox route (have used it before) but I am very tempted by Harvester (I like Rancher/Longhorn on my RKE2 clusters).

I've used Ceph briefly and I really liked the options. My only concern stems from the fire and brimstone sent over forums, about how you need 100 NVME drives, 1TB of RAM per MB of storage, and 1000Tbps redundant switches per NIC or you're going to wait a week to replicate data.

I do think a Mesh network would be pretty cool and it'd be nice to skip the $600+ switch for Ceph to gossip over. I am tempted to get cheap Mellanox 40/56gbps NICs and try it. Hmmmm

Thanks a ton for help!

 

[FingerBlaster]

My only concern stems from the fire and brimstone sent over forums, about how you need 100 NVME drives, 1TB of RAM per MB of storage, and 1000Tbps redundant switches per NIC or you're going to wait a week to replicate data.

Yeah, there's ideal, and there's good enough. In a home lab good enough is often fine. My buddy is running ceph on 10gig dac, and it's good enough for a dozen vms, ms ad, ma exchange, next cloud, nvr, home assistant, pfsense, etc. he has no performance issues

 

[SlowmoDK]

I'm going to setup 2 of the MS-01 12i5 (still waiting on both) as a HA OPNsense firewall. I have a Motorola modem that has a single 2.5gbs Ethernet LAN connection.

Why the HA firewall setup, if you still have a single point of failure in your Motorola modem ?

run OPNSense on one and proxmox on the other

 

[NerdAshes]

Why the HA firewall setup, if you still have a single point of failure in your Motorola modem ?

run OPNSense on one and proxmox on the other

I have a spare modem too (not just sitting in box, but setup, configured and activated)! I only have one ISP and they will only let me have one connection. Not sure why, if I'm willing to pay for two.. I'm digressing. If there is a way to use a coax splitter and feed two modems - I'd be down to try it!

I'm 90% setting up the HA firewall for giggles (never done it before) and 10% setting up the HA firewall for the Minisforum quality unknown. I'd actually be shocked if the Motorola modem died. They are Nokia of modems. I could probably dump coffee on it, leave it outside for the dog to play with and it'd still stay connected.

 

[FingerBlaster]

Why the HA firewall setup, if you still have a single point of failure in your Motorola modem

I have the ha setup (with vm's mind you), so that I can patch one and failover to the other.

 

[NerdAshes]

I have the ha setup (with vm's mind you), so that I can patch one and failover to the other.

I didn't even think of that... that's going to be nice.

 

[FingerBlaster]

Off topic, but I switched to vyos from pfsense/opnsense because it got much better performance virtualized on my r730xd. Opsense/pfsense could not max out my connection 2gbps fiber, and I'm waiting for 5gbps on my area. Vyos is could masquerade and firewall on my old gear up to like 15gbps, while opsense capped out at like 1.5gbps. I do miss the simplicity of a gui. Gonna test vyos vs opsense again on my ms01. Of course CPU/power utilization is still relevant even if it performs well enough.

 

[NerdAshes]

Off topic, but I switched to vyos from pfsense/opnsense because it got much better performance virtualized on my r730xd. Opsense/pfsense could not max out my connection 2gbps fiber, and I'm waiting for 5gbps on my area. Vyos is could masquerade and firewall on my old gear up to like 15gbps, while opsense capped out at like 1.5gbps. I do miss the simplicity of a gui. Gonna test vyos vs opsense again on my ms01. Of course CPU/power utilization is still relevant even if it performs well enough.

Interesting.. I took a very fast look. I'll have to look some more at Vyos when I get back from lunch. Does it have all the same abilities (IDS/IPS/DNS blackhole, etc)?
Looks like it's still FOSS so that's kewl.

 

[FingerBlaster]

Interesting.. I took a very fast look. I'll have to look some more at Vyos when I get back from lunch. Does it have all the same abilities (IDS/IPS/DNS blackhole, etc)?
Looks like it's still FOSS so that's kewl.

Not really, there's no crowdsec last time I checked, i think they do support suricatta, but you can also just run that stuff on separate vms. For dns I run adguard on a LXC container, I like the interface better.

vyos is a router first and foremost, and a firewall second. Pfsense/opsense are really firewall first and routing second.

 

[spuwho]

I have a spare modem too (not just sitting in box, but setup, configured and activated)! I only have one ISP and they will only let me have one connection. Not sure why, if I'm willing to pay for two.. I'm digressing. If there is a way to use a coax splitter and feed two modems - I'd be down to try it!

I'm 90% setting up the HA firewall for giggles (never done it before) and 10% setting up the HA firewall for the Minisforum quality unknown. I'd actually be shocked if the Motorola modem died. They are Nokia of modems. I could probably dump coffee on it, leave it outside for the dog to play with and it'd still stay connected.

While there are several still out there, the cable companies have been discouraging multiple modem setups at the same service address.

Seems that with the popularity of Taylor Swift over the years, more and more ticket scalpers had been trying to overcome the brokers unique IP rule.

Only so many tickets per IP address.

Since many scalpers are not technologists, they would order 10-15 modems on their single cable line because each modem had its own public facing IP address. Then they would setup a laptop to each modem to prime their ticket purchase across all those IP addresses.

Many providers would provide 5 public facing IP addresses per service port, but they didn't know that.

Some scalpers use multiple cloud instances to circumvent the IP limitation.

Back a few years ago, some scalps would get onto phone switches and override the system signaling to reach the call centers.

It got pretty ridiculous to what people would put themselves through to defeat Ticketmaster (or whomever was the vendor) just so they could sell a few tickets.

 

[javimaruso]

I'm looking for advice. I need to finish my home systems redo. It's going to be based off of the new Minisforum MS-01 mini PCs.

I'm going to setup 2 of the MS-01 12i5 (still waiting on both) as a HA OPNsense firewall. I have a Motorola modem that has a single 2.5gbs Ethernet LAN connection.

Q1: The modem connects to what? Should I connect the Modem to a 2.5gbs switch and use port mirroring to the two firewalls?

Q2: Should I go with Mikrotik behind the FW, since they have cheap but fast switches?

The other three MS-01 (I have already) all are the same 13i9 based. All three have 96GB RAM, a Samsung PM9A3 7.6TB NVMe U.2, a Corsair MP600 PRO NH 2TB PCIe in the Gen3 x4 slot. I'm not sure what to put in the open PCIe slot yet...
I'm planning to setup HCI/HA with the three PCs as the cluster nodes.

Q3: Proxmox/Incus/Harvester/XCP-ng all seem to have good HCI/HA options and all cost the same. Anything stand out as a best option or a flat out avoid?

Q4: Ceph/Longhorn all look to play nice with storage on the node and should perform well with the specs available to them. Anything stand out to you as preferred, or something to avoid given the hardware I'm using?

Q5: The storage network looks like it should be a dedicated 25gbs+ switch with nothing else on it. I have fast drives but only 3, would 25gps be enough. 100gps looks rather expensive but doable, if Mikrotik is an acceptable option.

Q6: Ceph needs a fast background sync network and public network - should they both be the same speed?

Q7: What PCIe NIC should I put into the PCs for the storage network? Single or Dual (for private/public)? The MS-01 has PCIe 4 x8 for the NIC.





Thanks for any help you may offer! Cheers!

Hi NerdAshes

wanted to ask. How is that Samsung PM9A3 performing?
Have you checked the power consumption at idle and load?
Do you know how much power doest it use?
According to specs 3w at idle and 11w under load.
What are you observing?
Does it get very hot?

Many thanks

 

[NerdAshes]

Hi NerdAshes

wanted to ask. How is that Samsung PM9A3 performing?
Have you checked the power consumption at idle and load?
Do you know how much power doest it use?
According to specs 3w at idle and 11w under load.
What are you observing?
Does it get very hot?

Many thanks

3 of them in 3 MS-01 are working great! No issues, heat seems fine? I don't know how much power it's using, I'm just happy it fits, it works and it's got great specs for the money.

I'd be happy to run some tests for you if you know of the commands (Debian).

I bought from Provantage (drop ship takes about 5 days)
Samsung PM9A3 7.6TB NVMe U.2 Enterprise SSD $596.26