Software Defined Home IT

What do you think about this setup

  • Total voters
    5
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
A

Ariston

New Member
Aug 28, 2023
7
0
1
For 2023 x-mas I decited to renew my aging home IT equiment and besides adding some performance my main goals are reducing power consumption and go a more software defined way. In detail that means:
  • Discard Mikrotik CCR
  • Discard Mikrotik CRS
  • Replace Xeon D-1541/8HDD server

So the new server will be server, switch, router and firewall all in one. I'm going to get a 25Gbit FTTH soon and with this setup I want to make sure that I can make use of it well ;)
All the hardware is ordered already and will arrive within the next 10 days.

Hardware
  • Supermicro X13SEM-TF board
  • AsRock SPC741D8-2L2T/BCM board
  • Intel Xeon Silver 4410Y CPU
  • 256GB DDR5-4800 Reg. ECC
  • 8x Intel DC P4510 8TB U.2
  • Intel E810-XXVDA2 Dual Port SFP28
  • Intel XL710-BM Quad Port SFP+
  • Fractal Design Meshify 2 Black case
  • 4 MCIO 74p to dual U.2 cables
  • 2 PCIe x16 to dual SlimSAS 8X riser cards
  • 4 SlimSAS 8X to dual U.2 cables
  • FSP 850W PSU

Considerations:
  • U.2 Hot-Swap backplanes look really fancy. However, in my professional life I never got VMD hot-swapping reliable. Also, each NVMe draws up to 22W of heat so I decided to install them fixed behind two quiet 140mm Noctua fans. This saves 700 bucks I might need elsewhere*.
  • The CPU architecure was choosen mainly because of QAT v2 support for Wireguard and the stunning mainboard that has the NVMe options built-in

Software
  • Proxmox PVE 8.1
    • KVM:
      • VyOS 1.5 with 25GBe NIC and QAT pass-trough
        • Container: Technitium DNS
    • LXC:
      • Samba AD DC
      • Samba FS
      • OCIS 4
      • Gitlab
      • Vaultwarden
      • Zabbix
      • Grocy


Considerations:
  • VyOS does not yet fully support VPP/DPMK so it might not work to get full 25Gbit. In this case I'd need to spend the saved money for TSNR.
  • I need to benchmark the storage options. Currently focussing on MDRAID-5/6 with XFS and ZFS RAIDz1/2.

Please share your thoughts and any experiences.
 
Last edited:
Laugh|nGMan

Laugh|nGMan

Member
Nov 27, 2012
38
7
8
Ariston said:
mainboard that has the NVMe options built-in
Click to expand...
Looks like some limitaton on these board or u still need correct backplane to utilize MCIO connectors….. to hit your target for all eight u.2 drives.

from sm faq section on their web site….

Question
Can X13SEM-F/-TF support 2 NVMe drives by directly connecting them to any one of the four onboard (CN2, CN3, CN5, CN6) MCIO ports with a Y cable such as CBL-MCIO-1245U2Y-E?
Can VROC RAID be supported with these MCIO ports?
Answer
Only one NVMe drive can be detected using a Y cable with direct connection to MCIO port due to hardware limitation. Two NVMe drives can only be supported per MCIO port with a Supermicro backplane. VROC RAID is supported by these MCIO ports. For a complete system reference with backplane, you can check out SYS-211E-FRDN2T.
 
Laugh|nGMan

Laugh|nGMan

Member
Nov 27, 2012
38
7
8
probably second u.2 drive on mcio dont want to initialize due there is seme logic put on these sm backplanes

funny thing is that same mcio cable is mentioned in sm faq but for mb model x13sei-tf….atx forrm factor i think and there is no such a problem :)

Question
Our motherboard is X13SEI-TF,can you provdie a cable P/N to support 2 U.2 NVME with onboard MCIO connector
Answer
Here is the cable P/N:CBL-MCIO-1245U2Y-E :MCIO x8 to 2x SFF-8639/U.2+Pwr

Question
We use a MCIO to 2x 8639 cable to connect twe U.2 NVME disk with X13SEM-TF, but only recognize one U.2 device, we have set the port to x4x4 mode. The cable also test with X13SEI-TF is OK.
Answer
X13SEM MCIO connector only supports one reset signal and was designed to work with our BPN, or special cable needed to be designed to buffer the reset to two in order to drive two devices.
 
Last edited:
L

louie1961

Active Member
May 15, 2023
221
96
28
I am not a fan of putting the networking (switch, router and firewall) on the same physical machine as the server. if you want to mess around with the server any reason, you run the risk of taking down your home network, which would never fly in my home. My wife would pitch a fit, since any/all of the things she cares about would be down (the TVs, Alexa, Spotify, her PC, etc.) My network has to stay running to keep her happy, so it is on its own physical device(s) apart from my proxmox servers, the NAS devices, etc.
 
  • Like
Reactions: itronin
A

Ariston

New Member
Aug 28, 2023
7
0
1
louie1961 said:
I am not a fan of putting the networking (switch, router and firewall) on the same physical machine as the server. if you want to mess around with the server any reason, you run the risk of taking down your home network, which would never fly in my home. My wife would pitch a fit, since any/all of the things she cares about would be down (the TVs, Alexa, Spotify, her PC, etc.) My network has to stay running to keep her happy, so it is on its own physical device(s) apart from my proxmox servers, the NAS devices, etc.
Click to expand...
Fair enough. I had no serious hardware issues in the past 20 years @Home so there's no reason for me to think of this as a blocker. And in the unlikely event of an emergency there's still a backup router in a cabinet that can do the job for a few days until the issue is fixed. I already run a ESXi/TNSR based server co-located with a comparable configuration regarding routing/firewalling for some time and never ran into any problems.

And for the wife... well, my wife is also digitally addicted, but as long as her mobile and 5G works there's no reason to bother ;)
 
Laugh|nGMan

Laugh|nGMan

Member
Nov 27, 2012
38
7
8
Just for reference. Not sure what i doing wrong, but i have boot issues with x13sem-tf with Xeon Gold 5515+ with pair sticks of ram, board spins up for second and nothing happens. BMC heart beat is ok. BMC and bios updated to latest, bios v2.1 with EmeraldRapids support, nothing changed,still boot isues.
 
Laugh|nGMan

Laugh|nGMan

Member
Nov 27, 2012
38
7
8
x13sem-tf + Xeon Gold 5515+ . Culprit (Supermicro PWS-865-PQ) in these combo was PSU. It just doesn't boot MB, only for half seconds cpu coolers spins-up and then stops.
Swapped to Corsair HX1000i 1000W and board booted properly.

I dunno, why sm single rail 865 psu with many amps was issues to boot 165W TDP cpu!

Bare board with Fedora 40 live idles at 60watts on wall with 8 sticks of 32GB RAM.

IMHO undocumented thing was about Intel VMD...in reallity i still need to choose between using pair M.2 drives or pair SlimSAS NVME U.2 drives thru CN1 (btw board recognizes only one drive thru this connector anyway). No way to see use them all under bios VMD section....so forget about two arrays (one with m.2, other with nvme)

ATM tmp config JONSBO UMX6 case:
Slot 4 - ConnectX®-4 100GbE dual-port (MCX456A-ECAT)
Slot 6 - NVIDIA RTX A4500
Slot 7 - 1xMicron 9400 MAX thru DeLOCK U.2 adapter (Item No. 90081)
M.2 - pair Micron 7450 MAX (one of them defo overheats heavy under slot 6)
SlimSAS NVME - 1xMicron 9400 MAX

1xFAN blowing thru Slot 4,6,7
 
Last edited:
You must log in or register to reply here.
Top Bottom