Home server/workstation with virtualization

[ruffpl]

Hi and sorry in advance for my bad english.

I have a lot of questions because I have no experience in virtual systems ( I only used VirtualBox several times on Windows) and unix just for fun/rooting smartphones, etc.
I am in the middle of setup of my home server because I no longer have time to assemble new pcs in home, just want to have one professional central unit. The initially assumed expanded diy NAS turned into a solid rack-mounted workstation. It will be a main home computer located in a separate closed server room (along with router/switch, LTO backup, UPS and all cabling). It will be used by all household members, connecting to it via ethernet from their weaker laptops/PCs to access of their files (and maybe also virtual systems in the future). The equipment on which it will be based is Supermicro x10drhc motherboard with two xeons e52640v3, 16gb ddr4 ecc for each processor (for now), dual port GbE LAN and 1000W PSU all connected to Dlink 48port gigabit switch (DGS-1210-48). I am currently waiting for the provider to connect fiber internet so I don't know what additional equipment I will get to it. Until now in my last house I stored everything on a complex pc/NAS based on intel board, corei5, pendrive as bootloader and 8 sata hdds with Xpenology system installed on it.

I have a dilemma of how and what system I should install. Here's a list of what I would like to use it for:

- I need space to store a large amount of data (ZFS or BTRFS) with option of recovering it in the event of a failure ( i got previously purchased sata WD reds and Seagate Ironwolfs)

- system as NAS multimedia server (movies, music, etc) to stream it on tvs and smartphones by DLNA

- an application that would take over the function of torrent downloader/ like Synology Download Station (a big plus would be to have extensive options and build in search engine).

- a guest OS on which I could work and play newest games by connecting to it from my home office pc- win10 + hi end GPU (maybe it would be possible to do the same for my kids but install some weaker cards in pcix 3.0 8x slots?)

- additionally- monitoring aplication/system for house and surroundings

- additionally- VPN- I never used it (only through the browser) but I would like to have it on all home computers

- additionally- access to files from outside network ( if it would be securely set) and maybe some webhosting for me in the future


What kind of virtualization system would be right for me and where should it be placed? On pendrive as hypervisor + virtual systems on other disks, or everything on one high capacity hdd disk, ssd or fast sas disks in raid?

Which system option would be best:

- proxmox (supposedly gpu pass-through work well on it) or esxi + virtual systems ( win10 for me, NAS system for multimedia, f.ex. freenas, openmediavault, xpenology)
- NAS system + virtual systems stored inside it?
- some unix distro + KVM (or windows server)+ virtual systems inside it?
- some other option

Is something like that possible to perform for proper every day use? I would be grateful for any help.

 

[gigatexal]

imo BTRFS is not production ready -- stick to EXT4 or ZFS or XFS or anything but it.

I have experience with VMware stuff as well as KVM based VMs but never had much luck with passthrough though I know many here have.

Just personal preference, I would try Ubuntu LTS + ZFS + KVM and Virt-manager (GUI management) and go to town.

 

[ruffpl]

It was the second option that I would prefer because it doesn't sound so complicated.
The main system could be on one partition (which I would have backed up in case of problems), and the rest of my data stored normal on separate hdds in ZFS (I don't know how virtual systems save files on partitions they created / the whole data partition as one big vdmk file?)
Thanks for the advice. I will wait to hear also some other suggestions.

 

[BoredSysadmin]

It's a bit tricky, but you could get away with VMWare ESXi (boot from SATAdom or USB) with a pass-thru SAS controller to a Freenas VM and present it back to ESXi as NFS storage. Since you'd be running VMs from your ZFS - I'd highly recommend getting about 1tb read-intensive SSDs as L2ARC and one very fast optane(or write-intensive SAS) drive for ZIL Ubuntu lts VM with Docker installed (managed by Portainer and Watchtower) for your downloading/media stuff. One Windows for gaming with Passthru GPU to it. You could use this to stream it: Moonlight Game Streaming: Play Your PC Games Remotely

 

[ruffpl]

This Moonlight nvidia game stream sounds great but the rest You wrote is a bit too complicated for me.
After spending the day on youtube I came to the conclusion that I can not use any NAS system (freenas, xpenology,etc) as hypervisor because will not be able pass-thru gpu. So now virtualization selection- Esxi is closed source (and I dont know is free version will not have some linits) so Proxmox would be a better choice and I see its also very popular (and "provides full access to all functionalities"). Now I need to compare what will be better ( and easier) to set up and use as hypervisor- Proxmox or some Unix (ubuntu,debian,centos,etc).
I don't understand clearly what You wrote about disk installation (I would like to have it done as easy as possible). Also dont want to spend a lot of money for system disk because I dont know how much space I will need. Maybe better option would be buying f.ex 2 smaler ssds in raid and keep hypervisor and virtual systems on them. Games could be installed on some separate enterprise hdd like f.ex sas-3 HGST ( probably even used one should survive a lot). Rest of my data would be stored on other sata hdds formated in ZFS (with backup).
Is it a good idea?

 

[gigatexal]

Ok let’s start simple. What hardware do you already have?

I would just repurpose what you have already to prove out your use case so as to not blow a bunch of money on something that might not work out exactly as you need.

But a relatively simple setup could be the following:

Dual spinning rust as zdev mirrors. SSD for cache. As much ram as you have/can afford. Linux and kvm or VMware. The tricky part is software and GPU passthrough. Look into guides that talk about VFIO for more there.

 

[ruffpl]

I have a Supermicro x10drhc with 10xsata, 8xsas ports, 2 xeons (each 8 core/16 threads),2x 16gb ddr4, couple new sata WD reds and Seagate ironwolfs, 1 new HGST 1tb sas drive. Dont know yet how and what to set up as system drives.

 

[gigatexal]

It’s really up to you. But I would make a few mirrors of drives of the same capacity and keep say the 1TB drive as a system drive and run Ubuntu LTS with kvm and ZFS. VFIO is what will get you you pass though and there are guides online though it’s not for the faint of heart.