Re-thinking my ESXi homelab storage setup...but to what?

[Ch33rios]

Much like when I was a kid and had random urges to switch up my room, I've suddenly had the desire to re-evaluate my storage setup on my homelab server and I'm wondering what I should really do based upon my use. I'll first start with the setup:

E3-1230v5
32GB DDR4 RAM
LSi 9211-8i in IT Mode w/ SAS expander card
10 x Intel S3500 SSDs (800 GB)
2 x 480 GB Toshiba consumer-grade SSDs (ESXi datastore)
4 x 2TB WD-Reds

VM's I'm running:

BTRFS NAS Server (LSi card is passed through with the aforementioned Intel/WD drives attached with each currently in separate RAID 10 btrfs pools)
|_Runs Plex Media Server as well and acts as a NFS/SAMBA file server for the network and other devices

Photon OS Container Server
|_Runs various containers with persistent storage of containers on an NFS mount from the NAS server

Kubernetes Cluster (3 x VMs)
|_No persistent storage right now but may want to play more with persistent apps mounted via NFS

Unifi Controller + Unifi Video Server

Windows 2016 Core

------------------------------

Its worked just fine for over a year but I'm just wondering if its an 'optimal' sort of setup. I've been interested in playing with things like glusterfs and while I could create a virtual cluster and perhaps attach individual drives to the VMs via RDM, I'm not sure if that would be the right choice as a 'production' homelab setup. Although with a proper backup setup perhaps it wouldn't matter?

In any case, I'm interested in hearing what others might do in my place. Thanks for any input!

 

[spali]

I had long time a similar setup, but with 3 hosts running ESXi, but storage/share was virtualized FreeNAS with ZFS, which replicated from one server to the other for backup purposes.
In general it worked fine, but had a lot the chicken and egg problems with the storage. So I moved to Hyper-V instead of ESXi, to have the storage/share not virtualized. At least for me this made it a lot easier to manage, because all VM's can be sure the storage is available
(The storage VM could still be starting or even not working at the time of other VM's booted).
That allows to let the VM's use the share and be always backed up by a single backup job on the physical server. Especially to use persistent storage for your containers, a setup like this would be handy.
Relying in VM's on the storage which is a VM on the same level was just to complicated and error prone for me.
I think this would applies also for a single server setup.
So I would setup a windows server with Hyper-V and File-Server role and put the rest in VM's on it.

 

[acquacow]

I'm in the process of going diskless for my next ESXi box. Going to just have a small ITX console-sized box with USB boot. All storage will be provided via iSCSI and NFS from my FreeNAS box.

 

[spali]

That's even better as long you have a good storage nerwork in between.

 

[acquacow]

Yup, I'm all 10gige with both spinning disk and flash pools.

By the end of this weekend, I'll also have an all-flash gluster setup too.