Storage for a cluster of 3 Esxi hosts

sovking

Member
Jun 2, 2011
58
5
8
In a lab I have 3 Esxi hosts managed by Vcenter, each with an EPYC 32 core and 256 GB of Memory plus a TrueNas host equipped with E5-2680v2, 64GB of Memory and a storage of 6x8TB HDD swith SLOG over SAS SSD, this storage is shared to the ESXi hosts by the means of NFS and iSCSI. Currently there are no local datastores on the 3 ESXi hosts. The 3 hosts are connected to the storage host with redundant 56 Gbits Ethernet connections.

Now I've available 30 enterprise SFF HDD 10k, 900GB size, and I would like to reuse it for creating local datastore in each ESXi hosts. I can add SSD memory (SSD, M.2, U.2) as needed.

Altough I could take the VSAN approach, I'm evaluating to add a local VM to each ESXi datastore, with installed TrueNAS with 16/32GB of RAM that could expose a pool of 10 drives in RAIDZ2 + SLOG and L2ARC as needed. In this way I could move the VM storage to the local ESXi and they could mount the datastore locally instead of using the external storage node, and use the latter for archival and backup purposes.

This approach in some way is similar to the the VSAN approach ?
What performance I could expect moving to this setup ?
What is you advice ?
 

Rand__

Well-Known Member
Mar 6, 2014
4,620
918
113
Dont think the multi Truenas approach is sensible from a maintenance point of view; its not really similar to vSAN either as vSAN is a cluster solution which enables HA and storage vmotion, your solution is a bunch of local storage units.
Why not use a raid adapter locally instead? Way less hassle (ie some advantages, o/c some disadvanatges too)

You might be able to emulate vSan with TrueNas.Scale though when that hits the shelf...
 
  • Like
Reactions: Patrick

sovking

Member
Jun 2, 2011
58
5
8
Thanks Rand_: two of these three hosts have HBA card on them, not Raid card, so I can choose only approached that are without hw-raid.

From what I'm reading Vsphere HA (which I'm not sure to use), does not require VSAN to work
When I wrote that is similar I was referring from a performance point of view (both of them use tiered storage, both require passtrhough disks, etc), why I should need TrueNAS scale ?

Anyway what will be the advantages of using VSAN instead of a bunch of TrueNAS ? (given the hardware I'm using)
 

Rand__

Well-Known Member
Mar 6, 2014
4,620
918
113
well older sas 2 raid controllers should be cheap enough, but sure if you dont want to spend anything its an option.
you'd want to make sure you got proper slogs though if your sharing via nfs locally, or else performance will be very slow on spinners.

HA does needs a common shared datastore, which you won't have unless you have one host sharing to all 3 boxes; o/c you can share all 3 host's local disks to all 3 ESXi boxes, but that spells trouble with a capital T ...

vSan (or TrueNas Scale when available) allows you to have a real cluster solution, i.e. pooled storage with a single shared datastore, and all the associated vmware goodies on top.
 

BoredSysadmin

Active Member
Mar 2, 2019
465
128
43
multi-FreeNAS arrays make the least sense to me. going with vSAN is possible. You'll need to have at least one SSD per host.
Read specific hardware requirements here:

Also, keep in mind that vsan isn't free. I assume this is a home lab and you already subscribed to VMUG.
I am running all-flash vsan at home also with 3 hosts. I am using a 10gig network, cheap consumer SSDs, and plain SATA onboard controllers. Works ok after some alerts tuning with advanced options. With all-flash vsan, I am able to turn on Dedup and Compression to get back some of the space back, but it's not supported with vsan Hybrid storage (HD+SSD)
You'd also have to understand that storage efficiency with vsan, especially with hybrid storage won't be very high.
My suggestion for the best usage for these 30 drives is to add a new SAS disk shelf to existing FreeNAS storage.
 

sovking

Member
Jun 2, 2011
58
5
8
To better depict this setup here I put a simple schema:
lab-hw.png
As you can see the TrueNAS host share its storage, and currently I span the rest of 30 SFF HDDs on the three ESXi hosts.
So if I understood weel, it could run Vsphere HA, with the 3 ESXi hosts and the shared storage, but in this case the HDDs in the hosts are not useful.
Following @BoredSysadmin I could move to the TrueNAS host connecting another JBOD chassis.
Moreover HA is not my primary requirement, DRS could be interesting.
I would like to check if VSAN using the three hosts together with the running VM on the same hosts could provide some benefits, and then if VSAN (VMUG licensed) could be replaced by a three TrueNAS, one for node, if the performance is on the same level given this hardware.
Reading the @Rand__ answer I understand that with vSAN the storege will be pooled and if is used wisely (with DRS ?) it should provide some benefits (while the 3x local node TrueNAS are not pooled, just three different datastore to manage manually from VSphere).
Is this right ?
 

Rand__

Well-Known Member
Mar 6, 2014
4,620
918
113
More or less yes.

If its just for learning then vSan is an option, else I'd go with @BoredSysadmin 's recommendation. While 10 disks/host sound a lot its not to be expected that spinners will provide a kind of performance that would be enjoyable.
O/c totally depends on #of VMs you want to run. Just for having higher availability /drs /reboot capability/ it might be worth it though, depending on your requirements/expectations.

If you got all drives then its simple enough to test
 

Net-Runner

Member
Feb 25, 2016
81
24
8
37