UnRAID/FreeNAS as ESXi Storage?

impulseGT

New Member
Sep 18, 2020
4
0
1
Firstly, I did try to search, but I couldn’t find anything specific to what I am trying to accomplish. If I missed it I apologies and if you could link to it I will go read that thread. A Pre thank you to any information you can provide.



What I am trying to do.

I am planning on building a Threadripper based VMware ESXi Host for my house. My goals are to be able to host multiple VMs for my house to serve various services. I also want the flexibility to stand up temp VMs to test software or system configurations for my work.

All of this is pretty simple for me as I have been managing VMware ESX servers since version 2. Where I have run into a bit of a question mark is that I also would like to separate compute and storage. I would really like to utilize an UnRaid/freeNAS ZFS backend linked via iSCSI at 10Gb (maybe?) to the compute esxi host. I have multiple reasons for wanting to do this, but other options may be available that I do not know about. I know very little about UnRAID/FreeNAS.

  • Being able to replace HDD to easily increase storage without having to migrate data/rebuild
  • If I need more compute horse power having storage be separate I could quickly and easily build a second ESXi host and also utilize vMotion.
  • I have some extra hardware that would be great for a storage node.
  • Being able to configure caching drives on my storage node should greatly increase speed.




So my questions...



Is this possible or am I trying to make UnRAID/FreeNAS do something it wasn't designed to do?

Has anyone configured unraid to link to VMware ESXi as the storage location for hosted VMs?

Is there anything I should consider?
 

BoredSysadmin

Active Member
Mar 2, 2019
453
124
43
This is very possible, at least with Freenas/iSCSI as I have done exactly this before, with only one concern - your storage needs to be fast enough to run VMs or it will be a pretty poor experience.
What I end up doing at home is separating my Freenas and VM entirely, as in 40TB Freenas only stores "a few family videos" and some Linux ISO file..., as for VMs - I build 3 node vSAN using recycled desktops - just beefed them using extra ram and cheap SSDs - I didn't have to pay for any of these as both memory and SSDs were considered leftovers and were given to me. Same for desktops. The only thing I had to pay for is a 10gig switch which is pretty much required if you thinking to do HCI.
 
  • Like
Reactions: impulseGT

impulseGT

New Member
Sep 18, 2020
4
0
1
I don't think I am going to need super fast storage requirements. I am only really looking at using the following services to my pretty small home network.

DC
File/Print
BIND
Security Camera VM (This is probably going to be the highest read/write system)
Plex (Maybe)
Some basic Win10 Workstations for RDP via Pi
(EDIT)
Plus some Test VMs so I can test different things for work like SharePoint/Splunk/etc



I currently have five 4TB WD Reds and I plan on pairing those with a 1TB SSD Cache running ZFS1, but I want to eventually replace the REDs and maybe go all flash. I feel FreeNAS would give me that flexibility.
 

BoredSysadmin

Active Member
Mar 2, 2019
453
124
43
"Security Camera VM " - I assume you mean a NVR software. How much disk speed it needs heavily depends on how many cameras you'll be recording. Check with NVR software for suggestions.
Also check that your WD Red are actually CMR and not SMR type, as the later is not a great fit for ZFS:
 

impulseGT

New Member
Sep 18, 2020
4
0
1
"Security Camera VM " - I assume you mean a NVR software. How much disk speed it needs heavily depends on how many cameras you'll be recording. Check with NVR software for suggestions.
Also check that your WD Red are actually CMR and not SMR type, as the later is not a great fit for ZFS:

They are CMR drives. I specifically avoid SMR drives because of the issues that came up with WD.

As for the cameras. I don't have all of the specific yet, but my goal is around 3-4 cameras, recording only during activity with a high res and low frame rate (2-4 fps). I believe that would not generate too much load on the disk, but I do not have any experience in that either. Thoughts?
 

gea

Well-Known Member
Dec 31, 2010
2,520
852
113
DE
Basically, the question is not FreeNas or Unraid but

- Free-BSD vs Linux vs Solarish
- realtime raid protection vs non raid with backup on demand on a redundancy disk
- realtime raid scale performance with number of datadisks while unraid performance is always like a single disk
- unraid can sleep all disks but the active one while with realtime raid all disks must be up

Independend from this is the question if you want a copy on write filesystem (no corrupt filesystem on a crash during a write, snaps) and protection against silent errors due data/metadate checksums (=ZFS)

Then it comes to resource needs of a storage VM for ZFS.
Less ressorce hungry on Free-BSD for use as a storage VM is XigmaNAS, lowest resource needs are with Solarish based solutions where ZFS is native like Oracle Solaris (native ZFS) or a Solaris fork like OmniOS (Open-ZFS where newest features like encryption and special vdevs are available for over a year unlike Free-NAS where they are still in beta).

-SAMBA (SMB server on all X platforms) vs ZFS embedded SMB server (Solarish only that has superiour Windows support due ntfs alike permissions, local SMB groups and troublefree snaps as Windows prevouous versions, is very easy to setup (just set to on) but with less overall features than SAMBA.
 
Last edited:

BoredSysadmin

Active Member
Mar 2, 2019
453
124
43
So is the recommendation FreeNAS over UNraid?
Can't say anything good or bad about UnRaid since I have zero experience with it. It isn't free, but most people who use it, seem to like it due to ease of use and easy to expand.
If you need a bulletproof enterprise-class filesystem that is robust, resilient, and reliable = go with ZFS - be wary assuming easy ZFS scaling up or migrating - it's tricker than you might think.
If you need more flexibility and easier to use/expand - Unraid may work better.
 

Rand__

Well-Known Member
Mar 6, 2014
4,592
912
113
Have not used Unraid, but from what I know
  • Being able to replace HDD to easily increase storage without having to migrate data/rebuild
Very simple with UnRaid, potentially not so simple with FreeNas/Napp-It (replacement/extension per vdev)

  • If I need more compute horse power having storage be separate I could quickly and easily build a second ESXi host and also utilize vMotion.
Check for both
  • I have some extra hardware that would be great for a storage node.
easy for UnRaid, depends on the HW for FN/Napp-IT

  • Being able to configure caching drives on my storage node should greatly increase speed.
No idea re UnRaid, kind of for FN/Napp-IT (a part of memory is used for caching)


Both can host VMs for ESXi, it really depends on what your secondary criteria are.
Commodity HW, single disk expandability -> UnRaid
More stringent HW req's, enterprise lvl features (both a pro and con) -> FN/Napp-It
 

Spartacus

Well-Known Member
May 27, 2019
688
265
63
Austin, TX
So I actually ran unraid in this scenario for a while with two R620s in a cluster with my VMs on unraid SSD cache, had it set so it kept the VM data on the cache and never moved to slow storage on the HDD.
I ended up upgrading all of my equipment to Xeon v3 stuff and made two mirrored unraid boxes to integrate the storage and compute together, its not HA anymore but for what I'm doing more than sufficient redundancy wise now.
  • Being able to replace HDD to easily increase storage without having to migrate data/rebuild
    • So how do you want to be able to increase storage? One drive at a time? or are you open to purchasing multiple?
      • This is one of unraids ideal features, you can add one drive at a time to increase storage, vs with Freenas (or ZFS on Linux) you have to either A) replace ALL of the drives with larger ones and let it rebuild the disk each time (it doesnt expand the storage usability until the entire vdev is made up of the larger drives), or create a new drive vdev to add to the existing one which means you still probably need an equal amount of drives. The drawback on unraid however is on slow data you're limited to each individual drive speed on the slower drives which is normally about 120-130MB/s on average freenass will scale with the number of drives, the standard size is a 6 disk raid 6 which yields about 600MB/s depending on the disks. Additonally freenas you really need WD red or at least NAS type drives because it is a more traditional raid, because unraid behaves more like individual disks in a large storage pool almost any drive type is acceptable. Lastly if you setup raid 6 with freenas and happen to lose 3 drives, all the data is gone, with unraid if you lose 3 drives, you only lose the data that was on that 3rd disk directly (assuming you had dual redundancy setup).
  • Being able to configure caching drives on my storage node should greatly increase speed.
    • This can be done on both freenas and unraid however for speed, cost and ease freenas does better with stuffing as much ram as you can into the box as it can use that for cache. This does make it more susceptible to dataloss though you would want a battery backup for the storage for sure. Between my two setups on unraid my main box is mirroring 2x M.2 for my VMs (basic usage and such) and on my secondary its 4x SAS SSD in raid 10 (for high write video encoding/decoding stuff).
  • If I need more compute horse power having storage be separate I could quickly and easily build a second ESXi host and also utilize vMotion.
    • Correct this works with whatever you choose as long as you have a 10Gb switch or multiple 10Gb ports on your storage.
      • I personally only had 2 hosts so I bought a dual SFP+ port card for the storage and two single port cards for the servers and then attached the datastores independently with NFS rather than using a switch.
  • I have some extra hardware that would be great for a storage node.
    • What kind? If its not server/workstation equipment with ECC I would avoid freenas, as I noted it does best with as much ram as you can give it and ECC is technically recommended and has alot larger stick than the UDIMM counterparts.
    • How many drives (both SSD and HDD) are you wanting to start with for your storage?