FreeNAS and ESXi hardware sanity check/advice

[bdolbs]

Hey guys,

I'm relatively new here and have been reading through the forums to try to learn about FreeNAS and how to set it up on my ESXi server. There are lots of good tutorials here and I feel like I have learned enough to get started but, before I do I was hoping to get a quick sanity check on my hardware. Any tips/advice will be most appreciated.

Several years ago, I wanted a home lab that I could use to run some experiments for work. I bought the following:

Chasis: SC721TQ-250B (SC721TQ-250B | Mini-tower | Chassis | Products | Super Micro Computer, Inc.)
Mainboard: Supermicro x10sdv-tln4f (Supermicro | Products | Motherboards | Xeon® Boards | X10SDV-TLN4F)
Processor: Intel® Xeon® processor D-1541
Memory: 128GB
Storage Drives: 4 - 3TB WD RED NAS drives
SSD: 1 Samsung EVO 850 512GB

This worked great for setting up VMs to experiment with (was evaluating various Hadoop distros) though to be honest I never used it as much as I hoped (I found it easier to spin up temporary VMs on my Mac with Vagrant and I never found much of a need to running VMs for long periods of time which may have made me use my ESXi server more). So now, I have decided that I want to repurpose this as a FreeNAS server but I want to run it on a VM in ESXi so that I can still run other VMs on the same server if I find a need to.

My main question is, does this hardware setup make sense or are there some problems with it?

I don't currently have all that much data to store (less than 1 TB so the space/# of disks doesn't seem to be a problem for my needs). I think I'll primarily use this for backups of the various machines on my network and maybe run Plex.

The main thing that I think may be wrong is that from what I read, it seems that I may need to buy an HBA so that the disks can be passed directly to the FreeNAS VM so that ZFS will work correctly. (Yes, You Can Virtualize FreeNAS - FreeNAS - Open Source Storage Operating System). Well, it sounds like it is possible to do it without the HBA but, it doesn't sound like it's recommended. I'd never even heard of an HBA before so I'm not terribly familiar with them...I only know the limited amount I've read in the past few days. Ideally, I'd be able to plug it in to the one PCI-E port on the mainboard and then connect all the drives to it internally though I worry about the way the case is setup with a HD backplane. Thoughts?

Other than that is there anything else I need to worry about?

 

[MiniKnight]

That setup will work fine. Many here use something very similar. You may want to add a SATA DOM SUPERMICRO - COMPONENTS SSD-DM064-SMCMVN1 SMC SATADOM 64GB SATA 6GB/S 672042221646 | eBay

They require no drive bay and no cable. Great if you want to install ESXi. That is the 64GB but there are smaller and larger sizes. Get a SM one that can be powered by the gold port on your motherboard because the small cables suck.

 

[Rand__]

Typical HBA for spinners would be Dell H200/310 or IBM M1015, both use the trusty old 2008 chipset and should be available for ~50 bucks used. You will need an adapter cable from the HBA to the backplane ports, similar to Supermicro Internal MiniSAS SFF-8087 to 4x SATA 50/50/50/50cm with sideband 50cm (CBL-0097L-02)
There should be one compatible from SM with your Chassis which has the correct cable lengths, check compatibility.

Oh and its totally working fine, using several virtualized FreeNas boxes for years.

 

[svtkobra7]

Hey guys,
Memory: 128GB
Other than that is there anything else I need to worry about?

• Not that I can see, just stay away from Corral (bad joke).
• With four drives, not much consideration has to go into pool layout, 2 x 2 mirror and done.
  
• On 11.2 BETA, the UI is getting better, but its still not there yet (for me). My preference remains the legacy UI.
  
• The FreeNAS forum guys get pretty worked up about your memory being ECC (not that I completely disagree). What type of RAM? [EDIT: Disregard the ECC question, I just remembered that board can only support 128 GB with RDIMMs]

You may want to add a SATA DOM SUPERMICRO - COMPONENTS SSD-DM064-SMCMVN1 SMC SATADOM 64GB SATA 6GB/S 672042221646 | eBay

They require no drive bay and no cable. Great if you want to install ESXi. That is the 64GB but there are smaller and larger sizes. Get a SM one that can be powered by the gold port on your motherboard because the small cables suck.

• I actually use this exact SATADOM, but unfortunately with an X9 (no gold ports for me), I have to use that tiny cable (you don't as noted). Interesting that your board has both the gold power which doesn't require that power cable (X10+) and yet still has a SATADOM power connector nearby.
  
• It seems as thought there are two camps here: (a) those that think SATADOMs are too pricey for the performance they provide, and since you have the spare SATA port and space, you should buy a better performing SSD for less, and therefore advocate this solution only when you don't have a drive bay available (b) those that think the SATADOM is a "neat" solution (like myself) and appreciate not having to deal with more fuss of cabling / space, etc. As a disclaimer, I use mine as my ESXi boot drive and it has been performing flawlessly since STH made me buy it as there was a deal for that model for ~$30.

Typical HBA for spinners would be Dell H200/310 or IBM M1015, both use the trusty old 2008 chipset and should be available for ~50 bucks used. You will need an adapter cable from the HBA to the backplane ports, similar to Supermicro Internal MiniSAS SFF-8087 to 4x SATA 50/50/50/50cm with sideband 50cm (CBL-0097L-02)
There should be one compatible from SM with your Chassis which has the correct cable lengths, check compatibility.

• Agree, with the use of an HBA, period. However,
  
• But if he goes that route he just burnt his one and only PCIe slot on an HBA.
• There is no onboard USB (only front and rear, so that rules USB boot out, which I'm not a fan of anyway), but what about using an m.2 boot drive, forgoing the HBA, and leaving that PCIe slot open for the future should he need it for something else? I haven't checked that specific motherboard, but I would imagine that there would be no issue passing through the SATA controller to the FreeNAS VM, obviating the need for an HBA??? Or getting more creative, since his board supports bifurcation, he could alternatively use the PCIe slot for two m.2 (instead of an HBA) and use one for boot, one for cache (ZIL, L2ARC very unlikely to be needed @ 128GB RAM).
  
• (just a thought and wanted to present an alternate approach for consideration)

 

[Rand__]

Its an option (passing through Sata) but unless there is an absolute need for the slot I'd not recommended this for a new user as it's a hack which
-might or might not work on a future ESX version
- is simple enough but not trivial if one has no experience editing Esx core files

 

[svtkobra7]

Its an option (passing through Sata) but unless there is an absolute need for the slot

• Fair point, I suppose if he needed to throw some Optane AIC in there later, he could revert to my suggestion (only made for the sake of providing options).

I'd not recommended this for a new user as it's a hack which
-might or might not work on a future ESX version
- is simple enough but not trivial if one has no experience editing Esx core files

• Didn't we already see our last ESX version around 4.0 or thereabouts?
• Regarding a formal recommendation, you are correct (of course) to provide guidance that doesn't blow something up. I misread his post in that I thought he had been using ESXi for two years and therefore SATA controller passthrough wouldn't be an issue, but upon reading again, I realized he noted he had mostly been using other virtualization solutions.
  
• But allow me to hold my comment that provided he is aware of the risk, he could initially buy a cheap HBA, become more comfortable with ESXi / FreeNAS and decide to have some fun with NVMe, thereby using that single PCIe slot to achieve IOPS orders of magnitude higher. Surely your recommendation is the smart, risk-averse one, but I know part of you wants him to experience Optane (thus requiring pass through in his case) too.

@ bdolbs

Since you referenced that FreeNAS post (similar to STH, I find the FN forum to be a knowledgeable and helpful community), let me put another one in front of you: Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]

The author of that post (Stux) really knows his stuff and I think you stand to benefit from reading, considering it is full cycle documentation of a FreeNAS install on ESXi, especially relevant as he uses your motherboard. As a trivial example, he details how to correct Xeon D sensor monitoring (not sure if that was fixed in 6.7 or not, as that tutorial is based on 6.5)

Simply to present an alternate option you may or may not be aware of, the current production stable release of FreeNAS is 11.1 U5, but the release of 11.2 is targeted for Nov 5th. 11.2, which is currently in Beta 2, has much stronger stronger and more stable support for VMs than current (11.1 U5), along with a number of other enhancements out of scope of this discussion, and natively supports Docker. You could install FreeNAS baremetal and run VMs from FreeNAS, but ESXi is a much, much more robust solution (and what nearly everyone would recommend, just providing an alternative play).

And I wouldn't worry so much about not knowing what an HBA is. My bad jokes aside, I mean that very seriously (but in a broader context) as I shunned COTS NAS appliances in May '17, and at that point in time, had never used ESXi, pfSense, or FreeNAS prior to a massive box showing up at my home that I purchased on ebay. Like yourself, initially, I didn't truly understand what an HBA was and if asked what "IT Mode" was I would have suggested it was something related to matrix organization format and not a firmware flash needed to properly present disks to FreeNAS. You can quickly overload yourself with information if you are starting from 0 (as I was, but I also don't have an IT background). Nevertheless, approximately 6 weeks after that chassis arrived, I had deployed a stable AIO solution, properly burned in, with relative ease which met my needs. It did take me a bit longer than 6 weeks to convert my coat closet into a server closet with sound abatement and an active airflow system. Of course it took a lot of time to achieve a stable / working build, and I failed a good bit before succeeding, but it was a fun experience (especially mounting my server upside down, irrelevant in your case I know). Thankfully there are always kind, helpful, and extremely well informed individuals like @Rand__ there to lend an experienced hand when you need it. Even though he calls "ESXi" "ESX" I promise he knows what ESXi is. <= bad inside joke.

My ultimate point is simply to offer you some support, as whether 1 year ago, or 10, we were all just getting started. And as @Rand__ might share, the only time I blow things up now is when I get the urge to "tinker," but that issue was remedied yesterday with the ebay 15% coupon + multiple ebay accounts = ~80% of components for 2 new server builds sourced (to replace my current/single AIO server).

Good luck with your install.

 

[K D]

If you are not set on esxi+ freenas thn look at proxmox as the base hypervisor. You will eliminate the need for a san/nas appliance and can use the onboard sata ports.

See my old post for instructions on using esxi with sata pass-through for a nas vm. I had used omnios in the post but the instructions can be used for any nas vm. Also, in your case, you can use an nvme drive for the host datastore and avoid usb pass-through.

https://forums.servethehome.com/index.php?threads/esxi-napp-it-all-in-one-with-usb-datastore.15897/

 

[Rand__]

@svtkobra7 - Old habits die hard with ESX and -i

And if he is keen on Optane which actually is quite awesome then he could either use a M2 card (less performance than the full optane's but still good), or he could use an optane via U2 to M2 adapter (or even PCIex4 to M2 adapter) and pass that through

Or maybe he will be bitten by the hardware bug and will get a a second nice system and then can tinker on his first one all he wants - and congrats on the new HW