Reevaluating my server config: ESXi+FreeNAS or OMV+Snapraid+MergeFS?

[Beer]

EDIT: SOLVED




Sorry if this isn't the right forum.. I wasn't sure where to put it since there are so many different moving parts here.

I'm at a bit of a crossroads here. I'm in the middle of setting up my home server build and starting to second-guess myself on the storage/VM configuration.

Hardware (already purchased, server mostly built):

• Xeon E5-2630L V3
• ASRock Rack EPC612D4U (8x SATA6.0)
• 32GB DDR4 RDIMMs
• 6x 4TB HGST NAS
• 2x 512GB SSD
• 3x USB flash drives

Original plan:

• ESXi on bare metal, installed to 1x USB flash drive
• FreeNAS in a VM, mirrored on 2x USB flash drives
  • All disks passed through to FreeNAS VM
  • 6x HDD in RAIDZ2 (16TB usable)
  • 2x SSD mirrored, volume shared back to ESXi host for VM storage

Usage:

• Media server/Plex
• Home backups
• VMs for various tasks

Fast forward to now. I'm still somewhat concerned about the issue with ZFS where VDEVs can't be grown in size (without the "replace-one-drive-at-a-time" workaround), should I ever want to increase storage.. and the heightened risk of disk failures during resilvering. I also like the idea of how the mergefs/snapraid combo helps with noise/heat/power consumption, as there is no striping and single disks are accessed individually, and that if you lose a drive, you only lose what was on that one drive.

I've been hearing more and more about using snapraid+mergefs for home media servers.. and it sounds nice but it's throwing off all my plans.

I don't have a clue if it's possible to do the passthrough and sharing volumes from the guest back to the host. I don't know how my SSD mirroring would work in this scenario. I don't know if my previous USB situation would work out with this, or if OMV/snapraid/mergefs needs to be on a hard drive rather than USB. All of these are brand new to me.

Should I just stick with my original plans or is this something worth looking into more? I guess what I'm looking for here is help trying to work out a gameplan for the alternative setup (with my existing hardware) to at least get me on track, because I'm a little overwhelmed at the moment. Or if this is even something I should be considering at all at this point..

 

[MiniKnight]

You can add capacity. Just add two mirrored drives at a time in ZFS and life is good.

ESXi is still the standard for virtualization.

To me it's the difference between using enterprise (ESXi + ZFS) or using a home setup. ESXi is everywhere in production. ZFS whether on Linux, FreeBSD or Solaris is popular in production. You're right that people aiming for reliability are going to use more power.

 

[nk215]

Is there anything specific about freeNas that you need to have beside ZFS?

if not, take a look at rockstor which uses btrfs. The nice thing about btrfs is in its flexibility to add disk, remove disk/resizing pools.

I've been playing around with btrfs and really like it. btrfs also does copy-on-write file system and self healing similar to zfs.

 

[vl1969]

I am somewhat at crossroads myself about to switch my build. (currently running OMV no virtualization.)
similar to OP but I am a bit confused.

OP has 2 completely different setups proposed.
one with Hypervisor and one without. what would be the purpose of the build ? is it a simply file share/NAS build or does it need to have VM capability ?

I am asking because my plans are to have a Virtualization server and file server on one hardware.
so I am trying to figure out how to share local storage and have a nice GUI for management on one box.

I am looking on Proxmox host with some VMs for special needs. but I also want a nice File server.
I want to use BTRFS for data store and ZFS-raid1 for OS drive as Proxmox now supports boot drive using ZFS/raid1 setup right in the installer.

 

[Beer]

Is there anything specific about freeNas that you need to have beside ZFS?
if not, take a look at rockstor which uses btrfs. The nice thing about btrfs is in its flexibility to add disk, remove disk/resizing pools.
I've been playing around with btrfs and really like it. btrfs also does copy-on-write file system and self healing similar to zfs.

ZFS and a powerful/flexible webui are about the only things I need out of FreeNAS. I prefer having a comfy UI to poke around in, rather than doing everything from the terminal. I will look into it, thanks

OP has 2 completely different setups proposed.
one with Hypervisor and one without. what would be the purpose of the build ? is it a simply file share/NAS build or does it need to have VM capability?

Hypervisor hasn't been removed from the equation. The thought was to still use ESXi, but use it along with OMV+Snapraid+mergefs running in a VM with disks passed through, and then share the volumes back over to ESXi same as I would with Freenas .. but I don't know if that's possible/stupid. Trying to wrap my head around how the new setup would even work

 

[cheezehead]

Have you been able to get FreeNAS to run off of mirrored pair of USB thumb drives?....i've tried this in the past but ESXi couldn't recognize them as datastores to do the actual install. I've need to run a mirror pair of SSDs or spinners for the FreeNAS install then pass-through the drives either as physical RDM's....works but messy or via passing the HBA through.

I ended up skipping ESXi on that box and just using FreeNAS with jails for the storage, backups, and Plex which have been working for awhile now. FreeNAS does offer a better virtualization engine which can run Windows VM's if you have a need for it.

 

[whitey]

Is there anything specific about freeNas that you need to have beside ZFS?

if not, take a look at rockstor which uses btrfs. The nice thing about btrfs is in its flexibility to add disk, remove disk/resizing pools.

I've been playing around with btrfs and really like it. btrfs also does copy-on-write file system and self healing similar to zfs.

Posts like these do NOT give me a 'warm fuzzy' abt the state of btrfs currently. YMMV, there's always raid0/1/10 that seems to be fairly stable/solid w/ btrfs.

RAID56 - btrfs Wiki
[BUG] Btrfs scrub sometime recalculate wrong parity in raid5
Re: [BUG] Btrfs scrub sometime recalculate wrong parity in raid5
Btrfs RAID 5/6 Code Found To Be Very Unsafe & Will Likely Require A Rewrite • /r/DataHoarder
PSA: Don't use BTRFS with Raid5/6 - Ars Technica OpenForum

EDIT: LOVE this comment in one of the threads:

"raid56 mode may eat your babies"

 

[cheezehead]

Current BTRFS status page: Status - btrfs Wiki

The Raid5/6 bugs are still present.

 

[Rand__]

Have you been able to get FreeNAS to run off of mirrored pair of USB thumb drives?...

Is there not a workaround to get those USB drives working as datastores? Nothing official o/c...
USB Devices as VMFS Datastore in vSphere ESXi 6.0


But why in the first place?
Unless you are passing through nic's beside the hba(s) your config will not be able to run natively without ESX anyway? Or what am I missing?

 

[vl1969]

Posts like these do NOT give me a 'warm fuzzy' abt the state of btrfs currently. YMMV, there's always raid0/1/10 that seems to be fairly stable/solid w/ btrfs.

RAID56 - btrfs Wiki
[BUG] Btrfs scrub sometime recalculate wrong parity in raid5
Re: [BUG] Btrfs scrub sometime recalculate wrong parity in raid5
Btrfs RAID 5/6 Code Found To Be Very Unsafe & Will Likely Require A Rewrite • /r/DataHoarder
PSA: Don't use BTRFS with Raid5/6 - Ars Technica OpenForum

EDIT: LOVE this comment in one of the threads:

"raid56 mode may eat your babies"

well to be fair the state of straight up no raid BTRFS and RAID 0/1/10 levels are pretty stable and have been for ages.
it does what is expected and does it well.

now if you want raid 5/6 than you need to look elsewhere.

 

[vl1969]

ZFS and a powerful/flexible webui are about the only things I need out of FreeNAS. I prefer having a comfy UI to poke around in, rather than doing everything from the terminal. I will look into it, thanks



Hypervisor hasn't been removed from the equation. The thought was to still use ESXi, but use it along with OMV+Snapraid+mergefs running in a VM with disks passed through, and then share the volumes back over to ESXi same as I would with Freenas .. but I don't know if that's possible/stupid. Trying to wrap my head around how the new setup would even work

ohh, I see. sorry I have misunderstood your goals.

what's funny is I am at pretty much the same situations myself.
Except that I am trying to choose(and find if it is even possible) between
running Proxmox bare setup + OMV VM with disks passthrough OR (and that is where my confusion begins) Proxmox + BTRFS RAID-10 pool as DataStore on the host + NFS shares to provide access to the datastore folders to VMs and PC clients.

!! Yes I want BTRFS for my local DAS, I understand BTRFS ,know how to work with it, and it works with mixed size drives just fine, unlike ZFS. !!

I do however want an easier way for administration of all of that.
if I had a GUI to work with I would maybe used individually formatted BTRFS with SnapRaid and MergerFS instead of raid10 but CLI is not my forte. wish OMV had a nice plugin for KVM so didn't need to run Proxmox.

I don't mind ESXi much but at home I do not have any windows PC so can not properly administrate free version of ESXi. also my hardware might not be supported by it properly, where Proxmox does supports it well.

 

[cheezehead]

Is there not a workaround to get those USB drives working as datastores? Nothing official o/c...
USB Devices as VMFS Datastore in vSphere ESXi 6.0


But why in the first place?
Unless you are passing through nic's beside the hba(s) your config will not be able to run natively without ESX anyway? Or what am I missing?

I've never gotten to work on my setups in the past...ended up splitting out my workloads and everything has been good.

The concept in all-in-one homelab scenario you would have a pair of usb thumb drives. The first would have ESXi installed and the second would be passed through as a datastore for your NAS/SAN distro of choice....I was looking at FreeNAS as the time. Then pass through the HBA(s) to the FreeNAS VM and you would get close to bare metal performance from FreeNAS plus the ability to run whatever VM's you want off of iSCSI/NFS (FreeNAS hosted) LUNs.

 

[Beer]

Have you been able to get FreeNAS to run off of mirrored pair of USB thumb drives?....i've tried this in the past but ESXi couldn't recognize them as datastores to do the actual install. I've need to run a mirror pair of SSDs or spinners for the FreeNAS install then pass-through the drives either as physical RDM's....works but messy or via passing the HBA through.

I ended up skipping ESXi on that box and just using FreeNAS with jails for the storage, backups, and Plex which have been working for awhile now. FreeNAS does offer a better virtualization engine which can run Windows VM's if you have a need for it.

Hey I'm glad you brought this up. This is what my intention was from the beginning (Freenas on mirrored USB).. and I ran into the exact issue you had last night. ESXi not detecting USB for a datastore. I tried to follow the guide @Rand__ posted but I'm not sure I'm doing it right. I SSH'd into the server as root, but when I tried running this command, exactly nothing happened:

esxcli storage vmfs snapshot list

No errors.. just no output at all.

(using the guide for 6.5 though, not 6.0: USB Devices as VMFS Datastore in vSphere ESXi 6.5)

I've decided to just stick with my original plans and just go with ESXi+FreeNAS .. if I can get this working.

 

[MiniKnight]

Current BTRFS status page: Status - btrfs Wiki

The Raid5/6 bugs are still present.

This.

 

[Rand__]

esxcli storage vmfs snapshot list

No errors.. just no output at all.

(using the guide for 6.5 though, not 6.0: USB Devices as VMFS Datastore in vSphere ESXi 6.5)

I've decided to just stick with my original plans and just go with ESXi+FreeNAS .. if I can get this working.

Did you get the errors in the vmkernel log? I.e. did you have issues with mounting the datastore in the first place?

 

[Beer]

Did you get the errors in the vmkernel log? I.e. did you have issues with mounting the datastore in the first place?

The USB stick doesn't show up in the available list for me to make a datastore in the first place. So I followed that guide I linked, which is supposed to get ESXi to recognize the USB devices so they can be used to create a datastore

USB shows in devices in vmware:

EDIT: Correction - The USB above is my ESXi installation (also on USB)

So the other USBs are simply not showing up.

[root@localhost:~] lspci | grep USB
0000:00:14.0 Serial bus controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller
0000:00:1a.0 Serial bus controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [vmhba32]
0000:00:1d.0 Serial bus controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [vmhba33]

But I cannot choose it for a datastore:

"esxcli storage vmfs snapshot list" command does nothing:

EDIT: Getting somewhere. If I tail /var/log/vmkernel.log and insert one of my USB sticks, I get this:

2017-01-17T19:25:04.714Z cpu6:66006)VMK_PCI: 393: Device 0000:00:1d.0 name: vmhba33
2017-01-17T19:25:04.714Z cpu6:66006)DMA: 646: DMA Engine 'vmhba33' created using mapper 'DMANull'.
2017-01-17T19:25:04.727Z cpu0:65609)ScsiScan: 446: Path 'vmhba33:C0:T0:L0': Failed to read VPD Serial id page: Not supported
2017-01-17T19:25:04.728Z cpu0:65609)WARNING: xpt_scsi_adapter_discover:1224: unable to find target 1: No connection
2017-01-17T19:25:04.728Z cpu0:65609)WARNING: VMW_SATP_LOCAL: satp_local_claim:366: VMW_SATP_LOCAL does not support multiple paths per device.  Refusing to claim path vmhba33:C0:T0:L0.
2017-01-17T19:25:04.728Z cpu0:65609)WARNING: VMW_SATP_LOCAL: satp_local_claim:370: preexisting dev:naa.2020030102060804 path:vmhba32:C0:T0:L0 vendor:Samsung model:Flash Drive FIT
2017-01-17T19:25:04.728Z cpu0:65609)WARNING: NMP: nmp_SatpClaimPath:2089: SATP "VMW_SATP_LOCAL" could not add  path "vmhba33:C0:T0:L0" for device "naa.2020030102060804". Error Not supported
2017-01-17T19:25:04.728Z cpu0:65609)WARNING: ScsiPath: 6606: Plugin 'NMP' had an error (Not supported) while claiming path 'vmhba33:C0:T0:L0'. Skipping the path.
2017-01-17T19:25:04.728Z cpu0:65609)ScsiClaimrule: 1393: Plugin NMP specified by claimrule 50 was not able to claim path vmhba33:C0:T0:L0: Busy
2017-01-17T19:25:04.728Z cpu0:65609)ScsiClaimrule: 1691: Error claiming path vmhba33:C0:T0:L0. Failure.

 

[Rand__]

Doe they show up in "esxcli storage core adapter list"?
And what does "cat /var/log/vmkernel.log" say?

 

[Beer]

Doe they show up in "esxcli storage core adapter list"?
And what does "cat /var/log/vmkernel.log" say?

See my edit for vmkernel info. Here is adapter list

[root@localhost:~] esxcli storage core adapter list
HBA Name  Driver    Link State  UID          Capabilities  Description                                            
--------  --------  ----------  -----------  ------------  ----------------------------------------------------------
vmhba0    vmw_ahci  link-n/a    sata.vmhba0                (0000:00:11.4) Intel Corporation Wellsburg AHCI Controller
vmhba1    vmw_ahci  link-n/a    sata.vmhba1                (0000:00:1f.2) Intel Corporation Wellsburg AHCI Controller
vmhba32   vmkusb    link-n/a    usb.vmhba32                () USB                                                 
vmhba33   vmkusb    link-n/a    usb.vmhba33                () USB                                                 
vmhba34   vmkusb    link-n/a    usb.vmhba34                () USB

I should also note that I have stopped usbarbitrator service before doing any of this and that all 3 of my USB sticks are identical. The one that ESXi is running on is the same model/capacity as the ones I'm trying use as datastores


EDIT: Results of: "esxcli storage core path list"

1st USB, which ESXi is booting from:

usb.vmhba32-usb.0:0-naa.2020030102060804 
  UID: usb.vmhba32-usb.0:0-naa.2020030102060804 
  Runtime Name: vmhba32:C0:T0:L0 
  Device: naa.2020030102060804 
  Device Display Name: Local USB Direct-Access (naa.2020030102060804) 
  Adapter: vmhba32 
  Channel: 0 
  Target: 0 
  LUN: 0 
  Plugin: NMP 
  State: active 
  Transport: usb 
  Adapter Identifier: usb.vmhba32 
  Target Identifier: usb.0:0 
  Adapter Transport Details: Unavailable or path is unclaimed 
  Target Transport Details: Unavailable or path is unclaimed 
  Maximum IO Size: 32768

2nd USB which is not being recognized:

usb.vmhba34-usb.0:0- 
  UID: usb.vmhba34-usb.0:0- 
  Runtime Name: vmhba34:C0:T0:L0 
  Device: No associated device 
  Device Display Name: No associated device 
  Adapter: vmhba34 
  Channel: 0 
  Target: 0 
  LUN: 0 
  Plugin: (unclaimed) 
  State: dead 
  Transport: usb 
  Adapter Identifier: usb.vmhba34 
  Target Identifier: usb.0:0 
  Adapter Transport Details: Unavailable or path is unclaimed 
  Target Transport Details: Unavailable or path is unclaimed 
  Maximum IO Size: 32768

 

[Beer]

Seems that this USB stick is unsupported and unrecognized by ESXi. Samsung Fit 32GB USB 3.0. I tried an ancient 2GB USB 2.0 stick I had lying around and got success messages. I formatted both flash drives in GParted as FAT32 and cleared all flags on both, old USB works, new USB doesn't.

With the instructions on that page, I was able to get a USB datastore up

2017-01-17T21:01:49.339Z cpu8:66003)VMK_PCI: 393: Device 0000:00:1d.0 name: vmhba33
2017-01-17T21:01:49.339Z cpu8:66003)DMA: 646: DMA Engine 'vmhba36' created using mapper 'DMANull'.
2017-01-17T21:01:49.339Z cpu8:66003)WARNING: ScsiPath: 7481: Adapter Invalid does not exist
2017-01-17T21:01:49.355Z cpu8:65609)ScsiUid: 279: Path 'vmhba36:C0:T0:L0' does not support VPD Device Id page.
2017-01-17T21:01:49.355Z cpu8:65609)ScsiScan: 1206: Path 'vmhba36:C0:T0:L0' : No standard UID: Failure. ANSI version 'SCSI-2' (0x2).
2017-01-17T21:01:49.358Z cpu8:65609)VMWARE SCSI Id: Could not get disk id for vmhba36:C0:T0:L0
2017-01-17T21:01:49.358Z cpu8:65609)WARNING: xpt_scsi_adapter_discover:1224: unable to find target 1: No connection
2017-01-17T21:01:49.363Z cpu8:65609)ScsiUid: 279: Path 'vmhba36:C0:T0:L0' does not support VPD Device Id page.
2017-01-17T21:01:49.365Z cpu8:65609)VMWARE SCSI Id: Could not get disk id for vmhba36:C0:T0:L0
2017-01-17T21:01:49.365Z cpu8:65609)StorageApdHandler: 977: APD Handle  Created with lock[StorageApd-0x4302ad371110]
2017-01-17T21:01:49.365Z cpu8:65609)ScsiEvents: 501: Event Subsystem: Device Events, Created!
2017-01-17T21:01:49.367Z cpu8:65609)VMWARE SCSI Id: Could not get disk id for vmhba36:C0:T0:L0
2017-01-17T21:01:49.367Z cpu8:65609)ScsiDeviceIO: 9120: Get VPD 80 Inquiry for device "mpx.vmhba36:C0:T0:L0" from Plugin "NMP" failed. Not supported
2017-01-17T21:01:49.367Z cpu8:65609)ScsiDeviceIO: 9130: Get VPD 83 Inquiry for device "mpx.vmhba36:C0:T0:L0" from Plugin "NMP" failed. Not supported
2017-01-17T21:01:49.367Z cpu8:65609)ScsiDeviceIO: 9167: Get VPD 86 Inquiry for device "mpx.vmhba36:C0:T0:L0" from Plugin "NMP" failed. Not supported
2017-01-17T21:01:49.369Z cpu8:65609)ScsiDeviceIO: 7851: Could not detect setting of QErr for device mpx.vmhba36:C0:T0:L0. Error Not supported.
2017-01-17T21:01:49.370Z cpu7:65576)ScsiDeviceIO: 2948: Cmd(0x439500a41000) 0x1a, CmdSN 0x94e from world 0 to dev "mpx.vmhba36:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2017-01-17T21:01:49.371Z cpu8:65609)ScsiDeviceIO: 8365: Could not detect setting of sitpua for device mpx.vmhba36:C0:T0:L0. Error Failure.
2017-01-17T21:01:49.458Z cpu8:65609)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 40, Parameter: 0x4302ad3b1b40, Registered!
2017-01-17T21:01:49.458Z cpu8:65609)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 200, Parameter: 0x4302ad3b1b40, Registered!
2017-01-17T21:01:49.458Z cpu8:65609)ScsiDevice: 4265: Successfully registered device "mpx.vmhba36:C0:T0:L0" from plugin "NMP" of type 0
2017-01-17T21:01:49.458Z cpu8:65609)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 180, Parameter: 0x43051028b0d0, Registered!

 

[Rand__]

Nice
But didnt you say you used the same as ESX installation media?
Can you create a datastore there? (Then you could simply install on a second stick and ignore/drop the esx partitions )