napp-in-one on dell T20 - my planned setup

[tnkrer]

Goal of this post is to collect all the information for myself in one place, get some insight and idea about any possible problems/gotchas I am not seeing, and possibly help another newbie in future.

What I want to achieve -
1. Serve photos, songs, media
2. Storage for TV recordings. (Windows Media Center will write to this storage)
3. upload and sync photos from phone from outside home network
4. document storage (instead of google drive) - so want to make it available outside home network
5. availability
6. backup (local backup, not considering offline backup at this point)
7. occasional VM (linux or windows) for any POC projects

Currently I have a windows box which is used to achieve some of this for last 5 years and I have been thinking of taking the jump into a real file server for some time now.

I looked at FreeNAS, installed it and found out that linux VM was not working very well on the virtuabBox jail. That was a deal-breaker for me. So considering going with napp-in-one. I have been using ESXi at work and it has worked very well for VMs.

Hardware -
Dell T20 with Xeon E3-1225v3, 4 GB ECC RAM (two 8 GB ECC sticks crucial CT2KIT102472BD160B to be added to get to 20GB )
2x 4TB WD RED drives used with passthrough to napp-in-one and setup in mirror RAID. When/If more space is needed another mirror pool will be added
1x 1TB Hitachi (came with the box) - for back up of photos, documents, songs. (All much less than 1 TB). Do not plan to back up TV recordings, if I lose it .. I will live with that.
1x 250GB ssd - ESXi datastore and host cache of 32GB
Dell Perc H310 (bought off ebay .. hopefully it fits in my case Fits nicely in the case, added it to the first slot after CPU)
- Can this cheaper card work? LSI Logic LSI00188 Logic 9200-8e? Is the problem that the ports are external instead of internal? or are there more problems?

I have installed ESXi 6.0 on a 16 GB usb drive. (It will stay on that drive)
I have set up another usb portable disk as datastore for ESXi - for testing only, moved datastore to ssd
I tested making AHCI as passthrough to make the disks available to napp-in-one
(thanks to this post)

While this will work, I think I should get the dedicated HBA card, so that I can move datastore to 1TB disk inside the box. (I think that will be more reliable than usb portable disk). It will also be more flexible if/when I want to expand capabilities. Dell Perc H310 seems to be the cheapest and most tested card. It will need the cable that goes from SAS to SATA (mini-SAS to SATA breakout cable), right?

Any obvious issues with this plan?

I will update issues I faced and resolutions here
Unavailable memory on ESXi when using passthrough (3GB out of 4 GB)
napp it share permissions, so that only authorized users can see the share
napp it (or zfs) share permissions how to and open questions

 

[gea]

The T20 is a fine and cheap piece of server technology
For a storage system, main restriction is that you can only use 4 disks internally

A typical AiO setup would be
- boot ESXi (from usb or sata disk) and use the sata disk as a local datastore (50GB+)
Put napp-it in a very base config onto and nothing else, virtualize all other OS/services on ZFS
You can use an USB disk as a datastore in ESXi 6.0u2 via the new webconsole but I would always prefer Sata/AHCI.

- add an LSI HBA in pass-through mode for ZFS storage
You can use the 9200-8e but need a lot of cabling and adapters. The 9200-8e is perfect to connect an external SAS Jbod. Internally I would prefer an adapter with internal ports.

- use a disk base ZFS pool as ESXi datastore via NFS or for general use via SMB
- opt create a second SSD only pool fromfor VMs (much higher iops, single disk or mirror).

 

[Bob T Penguin]

Might I suggest Omnios on the baremetal with Napp-it as the storage front end. Then use the inbuilt KVM hypervisor for virtual machines. I use this set up and have a Linux VM acting as an internet gateway that runs OpenVPN. I then VPN into my home network to gain access to the SMB shares etc.

 

[tnkrer]

Over the weekend received my dell Perc H310 card and cross-flashed it successfully thanks to this techmattr blog - SAS HBA Crossflashing ..

Gea, Thanks for your comments. I do not expect to outgrow 4 disks anytime soon.

A typical AiO setup would be
- boot ESXi (from usb or sata disk) and use the sata disk as a local datastore (50GB+)

booting ESXi from USB and using SATA disk as local datastore

Put napp-it in a very base config onto and nothing else, virtualize all other OS/services on ZFS

I am using the template that you have and using SATA disk as the datastore for that VM. Are you saying that I should virtualize on to this napp it appliance and not directly on ESXi? what virtualization do I use on napp it?

Bob, Thanks for introducing the idea of running OmniOS instead of ESXi. I read through bunch of posts talking about this approach. At the end, I decided that I am already comfortable with ESXi. Also, gea provides canned napp it template. So my installation is very simple for this setup. So I will test how this set up works for me. If it is sufficient for my purposes, I will stay with it. If it gives problems, I will look at OmniOS + KVM.

 

[gea]

You can virtualize
- below all systems with a type-1 hypervisor (ESXi)

main advantage
easy and fastest reinstall, smallest overhead, all systems are equal, best of all support for operating systems and pass-through

- within an OS ex with zones or containers
main advantage
lower resource needs if you virtualize same or similar systems as you can share system resources

- on top of an os ex virtualbox
use case: rare use or a VM with lower performance or reliability needs

 

[tnkrer]

They say a picture is worth thousand words .. so here is my set up. I think it matches the first bullet that gea has in above post

 

[dragonme]

nice diagram...

I am running a similar setup... 8g flashdrive since anything more is a waste I am told...

I am also hacking the esxi kernel to boot nappit off of a usb2.0 2.5 hard drive enclosure attached to an internal header so I could pass through my onboard sata controller as well as my LSI but I am going to have to scrap that ... it requires the usbarbitrator to be shut down and I will need to pass a usb device to a VM so I am shit out of luck

I am still getting some strange performance issues but I am going to reinstall napp it when moving it to a proper sata drive

I am not sure if long term I will be happy with the setup...

napp-it SMB and mac OSX dont seem to get along well speed wise or ACL wise and using it as storage for a osx server VM is also holding me up... I did find that passing a vmdk that is on a zfs dataset being shared to esxi is 5x faster than using smb from osx to the same drive... on an internal esxi only vswitch.. bizarre...

 

[abundantmussel]

I picked up a T20 two weeks ago, stuck in 32GB and a H200, 4 port NIC, 4 SSDs in RAID10 and 2 500GB spinners and a single 128GB SSD for OS. I love the size and quietness of this box. Good luck with yours.

Sent from my Nexus 5 using Tapatalk