1. Introduction
The goal is to up an All-In-One server with ZFS storage, SMB shares for media and files, NFS Shares for VM Storage
Hardware Build is detailed at Zeus V2 : U-NAS NSC-810A | X10SL7-F | E3-1265 V3.
Storage :
SanDisk UltraFit 8GB - Esxi Install location
SanDisk UltraFit 128GB - Napp-It Install location
8x 8 TB WD Reds - Bulk storage
2x 400GB DC S3500 - VM storage
1x DC S3700 - SLOG
The motherboard has 1x PCI-E 3.0 x8 (in x16) and 1x PCI-E 2.0 x4 (in x8). I needed a 10Gbe connection and added a single port Mellanox Connect-X2. With the desired storage configuration 8x8TB for bulk and mirrored 400GB for VM, I do not have enough ports on the onboard HBA to handle this. I tried adding an additional PCIe HBA but the U-NAS case thermals are poor and the system temperature remained high.
I will be using a USB thumb drive as a VM Datastore to store the SAN VM and pass through the onboard SATA ports to the SAN VM.
The base hypervisor is Esxi 6.5. Napp-IT will provide storage services.
A friend wanted me to give him instructions that he can follow to implement a similar setup and I have seen multiple questions about this in the Forums. So I decided to just document the software installation in detail and publish it as a way of giving back to the community.
This is intended to help a beginner like me to setup an All-In-One server. These instructions are what I put together after researching for a long time and after several iterations to make things work properly. Let me know if there are any errata and I'll correct.
Without more ado here you go.
2 ESXi Installation
Download the esxi installer from VMWare and register for a free esxi license. I used SuperMicro iKVM to mount the installer iso and setup ESXi on the system.
2.1 ESXi Initial Network Configuration
2.2 Enable SSH
ESXi by default disables SSH and if you enable it from the host menu, it gets disabled every reboot. Since I need SSH access during setup, I will enable it via the services screen to have it enabled during boot.
Method 1 – Enable from the host menu
SSH gets disabled after every reboot in this method.
Method 2 – Permanently enable SSH
I use a Windows AD and have all systems sync time with it. In System -> Time & Date, Edit Settings and set the startup policy for the NTP service as well as the NTP server details.
2.4 Resolve Driver Issues
The native driver (vmw-ahci) for the Lynx Point controller has serious performance issues. Disable it to use the sata_ahci driver.
2.8 Configure the ESXi Networking
The Host has 2 onboard NICs
NIC1 - VM Network and Management Network
NIC2 - Internal Host Only network – Napp-IT Storage
The VM Network and Management Network are already setup by default. No change was made except to rename the items. Below are the steps to setup the internal host only network for Napp-IT SAN.
3 Install and Configure Napp-IT
@gea has provided excellent instructions for the setup at napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux :Downloads .
3.1 Napp-IT VM Setup in ESXi
3.2 Napp-IT appliance configuration
This section is covered in detail in @gea ’s guide at napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux :Downloads
In addition, set the ip address of the second nic in Napp-It the 172.16.210.0/24 range.
4 Add NFS Datastore in ESXi
Refer to Post #2 for instructions. (Hit the image upload limit in this post)
You can now use esxi with the shared nfs datastore.
The goal is to up an All-In-One server with ZFS storage, SMB shares for media and files, NFS Shares for VM Storage
Hardware Build is detailed at Zeus V2 : U-NAS NSC-810A | X10SL7-F | E3-1265 V3.
Storage :
SanDisk UltraFit 8GB - Esxi Install location
SanDisk UltraFit 128GB - Napp-It Install location
8x 8 TB WD Reds - Bulk storage
2x 400GB DC S3500 - VM storage
1x DC S3700 - SLOG
The motherboard has 1x PCI-E 3.0 x8 (in x16) and 1x PCI-E 2.0 x4 (in x8). I needed a 10Gbe connection and added a single port Mellanox Connect-X2. With the desired storage configuration 8x8TB for bulk and mirrored 400GB for VM, I do not have enough ports on the onboard HBA to handle this. I tried adding an additional PCIe HBA but the U-NAS case thermals are poor and the system temperature remained high.
I will be using a USB thumb drive as a VM Datastore to store the SAN VM and pass through the onboard SATA ports to the SAN VM.
The base hypervisor is Esxi 6.5. Napp-IT will provide storage services.
A friend wanted me to give him instructions that he can follow to implement a similar setup and I have seen multiple questions about this in the Forums. So I decided to just document the software installation in detail and publish it as a way of giving back to the community.
This is intended to help a beginner like me to setup an All-In-One server. These instructions are what I put together after researching for a long time and after several iterations to make things work properly. Let me know if there are any errata and I'll correct.
Without more ado here you go.
2 ESXi Installation
Download the esxi installer from VMWare and register for a free esxi license. I used SuperMicro iKVM to mount the installer iso and setup ESXi on the system.
2.1 ESXi Initial Network Configuration
- Login to the console
- Configure Management Network.
- Select NICs to use for Management Network.
- Setup IPv4 Configuration (I am using Static IP).
- Setup IPv6 Configuration (I disable it).
- Configure hostname and DNS settings (I’ve already entered a static entry in my windows DNS server.
- Apply the changes and reboot the host.
- After reboot, Login to the host.
2.2 Enable SSH
ESXi by default disables SSH and if you enable it from the host menu, it gets disabled every reboot. Since I need SSH access during setup, I will enable it via the services screen to have it enabled during boot.
Method 1 – Enable from the host menu
SSH gets disabled after every reboot in this method.
Method 2 – Permanently enable SSH
- Enable and set to auto start in the services screen
- Disable SSH Warning : ESXi will display a warning when SSH is enabled. To hide the warning, In Manage -> System -> Advanced Configuration, Set the value of UserVars.SuppressShellWarning to 1.
I use a Windows AD and have all systems sync time with it. In System -> Time & Date, Edit Settings and set the startup policy for the NTP service as well as the NTP server details.
2.4 Resolve Driver Issues
The native driver (vmw-ahci) for the Lynx Point controller has serious performance issues. Disable it to use the sata_ahci driver.
- Login to the host using putty
- Enter Command to verify the available drivers
Code:esxcli software vib list | grep ahci
- Disable the driver and reboot
Code:esxcli system module set --enabled=false --module="vmw_ahci" reboot
- The Lynx Point Controller cannot be passed through to a VM by default.
- You need to add the following line to /etc/vmware/passthru.map to enable pass through. Add it to the end of the file. DO NOT EDIT any other line in the file.
Code:# INTEL Lynx Point AHCI 8086 8c02 d3d0 false
- Use vi from terminal/putty to make the edit. I used WinSCP to connect to the host and used notepad to make the changes.
- Reboot the host.
- Select the Lynx Point Controller and LSI HBA and Toggle Passthrough.
- Reboot the host to finalize the changes.
Code:
Note : I need an 8 drive storage array + 1 drive for SLOG and 2 drives for mirrored VM data store. That is a total of 11 drives. The X10SL7-F has 8 ports via the onboard HBA. I could have added another PCIe HBA and used that for the remaining 3 drives and used the SATA ports for the Napp-IT VM datastore. In fact I tried that out. But that did not work for me due to 2 reasons.
1. I needed to add a 10G NIC and the X10SL7-F has only one PCIEx8 slot. The other is a PCIEx2 slot without enough bandwidth.
2. The thermals for the U-NAS case are not too great and I am already stuffing 3 additional SSDs there.
That’s why I decided to use a USB data store. It is not supported and is a hacked way of doing things and may break in a future ESXi update. That’s an acceptable risk for me as this is a home server and I have another box running as a warm spare. I will just need to shut this down and change a couple of DNS entries to “fail over to the other”. Probably a few minutes down time which is OK with me.
You can skip this section if you are using the SATA ports for the Napp-IT VM datastore.
- Follow the steps listed in the website referenced here to setup the second USB drive as a datastore.
USB Devices as VMFS Datastore in vSphere ESXi 6.5
Code:[root@esxm1:~] /etc/init.d/usbarbitrator stop UsbUtil: Releasing all USB adapters to VMkernel watchdog-usbarbitrator: Terminating watchdog process with PID 66683 usbarbitrator stopped [root@esxm1:~] chkconfig usbarbitrator off [root@esxm1:~] ls /dev/disks/ naa.2020030102060804 [root@esxm1:~] [root@esxm1:~] partedUtil mklabel /dev/disks/naa.2020030102060804 gpt [root@esxm1:~] partedUtil getptbl /dev/disks/naa.2020030102060804 gpt 4164 255 63 66895872 [root@esxm1:~] eval expr $(partedUtil getptbl /dev/disks/naa.2020030102060804 | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1 66894659 [root@esxm1:~] partedUtil setptbl /dev/disks/naa.2020030102060804 gpt "1 2048 66894659 AA31E02A400F11DB959 0000C2911D1B8 0" gpt 0 0 0 0 1 2048 66894659 AA31E02A400F11DB9590000C2911D1B8 0 [root@esxm1:~] vmkfstools -C vmfs6 -S USB-Datastore /dev/disks/naa.2020030102060804:1 create fs deviceName:'/dev/disks/naa.2020030102060804:1', fsShortName:'vmfs6', fsName:esxm1-usb-01' deviceFullPath:/dev/disks/naa.2020030102060804:1 deviceFile:naa.2020030102060804:1 ATS on device /dev/disks/naa.2020030102060804:1: not supported
- After the above steps are complete, you will see USB drive as a storage device in ESXi.
- In this host, this will be the only local data store and will be used only for Napp-IT VMDK and nothing else.
2.8 Configure the ESXi Networking
The Host has 2 onboard NICs
NIC1 - VM Network and Management Network
NIC2 - Internal Host Only network – Napp-IT Storage
The VM Network and Management Network are already setup by default. No change was made except to rename the items. Below are the steps to setup the internal host only network for Napp-IT SAN.
- Add a new Standard Virtual Switch – Don’t add an uplink
- Add a new VMKernel NIC for the internal storage network. A corresponding port group will also be created. Choose a VLAN ID (I'm using 210). Setup a static IP address. Do not select any of the services.
- Add a new Port Group for the storage network. Use the same VLAN as above.
- For the Napp-It VM, you will add a second NIC connected to the storage network and setup an IP address in the 172.16.210.0/24 network in VLAN 210
3 Install and Configure Napp-IT
@gea has provided excellent instructions for the setup at napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux :Downloads .
3.1 Napp-IT VM Setup in ESXi
- Download your flavor of Napp-IT from napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux :Downloads
- Deploy the Napp-IT OVA to ESXi to the USB Data store you created. Select VM Network when prompted. Do not select poweron after deploy.
- Click Finish and wait for the appliance to be deployed. You can watch the progress in the Recent Tasks pane. This will be slow and will take some time as you are using a USB data store. If using an SSD data store, it usually takes about 2-3 mins for me.
- Edit the VM Settings
- Set the amount of RAM you want to allocate and Reserve that RAM allocation.
- Connect the second NIC to the Storage Network Port Group
- Add the SATA controller and the HBA as PCI passthrough devices
3.2 Napp-IT appliance configuration
This section is covered in detail in @gea ’s guide at napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux :Downloads
In addition, set the ip address of the second nic in Napp-It the 172.16.210.0/24 range.
4 Add NFS Datastore in ESXi
Refer to Post #2 for instructions. (Hit the image upload limit in this post)
You can now use esxi with the shared nfs datastore.
Last edited: