ESXi 6.0 NFS with OmniOS Unstable - consistent APD on VM power off
- Thread starter uto
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
VMware vSphere 5: Private Cloud Computing, Server and Data Center Virtualization
For Example
ESXi 5.5 Update 2d ISO image (Includes VMware Tools)
Boot your server with this image in order to install or upgrade to ESXi. This ESXi image includes VMware Tools. NOTE: ESXi requires 64-bit capable servers for installation and execution. Please refer to the VMware Compatibility Guide for a list of qualified server hardware.
MD5SUM: 1e0e128e678af54657e6bd3b5bf5f124
SHA1SUM: e9d24730d6c9edbc7b273745428a9871836084dd
You can run it as powershell get-filehash -algorithm sha1 <file_to_check> and then compare them
For Example
ESXi 5.5 Update 2d ISO image (Includes VMware Tools)
Boot your server with this image in order to install or upgrade to ESXi. This ESXi image includes VMware Tools. NOTE: ESXi requires 64-bit capable servers for installation and execution. Please refer to the VMware Compatibility Guide for a list of qualified server hardware.
MD5SUM: 1e0e128e678af54657e6bd3b5bf5f124
SHA1SUM: e9d24730d6c9edbc7b273745428a9871836084dd
You can run it as powershell get-filehash -algorithm sha1 <file_to_check> and then compare them
Ahh oke checked that..!
Still waiting for Solaris to Finish...guy in hardforum wasn't kidding with E1000 ... downloading is crazy slow and cpu usage in the back..
@TechIsCool @whitey @gea,
Which 5.5 build are u guys on?
reference:
VMware ESXi Release and Build Number History | Virten.net
Which 5.5 build are u guys on?
reference:
VMware ESXi Release and Build Number History | Virten.net
More info for my System is contained here.
https://forums.servethehome.com/index.php?threads/needle-in-a-hay-stack-nfs-r-w-latency-esxi.5847/
OmniOS 5.11 omnios-7648372 July 2015
I have run 5.1 u3 (2323236) which was stable for 3 years
6.0b (2494585, 2615704, 2715440, 2809209) Current version running since basically Veeam Supported it. I jumped day 1 and had to roll back and I have patched up every release.
Oke, status update for today..
After reading that you guys are on ESXi 6.0 I fired up Rufus to make me a ESXI 6 install USB thumbdrive. (I have a spare usb thumdrive so I should be able to switch between 6 and 5.5u2 to see what happens)
Before I shut my host down I looked at my Solaris VM I build yesterday night...I thought what the ****, installing updates and Napp-IT took forever (don't know why), might as well give it a shot.
shut down OI, and connected the M1015 to the Solaris VM.
Tried to import the test pool..didn't work, Solaris didn't like the fact that is was created by Omni-OS. (Incompatible message).
So after destroying the OmniOS pool, I :
-added the following lines to /kernel/drv/e1000g.conf
#tcp offload disable
tx_hcksum_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
lso_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
(read on the fora that this is needed for E1000) and Gea puts it in his image although I wasn't able to find it in his latest appliance but I might be wrong about that.
Is this setting still necessary with current OmnisOS/Solaris versions when using E1000?
- Created a pool using Napp-IT under Solaris.
- Created a NFS share
- Mounted it under ESXi
So everything the same..no All Paths Down error..so far..
- Created a Windows 8 VM, started it...no All Paths Down error..so far..
- Installed Windows 8, installed vmware tools, still no issues...(That is about as far as I got with OmniOS once!)
- Went into the datastore and copy pasted the Windows 8 vmdk into a new folder...........still no issues,
- Created a new VM based on the copy and booted the 2nd VM, still solid
- Used Putty to do LS commands in the datastore (was often (not always) enough to make the datastore under OmniOS go APD for a certain amount of time before it would come back)
- Left the VM's running for a couple of hours came back and checked the vobd.log under ESXi...no errors!
- Rebooted the Solaris VM (Datatore under ESXi /OmniOS combo also didn't like that after my initial success of being able to create and install a VM.)
- Had to wait a couple of minutes but the datastore came back on its own and I was able to start both VM's without issue..
This is by no means scientific testing but so far I hadn't gotten this far without running into problems with the OmniOS/ESXi combo. Have to do additional testing to see if it stays stable with Solaris. So far haven't had no problem re-creating problems as soon as I started to use OmniOS
So yeah...ummm it seems to work with Solaris 11. Still have to do a lot of reading on Solaris 11 on how it differs from OmniOS because I wasn't planning on using Solaris wanted OmniOS because Gea uses it for his Napp-IT work (plus all the other cool kids are using OmniOS
)
Next step will be to install ESXi 6 on the spare usb thumbdrive and reimport Gea's NappIT appliance to see how that works out.
After reading that you guys are on ESXi 6.0 I fired up Rufus to make me a ESXI 6 install USB thumbdrive. (I have a spare usb thumdrive so I should be able to switch between 6 and 5.5u2 to see what happens)
Before I shut my host down I looked at my Solaris VM I build yesterday night...I thought what the ****, installing updates and Napp-IT took forever (don't know why), might as well give it a shot.
shut down OI, and connected the M1015 to the Solaris VM.
Tried to import the test pool..didn't work, Solaris didn't like the fact that is was created by Omni-OS. (Incompatible message).
So after destroying the OmniOS pool, I :
-added the following lines to /kernel/drv/e1000g.conf
#tcp offload disable
tx_hcksum_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
lso_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
(read on the fora that this is needed for E1000) and Gea puts it in his image although I wasn't able to find it in his latest appliance but I might be wrong about that.
Is this setting still necessary with current OmnisOS/Solaris versions when using E1000?
- Created a pool using Napp-IT under Solaris.
- Created a NFS share
- Mounted it under ESXi
So everything the same..no All Paths Down error..so far..
- Created a Windows 8 VM, started it...no All Paths Down error..so far..
- Installed Windows 8, installed vmware tools, still no issues...(That is about as far as I got with OmniOS once!)
- Went into the datastore and copy pasted the Windows 8 vmdk into a new folder...........still no issues,
- Created a new VM based on the copy and booted the 2nd VM, still solid
- Used Putty to do LS commands in the datastore (was often (not always) enough to make the datastore under OmniOS go APD for a certain amount of time before it would come back)
- Left the VM's running for a couple of hours came back and checked the vobd.log under ESXi...no errors!
- Rebooted the Solaris VM (Datatore under ESXi /OmniOS combo also didn't like that after my initial success of being able to create and install a VM.)
- Had to wait a couple of minutes but the datastore came back on its own and I was able to start both VM's without issue..
This is by no means scientific testing but so far I hadn't gotten this far without running into problems with the OmniOS/ESXi combo. Have to do additional testing to see if it stays stable with Solaris. So far haven't had no problem re-creating problems as soon as I started to use OmniOS
So yeah...ummm it seems to work with Solaris 11. Still have to do a lot of reading on Solaris 11 on how it differs from OmniOS because I wasn't planning on using Solaris wanted OmniOS because Gea uses it for his Napp-IT work (plus all the other cool kids are using OmniOS
)Next step will be to install ESXi 6 on the spare usb thumbdrive and reimport Gea's NappIT appliance to see how that works out.
Last edited:
Your import problem seems quite strange, I would suggest that you only check results for a clean install (for example on pool build from a new testdisk) and a create new/ copy over old VMs.
about ESXi
- the e1000 driver modification was only needed in ESXi 5.5 (initial release)
- ESXi 5.5 U1 was buggy related NFS
- ESXi 5.5 U2 is what I am using in my production systems with NFS and newest OmniOS 151014
- ESXi 6.0 initial release: testings only, require newer vmware tools (included in napp-it appliance 15b)
about Solaris vs OmniOS
Ilumos (base of NexentaStor/OmniOS/OI/SmartOS) is a more or less fork of the last free Oracle Solaris 11 Express without encryption as Oracle closed Solaris. Since then the two evolved in two different and incompatible developmemt lines, Solaris 11.1-11.3 with ZFS v29-37 and Illumos with OpenZFS v5000 and features flags.
As they are incompatible regarding ZFS its a one or the other. A switch means a re-create pools. With current Solaris 11.3, Solaris is a strong option as it includes ZFS encryption, LZ4 and SMB2.1. But it is free only for demo and development use while OmniOS is completely free but with a commercial support option.
about ESXi
- the e1000 driver modification was only needed in ESXi 5.5 (initial release)
- ESXi 5.5 U1 was buggy related NFS
- ESXi 5.5 U2 is what I am using in my production systems with NFS and newest OmniOS 151014
- ESXi 6.0 initial release: testings only, require newer vmware tools (included in napp-it appliance 15b)
about Solaris vs OmniOS
Ilumos (base of NexentaStor/OmniOS/OI/SmartOS) is a more or less fork of the last free Oracle Solaris 11 Express without encryption as Oracle closed Solaris. Since then the two evolved in two different and incompatible developmemt lines, Solaris 11.1-11.3 with ZFS v29-37 and Illumos with OpenZFS v5000 and features flags.
As they are incompatible regarding ZFS its a one or the other. A switch means a re-create pools. With current Solaris 11.3, Solaris is a strong option as it includes ZFS encryption, LZ4 and SMB2.1. But it is free only for demo and development use while OmniOS is completely free but with a commercial support option.
Hi Gea,
All the tests I've been doing, have been from a clean disk and pool using a 160 GB disk that I had laying around. (so not even testing my current disks/pools)
When I created a pool using Napp-IT under Solaris I thought I saw the option v28 (not v29-37)
would using this option keep the pools compatible between Solaris and OmniOS?
LZ4 are both possible in OmniOS and Solaris right? (but probably only if you "choose" 1 over the other and go beyond v28?)
As for licenses go..for my usecase as long as Solaris doesn't stop after 30 days, I'm good with that..
Like I said I really want to use OmniOS because you use it (true story) but if I can't get it to run stable I might have to look at Solaris...
Hopefully ESXi 6 will be more stable for me running Omni-OS.. (will test as soon as I can find the time )
Also this endeavor has caused me to reconsider my storage backup solution.
When using the Napp-IT ESXi hot-snap solution is it possible to backup VM's from a local SSD to a ZFS filesystem when it's running on the same AIO machine?
Or should the source always be an NFS share?
All the tests I've been doing, have been from a clean disk and pool using a 160 GB disk that I had laying around. (so not even testing my current disks/pools)
When I created a pool using Napp-IT under Solaris I thought I saw the option v28 (not v29-37)
would using this option keep the pools compatible between Solaris and OmniOS?
LZ4 are both possible in OmniOS and Solaris right? (but probably only if you "choose" 1 over the other and go beyond v28?)
As for licenses go..for my usecase as long as Solaris doesn't stop after 30 days, I'm good with that..
Like I said I really want to use OmniOS because you use it (true story) but if I can't get it to run stable I might have to look at Solaris...
Hopefully ESXi 6 will be more stable for me running Omni-OS.. (will test as soon as I can find the time )
Also this endeavor has caused me to reconsider my storage backup solution.
When using the Napp-IT ESXi hot-snap solution is it possible to backup VM's from a local SSD to a ZFS filesystem when it's running on the same AIO machine?
Or should the source always be an NFS share?
Last edited:
@nostradamus99 If you don't have the free version of ESXi you can use Veeam for the Backup solution and use powershell to automate it for free. If you do have the Free version of ESXi you could snapshot using Napp-It and then zfs send the snapshot to a secondary location be it on the AIO with a External Disk or a Desktop running ZFS
Just an info
The napp-it 15b appliance is on OmniOS build 170cea2 (April). This is what I am using on my ESXi machines as well.
There is a report at Hardforum from a user who had NFS problems on ESXi with the newer OmniOS july build 7648372.
The NFS problems were not the case on his machine with the april build (on 5.5U2 and 6.0.0b)
[H]ard|Forum - View Single Post - OpenSolaris derived ZFS NAS/ SAN (Nexenta*, OpenIndiana, Solaris Express)
The napp-it 15b appliance is on OmniOS build 170cea2 (April). This is what I am using on my ESXi machines as well.
There is a report at Hardforum from a user who had NFS problems on ESXi with the newer OmniOS july build 7648372.
The NFS problems were not the case on his machine with the april build (on 5.5U2 and 6.0.0b)
[H]ard|Forum - View Single Post - OpenSolaris derived ZFS NAS/ SAN (Nexenta*, OpenIndiana, Solaris Express)
Thanks Gea, that must be the case for me also no doubt about it. (will deploy a new vm using your appliance without upgrading to doublecheck)
You mentioned earlier that:
Feature Flags - OpenZFS
Been trying to find the pro's+cons of ZFS on Solaris and OpenZFS on OmniOS
What would be your advice, keep my pools at v28 or just choose Solaris or OmniOS and upgrade my pools after choosing either Solaris or OmniOS?
You mentioned earlier that:
Doesn't OmniOS support LZ4 compressions? (LZ4 Compression - illumos - illumos wiki )
Feature Flags - OpenZFS
Been trying to find the pro's+cons of ZFS on Solaris and OpenZFS on OmniOS
What would be your advice, keep my pools at v28 or just choose Solaris or OmniOS and upgrade my pools after choosing either Solaris or OmniOS?
LZ4 is included in OmniOS for quite a long time but it is new for Solaris 11.3
Unless you are decided, V28/5 is a good idea but it lacks all the new OmniOS
or Solaris features. So at some point you should upgrade your pool.
If you want to switch afterwards, you must use a filecopy to transfer the data.
Unless you are decided, V28/5 is a good idea but it lacks all the new OmniOS
or Solaris features. So at some point you should upgrade your pool.
If you want to switch afterwards, you must use a filecopy to transfer the data.
tested today: (haven't tested Gea's latest creation but if nothing changed except the ovf that doesn't change anything.)
gea appliance 1.5b stock: esxi 5.5u2 latest patch -> APD Down error after starting a newly created VM
gea appliance 1.5b stock: esxi 6.0 latest (build 2809209) -> APD Down error after starting a newly created VM
solaris 11.2: esxi 6.0 latest (build 2809209) -> no problems creating, installing en booting a new Windows 8 VM
Current OI VM: after all the testing, reconnected the M1015 back to my OI and booted it. So far running stable under ESXi 6 (latest)
Used Hardware (running solid past 3 years):
MSI Z77MA-G45
Core i5-3550
32GB RAM
Intel 82571EB Network cards
M1015 flashed to IT using P19 passed through to VM in question.
VM's used:
Gea appliance 1.5b VM with 2 E1000 NICS (1 for CIFS, 1 for NFS)
Solaris 11.2 VM (bare install) with 2 E1000 NICS (1 for CIFS, 1 for NFS)
Within every test:
- Created new pool using empty harddrive using the Nappit menu. (version set to default)
- Created new ZFS filesystem and shared this through NFS
- Connected the new datastore to ESXi using NFS
- Create a new Windows 8 VM 32bit on the new NFS share. (version 8 hardware)
- Give the VM an installation iso that is located on local datastore.
- Power on the VM
- Open the Console from vsphere client
- Check the ESXIi vmkernel.log and vobd.log
- After installation of Windows 8 (if succesful) power down installed VM
- Power down the OmniOS or Solaris VM
- Wait for 140 sec and message in vobd.log :
[esx.problem.storage.apd.timeout] Device or filesystem with identifier [6178bcc3-5b14787a] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
- Power on the OmniOS or Solaris VM en see if the NFS share comes back
- Power on the installed Windows 8 VM
That has been my testing so far.
I'm going to drop NFS to store my VM's.
Switching to a local SSD and making backup's to a CIFS Nappit pool every day.
Still undecided on which route to go but I can't get NFS stable with OmniOS so I'll need to switch to Solaris. (dunno what the differences between the NFS implementations are but there are most certainly different beasts )
I've ordered an Microserver Gen8 so that I can test some VMWare and backup stuff for my work. If I can find the time maybe I can test OmniOS on that new server.
gea appliance 1.5b stock: esxi 5.5u2 latest patch -> APD Down error after starting a newly created VM
gea appliance 1.5b stock: esxi 6.0 latest (build 2809209) -> APD Down error after starting a newly created VM
solaris 11.2: esxi 6.0 latest (build 2809209) -> no problems creating, installing en booting a new Windows 8 VM
Current OI VM: after all the testing, reconnected the M1015 back to my OI and booted it. So far running stable under ESXi 6 (latest)
Used Hardware (running solid past 3 years):
MSI Z77MA-G45
Core i5-3550
32GB RAM
Intel 82571EB Network cards
M1015 flashed to IT using P19 passed through to VM in question.
VM's used:
Gea appliance 1.5b VM with 2 E1000 NICS (1 for CIFS, 1 for NFS)
Solaris 11.2 VM (bare install) with 2 E1000 NICS (1 for CIFS, 1 for NFS)
Within every test:
- Created new pool using empty harddrive using the Nappit menu. (version set to default)
- Created new ZFS filesystem and shared this through NFS
- Connected the new datastore to ESXi using NFS
- Create a new Windows 8 VM 32bit on the new NFS share. (version 8 hardware)
- Give the VM an installation iso that is located on local datastore.
- Power on the VM
- Open the Console from vsphere client
- Check the ESXIi vmkernel.log and vobd.log
- After installation of Windows 8 (if succesful) power down installed VM
- Power down the OmniOS or Solaris VM
- Wait for 140 sec and message in vobd.log :
[esx.problem.storage.apd.timeout] Device or filesystem with identifier [6178bcc3-5b14787a] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
- Power on the OmniOS or Solaris VM en see if the NFS share comes back
- Power on the installed Windows 8 VM
That has been my testing so far.
I'm going to drop NFS to store my VM's.
Switching to a local SSD and making backup's to a CIFS Nappit pool every day.
Still undecided on which route to go but I can't get NFS stable with OmniOS so I'll need to switch to Solaris. (dunno what the differences between the NFS implementations are but there are most certainly different beasts )
I've ordered an Microserver Gen8 so that I can test some VMWare and backup stuff for my work. If I can find the time maybe I can test OmniOS on that new server.
Last edited: