I'm with you that it SHOULD work..but some how it doesn't dunno what else to say, OmniOS: No, Solaris: Yes (irritating the **** out of me).I KNOW it works on Unix/Illumos as that's ALL I have EVER used in the past so I'm losing my mind here.
Please try Gea's update as well as a stripped/base OmniOS/ZoL config w/ nothing but a headless barebones (JEOS) setup that you can get to try to rule SOMETHING out. Simply replicating my test woule tell us SW/HW issues I'd assume.
Haven't been able to do much this week..getting pulled in 5 directions..of course will post progess as info comes along..I will gather my 'ohh hell no' notes up and see if I have any other tidbits of knowledge to share or get you ironed out.
If you want to use the removed drive on another machine as a pool you will need to use the zpool split command. If you just want to remove and not use same pool in different machine then you use detach.What is the correct procedure to remove 1 of my drives from the 3way mirror? (permanently)
ZFS Detach? Offline the drive?
2015-09-30T18:05:00.630Z: [APDCorrelator] 454725050623us: [vob.storage.apd.exit] Device or filesystem with identifier [93b0e02f-35d2f843] has exited the All Paths Down state.
2015-09-30T18:05:00.630Z: [APDCorrelator] 454725050996us: [esx.clear.storage.apd.exit] Device or filesystem with identifier [93b0e02f-35d2f843] has exited the All Paths Down state.
2015-09-30T18:05:00.630Z: [vmfsCorrelator] 454725050876us: [esx.problem.vmfs.nfs.server.restored] 192.168.20.5 /vaultpool01/vmbackup01 93b0e02f-35d2f843-0000-000000000000 vmbackup01
2015-09-30T18:05:13.631Z: [APDCorrelator] 454738051106us: [vob.storage.apd.start] Device or filesystem with identifier [93b0e02f-35d2f843] has entered the All Paths Down state.
2015-09-30T18:05:13.631Z: [APDCorrelator] 454738051328us: [esx.problem.storage.apd.start] Device or filesystem with identifier [93b0e02f-35d2f843] has entered the All Paths Down state.
2015-09-30T18:07:01.631Z: [vmfsCorrelator] 454846051270us: [esx.problem.vmfs.nfs.server.disconnect] 192.168.20.5 /vaultpool01/vmbackup01 93b0e02f-35d2f843-0000-000000000000 vmbackup01
2015-09-30T18:07:33.631Z: [APDCorrelator] 454878051572us: [vob.storage.apd.timeout] Device or filesystem with identifier [93b0e02f-35d2f843] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
Wow, sorry bro, very frustrating I am sure and I hate to do the poke in the eye but 'I'm rock solid over here'. I think I posted all relevant info in previous post and even spun up ZoL (ubuntu 14 LTS)/napp-it (latest as of a few weeks back and latest omni as well)...freaking strange.Well the datastore was up and down most of the day today when I checked the vobd.log.
I then ran a ubuntu iso and installed the nfs client:
$ sudo apt-get update
$ sudo apt-get install nfs-common
I had the console open from vsphere client and then connected the datastore from ubuntu which took a loooong time.
when it did connect I saw my datastore in esxi also come back.
Then triggered an APD by doing a esxcfg-nas -l from an ssh session into ESXi
the nfs datastore then also was not available from ubuntu so it's not just esxi that doesn't have access to the datastore.
( I tried creating an directory from ubuntu when the datastore was not available under esxi: )
I'll try the update to the latest OmniOS but have lost hope for NFS and OmniOS..