All-In-One with SmartOS/OI on mirrorred ZFS bootdisks

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
correction: This is for OmniOS not SmartOS, SmartOS is not supported


I have improved my all-in-one concept to support mirrorred ZFS boot-disks
This gives a better uptime and allows to update ESXi independently from Omni/OI

basic steps:
- you need two Sata disks for ESXi (optionally an additional USB stick for ESXi)
- install ESXi on first disk or on a USB stick as usual
- install vsphere on a PC
- use vsphere to create a datastore on both disk
- use vsphere to create a virtual disk (20 GB+) on both datastores
- Install Omni/OI on the disk on the first datastore (which is first in boot-order)
- Install napp-it, connect via browser and http://ip:81
- Goto menu Disk - mirror bootdisk and mirror rpool to your second disk

optionally
- Edit ESXi VM-settings to modify Bios of this VM: setup boot order to boot from both disks, with second first if ESXi is on first disk

If you need to use a second disk with a different ESXi install, it does not affect Omni/OI beside an optional remirror
If you use an USB stick for ESXi, you have a fully independant mirrorred Storage VM

see
http://napp-it.org/manuals/all-in-one.html
 
Last edited:

geppi

New Member
Jan 24, 2013
11
2
3
I did setup my all-in-one with a mirrored rpool for the OmniOS fileserver manually some time ago.
To prevent the OmniOS VM from freezing in case of a disk failure I did enter the following configuration parameters in the .vmx file (can also be done via the vSphere Client GUI):

scsi0.returnNoConnectDuringAPD True
scsi0.returnBusyOnNoConnectStatus False

This was motivated by the following blog post:

http://www.vmdamentals.com/?p=919

and the VMware vSphere documentation here:

http://pubs.vmware.com/vsphere-50/i...UID-F78EB579-B11C-4E65-9EE3-145888A005F6.html

I can fail either of the two disks (just eject them from their hot swap bays) and the OmniOS VM keeps running with a degraded rpool.

However, I found it a little tricky to reestablish the mirror since ESXi is not happy about the missing datastore. There are two scenarios of which the more complicated one is the failure of the disk with the datastore that holds the virtual machine configuration files. Therefore first on the failure of the other disk.

Before I could use a new disk (in fact I used the old one just wiped) I had to shutdown the OmniOS VM to get rid of the unaccessable datastore zombie in the ESXi host configuration.
As long as the VM was running all actions on the datastore zombie failed with "device busy" or other messages which indicated that this zombie datastore was in use by the OmniOS VM's virtual disk.

I tried to hot remove the virtual disk on the failed datastore from the OmniOS VM settings but unfortunately you cannot hot remove a virtual disk from a running VM when PCI passthrough is configured for this same VM. Therefore I had to shutdown the OmniOS VM. But at least a clean shutdown is possible !

With the OmniOS VM shut down it was now possible to get rid of the failed datastore via "Rescan All" in the vSphere Client. Thereafter I inserted the new disk and configured it as a new second datastore.

Before powering up the OmniOS VM again I had to remove the failed virtual disk from it's settings and create a new one on the new datastore. The rest was done in OmniOS by first partitioning the virtual disk for use as a rpool device, resilvering the pool with "zpool replace" and finally installing GRUB.

If I had to deal with the disk that holds the datastore with the VM configuration files, I also had to remove the VM zombie entry in the ESXi inventory. Before doing this I loged into the OmniOS VM to properly shut it down. Then after replacing the failed disk and configuring a new datastore on the replacement disk I loged into the ESXi host and copied the VM configuration files from the disk that had survived to the new datastore into a directory with the name of the VM.
( Haha, what a wise decision it was to copy those VM configuration files to the second datastore before, when all was still in good shape ;-)

Using the vSphere Client datastore browser I did add the VM back to the ESXi inventory, removed the old first virtual disk from the settings and added a new one on the new datastore. After booting from the survived virtual disk the procedure to reestablish the rpool mirror was pretty much the same as already described for the easier scenario.

It is not as easy as with a hardware RAID controller running a mirrored pair of disks for the ESXi datastore, where one just needs to swap disks. I think one should really have exercised the whole procedure life and written it down before feeling safe.

However, I'm using two SSD's for the ESXi datastore and had the strong feeling that adding a hardware RAID controller would just enter another electronic component whith pretty much the same probability to fail as one of the SSD's but with one major difference: the RAID controller is an additional single point of failure.
Therefore I think that this "poor mans mirroring" under ESXi even has some advantage, at least for a lab or home server.

Good to see that napp-it is supporting this setup now, which will hopefully make part of the procedure less cumbersome.
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
Thank you for this detailed info. It helps to understand some of the restrictions, especially the needed reboot.
Beside that, you have a very short time to regain services and the quality of a ZFS Raid-1 is much better than with any hardware raid.

I have used a lot of hardware-Raid with ESXi and the lack of checksumming can give a problems because usual hardware controller cannot decide which part of a mirror is valid in case when they are different for whatever reason - mostly this type of error is not even detected. Usual Raid-1 solutions are only ok if one disk dies completely but often they are semi-dead resulting in stability problems without knowing the reason.
 

geppi

New Member
Jan 24, 2013
11
2
3
I have two questions regarding your all-in-one setup instructions.

You say: "Create a new VM (Solaris 10-64Bit, min 20 GB systemdisk, 4 GB RAM+ the more the faster, single core, with ESXi5 dualcore, VMCI enabled, added SAS-PCI adapter, DVD connected to your uploaded Omni/ OpenIndiana ISO"

1. Do you know what the difference is between creating a VM with the Solaris 10-64bit setting and the Solaris 11-64bit setting ?

2. How do you use VMCI with the OmniOS VM ?
As far as I understand it an application has to be written against the VMCI API to make use of the functionality it provides. Are there such applications in OmniOS or OI ?

BTW, in the vSphere 5.1 release, guest to guest VMCI has been removed. VMCI support continues only for host to guest communications. See http://www.vmware.com/support/developer/vmci-sdk/VMCI-510-Relnotes.html
 

GaryM

New Member
Feb 27, 2013
8
1
3
Belleville, ON
rpool mirror problem

Hello all, I have a Supermicro H8SCM -f motherboard, Opteron 4170HE processor, SASUCI8 controller with 5 2 TB Seagate Barracuda XTs and 3 1TB WD RE4s. I boot from a Seagate 80 gig 2.5' HD and I have 2 WD 2.5" 500 GB Blacks for my datastores which connect to the motherboard controller.
I have been running ESXI 5.0 and now 5.1 with Openindiana ver. 151a7 and napp-it 0.9a7 in a VM.
The problem I have is that napp-it does not see my second datastore drive, so it cannot create a rpool mirror. If I create an RDM of the second drive and give it to OI/napp-it then it sees the drive and makes a mirror so it says. When I removed the first drive of the mirror OI/napp-it won't start leading me to believe I don't have a true mirror using the RDM pass through to Openindiana.
Do I have a proper zpool mirror with the RDM and why doesn't the 2nd datastore drive get seen by OI/napp-it? I've gone through several online methods of making a zpool mirror, but still nothing.
Thanks in advance for any help you are able to give me.
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
I have two questions regarding your all-in-one setup instructions.

You say: "Create a new VM (Solaris 10-64Bit, min 20 GB systemdisk, 4 GB RAM+ the more the faster, single core, with ESXi5 dualcore, VMCI enabled, added SAS-PCI adapter, DVD connected to your uploaded Omni/ OpenIndiana ISO"

1. Do you know what the difference is between creating a VM with the Solaris 10-64bit setting and the Solaris 11-64bit setting ?

2. How do you use VMCI with the OmniOS VM ?
As far as I understand it an application has to be written against the VMCI API to make use of the functionality it provides. Are there such applications in OmniOS or OI ?

BTW, in the vSphere 5.1 release, guest to guest VMCI has been removed. VMCI support continues only for host to guest communications. See Release Notes

These Infos are from ESXi 4.1 but I suppose it will work with Solaris 11 setting as well
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
Hello all, I have a Supermicro H8SCM -f motherboard, Opteron 4170HE processor, SASUCI8 controller with 5 2 TB Seagate Barracuda XTs and 3 1TB WD RE4s. I boot from a Seagate 80 gig 2.5' HD and I have 2 WD 2.5" 500 GB Blacks for my datastores which connect to the motherboard controller.
I have been running ESXI 5.0 and now 5.1 with Openindiana ver. 151a7 and napp-it 0.9a7 in a VM.
The problem I have is that napp-it does not see my second datastore drive, so it cannot create a rpool mirror. If I create an RDM of the second drive and give it to OI/napp-it then it sees the drive and makes a mirror so it says. When I removed the first drive of the mirror OI/napp-it won't start leading me to believe I don't have a true mirror using the RDM pass through to Openindiana.
Do I have a proper zpool mirror with the RDM and why doesn't the 2nd datastore drive get seen by OI/napp-it? I've gone through several online methods of making a zpool mirror, but still nothing.
Thanks in advance for any help you are able to give me.
I would create a second virtual disk (same size) on the second datastore and use it for a ZFS mirror.
The mirror is mainly a way to protect against bad sectors.

In case of problems, you can recreate/restore the VM on the first disk, and set VM-Bios to boot from second disk
(and rebuild mirror within napp-it)
 

GaryM

New Member
Feb 27, 2013
8
1
3
Belleville, ON
Thanks for your reply Gea. I found my problem. When one follows instructions too closely one tends not to think. I had followed your instructions to the letter but forgot to add this 2nd virtual disk to the OI VM settings. It just came to me as I sat down to see if there was an answer to my questions. No wonder OI couldn't see the 2nd disk.
Thanks again.
BTW, I have been using OI/napp-it for a few months now and love it.
 

joisey04

Member
Jul 12, 2012
30
0
6
Mirror Omni/OI bootdisk with napp-it menu Disk - Mirror bootdisk
Setup VM-Bios to boot from the second disk first and the first disk (with ESXi on it) at second.
You can the replace/update the ESXi bootdisk without a problem for Omni/OI and remirror Omni/OI afterwards.
Could you please explain? If lets say, ESXi is on the first disk, why would I boot from Disk2?
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
Could you please explain? If lets say, ESXi is on the first disk, why would I boot from Disk2?
so many options,
basically you need to boot ESXi either from
- USB stick
- Sata disk
- Raid array

beside USB stick, you can also use the remaining space as a local datastore for VMs

For All-in-One, you need to put a SAN-VM itself on a local datastore disk
(and a mirror on a second datastore disk if you intend to ZFS mirror).

So your main options are
- USB stick for ESXi
- Sata disk for VMs
-opt. second Sata disk for VM-mirror datastore

- Sata disk for ESXi
- use the disk as datastore too
- opt. second Sata disk for VM-mirror

- use a raid array for Esxi + local datastore
 

joisey04

Member
Jul 12, 2012
30
0
6
thanks gea for your reply and I understand that

what I dont understand is the step "switching the boot device" after mirroring, assuming this is a setup where ESXi is on the first disk and not on a USB stick.....

Mirror Omni/OI bootdisk with napp-it menu Disk - Mirror bootdisk
Setup VM-Bios to boot from the second disk first and the first disk (with ESXi on it) at second.
You can the replace/update the ESXi bootdisk without a problem for Omni/OI and remirror Omni/OI afterwards.
 

xnoodle

Active Member
Jan 4, 2011
259
48
28
Making sure you two are on the same page, gea is talking about changing the virtual machine boot order and not changing the boot order on the physical machine.
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
A tricky setting...

A boot disk failure on a Solarish NAS or SAN is mostly not a problem.
Even if you do not have a recent backup or a spare boot disk with a recent state, you can install and setup your NAS/SAN in less than an hour without deep setup knowledge and import the pool with most settings kept in the pool.

But sometimes ex with a database or other services where you need current state of your boot-system, you do ZFS mirrors of rpool. (They are much better than conventional Raid due the checksums and auto-healing of ZFS mirrors). In a barebone system with a ZFS mirror with a failed boot-disk, you enter your mainboard bios, select the mirror as boot disk and you are up again.

In a virtualized environment, this is a little bit more complicated. If your boot disks fails and you do not have a current backup or spare disk, your VM config is missing as well as your virtual boot disk. In such a case, you need to recreate the VM, add a new empty VM boot-disk (keep size) and add the disk with the ZFS mirror (from a datastore from a second real disk) as a second disk.

Now you must edit ESXi settings of your VM and enable "Enter virtual Bios on next reboot" or similar to enable start from the intact ZFS mirror. When your Solarish system is up, you can replace the missing boot-disk with the new and empty one.

You are online again with exact data from the time when your first datastore disk fails.
 
Last edited:

joisey04

Member
Jul 12, 2012
30
0
6
understand that too

in this scenario, if I'm correct, ESXi plus the datastore for OMNI is on the first disk.
The second physical disk is configured as a ESXi datastore and the OMNI VM is then using this second datastore to mirror the rpool. There is NO ESXi on that second disk.
So in case of a bootdisk failure I would install ESXi on a new disk, create a VM and point to the datastore on the second (mirrored) disk. By mirroring that second disk to the datastore on the new disk, I would be mirrored and good again....

- Why would I now change as a default the boot drive for the VM to the second disk without ESXi??? Or am I reading that line wrong?
Setup VM-Bios to boot from the second disk first and the first disk (with ESXi on it) at second.
- And as a second thought: Isn't the ZFS on the datastores in "user space" and therefore not really "safe" since zfs doesn't work on tha actual disk but on a ESXi datastore?
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
- Why would I now change as a default the boot drive for the VM to the second disk without ESXi??? Or am I reading that line wrong?


- And as a second thought: Isn't the ZFS on the datastores in "user space" and therefore not really "safe" since zfs doesn't work on tha actual disk but on a ESXi datastore?
1. You have a two stage boot.
ESXi needs to boot from "real hardware" and it cannot do soft-mirrorring (unlike ZFS).
So you can only boot from USB or first Sata disks (beside real Hardware Raid but there you have no advanced ZFS mirror features)

Next, you need to boot a virtual instance of Solarish (=any Solaris like OS).
You have lost your original boot-disk. On your new bootdisk, there is now ESXi and a datastore with an empty virtual disk for Solarish. On your second disk there is an old datastore with your valid ZFS mirror (a ESXi vdmk disk), you want to boot from. You have added this vdmk-disk as an additional disk to your VM.

If you like to boot from this second virtual disk, you must tell ESXi (the virtual Bios of your VM)

(Other option would be to import this disk as bootdisk but then you have one VM on two disk - Config and disk - bad)

2. ZFS on virtual disks is ok if you can accept:
You have not the best performance, cannot read smart values. And you should not need sync writes, but this is not needed on rpool. So everything is fine
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
- Why would I now change as a default the boot drive for the VM to the second disk without ESXi??? Or am I reading that line wrong?


- And as a second thought: Isn't the ZFS on the datastores in "user space" and therefore not really "safe" since zfs doesn't work on tha actual disk but on a ESXi datastore?
1. You have a two stage boot.
ESXi needs to boot from "real hardware" and it cannot do soft-mirrorring.
So you can only boot from USB or first Sata disks (beside real Hardware Raid but there you have no advanced ZFS mirror features)

Next, you need to boot a virtual instance of Solarish (=any Solaris like OS).
You have lost your original boot-disk. On your new bootdisk, there is now ESXi and a datastore with an empty virtual disk for Solarish. On your second disk there is an old datastore with your valid ZFS mirror (a ESXi vdmk disk), you want to boot from. You have added this vdmk-disk as an additional disk to your VM.

If you like to boot from this second virtual disk, you must tell ESXi (the virtual Bios of your VM)

(Other option would be to import this disk as bootdisk but then you have one VM on two disk - Config and disk - bad)

2. ESXi on virtual disk is ok if you can accept:
You have not the best performance, cannot read smart values. And you should not need sync writes, but this is not needed on rpool.
So everything is fine
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
the hpvsa driver can use intel c600 sata or lsi 2308 and perform smart array in software with selectable read-ahead cache.

Does intel not have a similar driver? ERST2? or LSI2308?
 

joisey04

Member
Jul 12, 2012
30
0
6
ok, so the whole thing with changing the boot disk to the second disk with only the solaris mirror on it is just for the fail scenario

in regular mode, I boot from my first datastore where ESXi and my *.vmdk file is stored, mirror that to the second datastore on disk2

after fail, I install new boot disk, install ESXi + datastore and a empty solaris VM. Then I change the boot drive in the VM bios to datastore on the second disk. mirror that again with datastore on disk one and all is good