A few days ago, I saw a thread in this forum about a DIY Oracle ZFS Storage Appliance. I realized that my own experience on that could be interesting for others as not that much information are available on the net.
The beginning of the story is the Oracle ZFS Storage simulator for Virtual Box that you can download from Oracle. I gave it a try and really liked the efficient UI and most important the analytics part with so much option to see what is happening on your storage. After digging a bit, I realized that the simulator is the full software of Oracle ZFS Storage Appliance. I then started to investigate to see if it would be possible to use it under VMware ESXi.
The following are the steps that are needed for that.
First download the Oracle/Sun Storage simulator : Oracle ZFS Storage Simulator Downloads
Create a new virtual machine in ESXi. You will need to select at least virtual hardware 10 (ESXi 5.5 and above) as we will need a SATA controller for the VM. Select Solaris 11 as the guest OS. Give at least 2560 MB of memory to the VM. Remove the default hard drive and the "LSI Logic Parallel Controller" (as the ZFS-SA won't like it). Also add a SATA controller to the VM.
Unzip the downloaded file of the storage simulator. Upload "Oracle_ZFS_Storage-disk1.vmdk" to your ESXi host in the directory of the VM you have created above.
You will need to convert the vmdk file. If you attach it directly to the VM it will appear to work but whenever you will want to change your VM you will run into trouble. Login to your ESXi ht using ssh, go to the directory of the VM and use vmkfstools to make the conversion:
vmkfstools -i Oracle_ZFS_Storage-disk1.vmdk Oracle_ZFS_Storage-boot.vmdk -d thin -a sata
Add a new hard drive to the VM, select use an existing disk and select the "Oracle_ZFS_Storage-boot.vmdk" file and attach it to the SATA controller (if a LSI controller is created, remove it)
At that point the system won’t boot because the Solaris boot disk is not configured properly. In order to fix that I followed the instructions of RIAAN'S SYSADMIN BLOG
Boot from a Solaris image. In ESXi, remember that pressing ESC during boot will bring you the boot menu where you can select to boot from the CDROM. Select keyboard, language then select shell. The purpose is to set the right path for the boot disk
root@solaris:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c2t0d0 <ATA-VMware Virtual S-0001 cyl 6524 alt 2 hd 255 sec 63>
/pci@0,0/pci15ad,790@11/pci15ad,7e0@2/disk@0,0
Specify disk (enter its number): ^C
"format" will give you the correct path, you then need to update ZFS-SA boot configuration
root@solaris:~# zpool import -f system
root@solaris:~# zfs list | grep root
system/ak-SUNW_ankimo-2013.06.05.4.0_1-1.7/root 1,64G 36,8G 1,63G legacy
root@solaris:~# mkdir /a
root@solaris:~# mount -F zfs system/ak-SUNW_ankimo-2013.06.05.4.0_1-1.7/root /a
root@solaris:~# zfs set readonly=off system/ak-SUNW_ankimo-2013.06.05.4.0_1-1.7/root
root@solaris:~# cp /etc/path_to_inst /a/etc
root@solaris:~# echo "setprop boot /devices/pci@0,0/pci15ad,790@11/pci15ad,7e0@2/disk@0,0:a" >>/a/boot/solaris/bootenv.rc
root@solaris:~# bootadm update-archive -R /a
updating /a/platform/i86pc/boot_archive
updating /a/platform/i86pc/amd64/boot_archive
root@solaris:~# umount /a
root@solaris:~# zpool export system
root@solaris:~# init 0
At that point, the ZFS-SA should boot correctly under ESXi.
Proceed with the basic configuration of the appliance. Connect to the web interface to perform the full setup.
Once the setup if finished, login into the appliance (you can use ssh for that). You will login into the appliance shell (named aksh). in order to get a "real" shell, type "shell" to get a bash shell.
First we need to make the root fs writable :
mount -o rw,remount /
Likely you will want to install the vmxnet3s driver. You can do it in the following way. Initiate vmware tool installation in ESXi. The CDROM won't mount automatically so the following steps are needed :
mkdir /tmp/cdrom
mount -F hsfs /devices/pci\@0\,0/pci-ide\@7\,1/ide\@1/sd\@0\,0:a /tmp/cdrom/
cd /tmp/
tar xzf cdrom/vmware-solaris-tools.tar.gz
cd vmware-tools-distrib
install -f /kernel/drv/amd64/ -u root -g sys -m 0755 ./lib/modules/binary/11_64/vmxnet3s
install -f /kernel/drv/ -u root -g sys -m 0644 ./lib/modules/binary/11/vmxnet3s.conf
add_drv -i '"pci15ad,7b0"' vmxnet3s
MTU for vmxnet3s on Solaris can not be set in a conventional way and the appliance will not like that so you need to edit edit "/usr/lib/ak/svc/method/akdatalink" and comment the following 3 lines
# if [[ $? -ne 0 ]]; then
# dl_cfgfail "$1: could not set MTU $mtu"
# fi
VMware is not Virtual Box so for the web UI to look good, some more editing is needed. For that what's need to be edited are the platform hc topology and the platform xml description.
cd /usr/platform/i86pc/lib/fm/topo/maps/
cp VirtualBox-hc-topology.xml VMware-Virtual-Platform-hc-topology.xml
vi VMware-Virtual-Platform-hc-topology.xml
Replace ‘VirtualBox’ by ‘VMware-Virtual-Platform’
Correct the PCI path for the SATA disks to point to the correct location (as found when you made the vmdk file bootable)
:%s/pci8086,2829@d/pci15ad,790@11\/pci15ad,7e0@2/g
cd /usr/lib/ak/metadata/appliance/SUNW,ankimo/chassis/
cp VirtualBox.xml VMware-Virtual-Platform.xml
vi VMware-Virtual-Platform.xml
Replace ‘innotek GmbH’ by ‘VMware’ and ‘VirtualBox’ by ‘VMware-Virtual-Platform’
Reboot the appliance in reconfiguration mode.
reboot — -r
if it hangs, just reset it.
This is what you get
The beginning of the story is the Oracle ZFS Storage simulator for Virtual Box that you can download from Oracle. I gave it a try and really liked the efficient UI and most important the analytics part with so much option to see what is happening on your storage. After digging a bit, I realized that the simulator is the full software of Oracle ZFS Storage Appliance. I then started to investigate to see if it would be possible to use it under VMware ESXi.
The following are the steps that are needed for that.
First download the Oracle/Sun Storage simulator : Oracle ZFS Storage Simulator Downloads
Create a new virtual machine in ESXi. You will need to select at least virtual hardware 10 (ESXi 5.5 and above) as we will need a SATA controller for the VM. Select Solaris 11 as the guest OS. Give at least 2560 MB of memory to the VM. Remove the default hard drive and the "LSI Logic Parallel Controller" (as the ZFS-SA won't like it). Also add a SATA controller to the VM.
Unzip the downloaded file of the storage simulator. Upload "Oracle_ZFS_Storage-disk1.vmdk" to your ESXi host in the directory of the VM you have created above.
You will need to convert the vmdk file. If you attach it directly to the VM it will appear to work but whenever you will want to change your VM you will run into trouble. Login to your ESXi ht using ssh, go to the directory of the VM and use vmkfstools to make the conversion:
vmkfstools -i Oracle_ZFS_Storage-disk1.vmdk Oracle_ZFS_Storage-boot.vmdk -d thin -a sata
Add a new hard drive to the VM, select use an existing disk and select the "Oracle_ZFS_Storage-boot.vmdk" file and attach it to the SATA controller (if a LSI controller is created, remove it)
At that point the system won’t boot because the Solaris boot disk is not configured properly. In order to fix that I followed the instructions of RIAAN'S SYSADMIN BLOG
Boot from a Solaris image. In ESXi, remember that pressing ESC during boot will bring you the boot menu where you can select to boot from the CDROM. Select keyboard, language then select shell. The purpose is to set the right path for the boot disk
root@solaris:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c2t0d0 <ATA-VMware Virtual S-0001 cyl 6524 alt 2 hd 255 sec 63>
/pci@0,0/pci15ad,790@11/pci15ad,7e0@2/disk@0,0
Specify disk (enter its number): ^C
"format" will give you the correct path, you then need to update ZFS-SA boot configuration
root@solaris:~# zpool import -f system
root@solaris:~# zfs list | grep root
system/ak-SUNW_ankimo-2013.06.05.4.0_1-1.7/root 1,64G 36,8G 1,63G legacy
root@solaris:~# mkdir /a
root@solaris:~# mount -F zfs system/ak-SUNW_ankimo-2013.06.05.4.0_1-1.7/root /a
root@solaris:~# zfs set readonly=off system/ak-SUNW_ankimo-2013.06.05.4.0_1-1.7/root
root@solaris:~# cp /etc/path_to_inst /a/etc
root@solaris:~# echo "setprop boot /devices/pci@0,0/pci15ad,790@11/pci15ad,7e0@2/disk@0,0:a" >>/a/boot/solaris/bootenv.rc
root@solaris:~# bootadm update-archive -R /a
updating /a/platform/i86pc/boot_archive
updating /a/platform/i86pc/amd64/boot_archive
root@solaris:~# umount /a
root@solaris:~# zpool export system
root@solaris:~# init 0
At that point, the ZFS-SA should boot correctly under ESXi.
Proceed with the basic configuration of the appliance. Connect to the web interface to perform the full setup.
Once the setup if finished, login into the appliance (you can use ssh for that). You will login into the appliance shell (named aksh). in order to get a "real" shell, type "shell" to get a bash shell.
First we need to make the root fs writable :
mount -o rw,remount /
Likely you will want to install the vmxnet3s driver. You can do it in the following way. Initiate vmware tool installation in ESXi. The CDROM won't mount automatically so the following steps are needed :
mkdir /tmp/cdrom
mount -F hsfs /devices/pci\@0\,0/pci-ide\@7\,1/ide\@1/sd\@0\,0:a /tmp/cdrom/
cd /tmp/
tar xzf cdrom/vmware-solaris-tools.tar.gz
cd vmware-tools-distrib
install -f /kernel/drv/amd64/ -u root -g sys -m 0755 ./lib/modules/binary/11_64/vmxnet3s
install -f /kernel/drv/ -u root -g sys -m 0644 ./lib/modules/binary/11/vmxnet3s.conf
add_drv -i '"pci15ad,7b0"' vmxnet3s
MTU for vmxnet3s on Solaris can not be set in a conventional way and the appliance will not like that so you need to edit edit "/usr/lib/ak/svc/method/akdatalink" and comment the following 3 lines
# if [[ $? -ne 0 ]]; then
# dl_cfgfail "$1: could not set MTU $mtu"
# fi
VMware is not Virtual Box so for the web UI to look good, some more editing is needed. For that what's need to be edited are the platform hc topology and the platform xml description.
cd /usr/platform/i86pc/lib/fm/topo/maps/
cp VirtualBox-hc-topology.xml VMware-Virtual-Platform-hc-topology.xml
vi VMware-Virtual-Platform-hc-topology.xml
Replace ‘VirtualBox’ by ‘VMware-Virtual-Platform’
Correct the PCI path for the SATA disks to point to the correct location (as found when you made the vmdk file bootable)
:%s/pci8086,2829@d/pci15ad,790@11\/pci15ad,7e0@2/g
cd /usr/lib/ak/metadata/appliance/SUNW,ankimo/chassis/
cp VirtualBox.xml VMware-Virtual-Platform.xml
vi VMware-Virtual-Platform.xml
Replace ‘innotek GmbH’ by ‘VMware’ and ‘VirtualBox’ by ‘VMware-Virtual-Platform’
Reboot the appliance in reconfiguration mode.
reboot — -r
if it hangs, just reset it.
This is what you get
Last edited: