Project Update 18 Feb-16
Virtual Environment
After playing around with a few different virtual environments I decided to go with
Proxmox. It installed easily and the 10GB NICs came up after the initial installation without any hassle. For instructions on how to install, pls refer to
this guide made by Patrick from STH.
In my case the host server uses 2x250GB SSDs (SAMSUNG SSD 850 EVO 2.5" SATA III 250GB) in a RAID-1 configuration as Proxmox boot drives, and to store the guest VMs. VMs are backed up to a secondary NAS using NFS share. This is a less fancy solution compared to the Ceph cluster in Patrick's guide, but it gets the job done.
In addition the host server also has a 2x6TB HDD configuration (HGST Deskstar NAS Hard Disk Internal SATA III 7200 RPM 6TB). Each drive is configured for PCI pass-through to the FreeNAS VM. In the below picture we can see the host server, "
unas-host", running one guest VM, "
freenasvm100", as well as the external NFS share used for VM backups, "
readynas_nfs_backup".
FreeNAS Virtual Machine
The two HDDs are configured as a mirrored ZFS pool in FreeNAS, seen in the below GUI screen-shoot as "
bigpool".
Data Migration
When I researched this topic I wanted to find methods for migrating data between two NAS systems which supported data integrity on file content level. In the real world a new storage deployment is seldom done in isolation. But rather it is part of a bigger scenario where existing data needs to be migrated. For someone working with unique content like photo or video, any data corruption would be disastrous. But despite my best efforts it proved challenging to find a good guide on how to migrate files across storage systems from a data integrity perspective.
I would be very interested in hearing from professionals about what procedures they use to ensure data integrity during IT migration projects.
My old NAS from which I am migrating from,contains about 3 TB of data. The migration strategy follows the following steps:
- Initial bulk migration: Do a complete copy of all files in the source NAS using a NFS share on the target NAS.
- Incremental migration: Maintain data integrity my doing period incremental copies using rsync.
- Repeat #2 until new target NAS is considered stable and ready for service
Using NFS to copy files is far quicker than
rsync, but obviously not very intelligent. To add some data integrity I TARed each directory to be migrated, piped the file names to MD5 and saved each individual file's MD5 hash in a separate file. The big idea being that once I unpack the tar on the target NAS, the data integrity of the file content could be validated using the MD5 hash file as master reference. That was the concept.
In reality, my source NAS is running a ReadyNAS fork of Debian Sid containing
md5sum tool. And the target NAS is based on Freebsd which ships with
md5 tool. And the format between these two is obviously different. So practically, using md5 checks between two platforms for data integrity was more complicated than anticipated. Being able to do integrity checks on individual file content level would be great, but it's fiddly to make it work between BSD and linux systems.
A simpler approach would be to do integrity checks based a tool supported on both platforms, e g gzip. Here the CRC checks are done on archive level, rather than on individual file level. Meaning unpacking would detect a corrupted archive, but would not be able to repair the data. So the trade off is that in event of data corruption detected in the target NAS, the entire archive needs to be reprocessed in the source NAS and copied across again.
Stability Period
Right now my FreeNAS is in a stability/monitoring phase during which it mainly serves as a backup for my old NAS. I need to improve my UPS set-up before I deploy my UNAS as primary NAS server.
Project Findings So Far
This has been a very enjoyable project with a lot of challenges. Just getting the system to post and run the CPU without overheating was a major hurdle. I expected the visualization to and FreeNAS to be the most difficult parts to grasp. But I was impressed with the level of maturity that Proxmox and FreeNAS has reached, and found the initial learning curve to be less steep than expected. This is mainly thanks to a very active user community and forums. Going from zero to actual deployment is achievable thanks to that support, and it builds confidence that if I were to run in to operational issues in the future there are experienced willing to share their knowledge.
//Jimmy