Colocation Server Setup Sanity Check

[sachem87184]

So I'm finalizing a build setup for one of the servers I'm send out to be colocated. It will be sitting behind a physical PFsense appliance and be mainly used for storage and VMs.

• Setup:
• OS:
  • Debian 8 Jessie
• Hardware:
  • Supermicro 847 Chassis
  • Supermicro X9DRi Motherboard
  • 2x Intel e5-2670
  • 128gb-192gb DDR3
  • LSI 9211-8i SAS Card
  • Adaptec 6805 SAS Card
  • 28x 2TB HDD
  • 6x 3TB HDD
  • 2x 300GB SSD
  • 4x 400GB SSD
• Steps:
  • Create RAID1 of 2x 300GB SSD (Intel S3500) with Adaptec SAS Card
  • Install Debian on RAID1 with luks for all partitions but boot
  • Install dropbear, setup keypairs, and setup for boot decryption of luks
  • FDE Encrypt all other disks, created derived keys for unlocking, and save to root fs
  • Create Snapraid setup using 8x 2TB as data, 4x 3TB as parity, and 2x 3TB as spares
  • User mergerfs on Snapraid setup
  • Install zfs on linux
  • Create pool of 3x vdev, each containing 6x 2TB in RAIDZ2 (2x 2TB Spares, 1x 400GB Cache, 1x 400GB ZIL)
  • Install kvm, libvirt, and virt-manager
• After Thoughts:
  • How to leverage the mergerfs mount for plex (I'm thinking cifs from samba in debian, or a pass-through like bsd jails in some fashion to kvm)
  • Need to find a good webui for kvm
  • Need to lookup security models regarding libvirt networking

With this being my first non-esxi colo build I was looking for any insight or advice into this.

 

[Patrick]

My one question looking at the specs is why use LSI + Adaptec? Twice as many drivers, different management toolsets and etc.

BTW - did you look at Proxmox VE by chance? Maybe a decent option since you are doing ZoL and such. You could then run everything using ZFS and just be done with thinking about it. You also get a GUI for KVM.

My general sense is, try to minimize the number of moving parts you have. One parity RAID solution, not hardware, Snapraid and ZFS.

How I got colocation to be absolutely a breeze was to change my perspective:

• Ditch HW RAID for SW RAID (ZFS) or cluster (Ceph)
• Minimize the number of hard disks used (each has a 5% AFR)
• Ensure you have a plan for spares (e.g. when something fails)
• Test failure well
• Minimize management interfaces

That has me down to visiting the colo once every year or so. I now just allow stuff to fail in place.

Just for reference, the colo is now about 20 minutes from home and deemed too inconvenient to get to more than once or twice a year.

 

[awedio]

@Patrick I'm going to start a new thread using your last reply? Got some questions!

 

[Patrick]

@Patrick I'm going to start a new thread using your last reply? Got some questions!

OK.

 

[sachem87184]

@Patrick The LSI is just a plain ol' HBA for passthrough of the disks. The Adapted I already had laying around and I needed a good RAID1 setup for the boot drive. I decided on Hardware RAID because of the complexities of combining Debian boot/root, luks, and lvm/mdadm mirroring just to mirror my boot disk


Also, I wanted Snapraid for the simplicity and easy of recovery for my media in Plex (going on 10+TB). When I use ZFS for this I think it's overkill and a waste of good useable I/o.


Edit: I've been testing failures and luckily I found that bad motherboard I messaged you about. I like to do some heavy burn in if it's going remote. Its an easy fix when I'm 10 minutes away, but this data center is a 7hr drive for me.

 

[Patrick]

@sachem87184 I understand that Snapraid/ hardware RAID part, but you are introducing a lot more complexity.

Try Proxmox. You can setup a ZFS mirror as a root ZFS pool easily with the installer.

 

[sachem87184]

@Patrick I initially went that route with Proxmox, but ran into an issue looking into how to easily integrate luks encryption with a dropbear ssh server for remote unlocking.