Hey guys,
Currently I have a couple of Dell MD3200 (SAS) plugged to a few ESXi hosts, which has done the job back in the day but now is a performance bottleneck, plus it is connected through SAS, limiting the quantity of hosts we can connect to it, and also has the propriety rebranded disks, so I can't use what I feel like. My infra structure is all VMware ESXi and Windows guest, ranging from MS IIS, MS Exchange servers to file servers, RDS server and Citrix servers, and some SQL servers. I've got about 10 Hosts with a mix of 2 dual sockets 4 and 6 cores CPUs, so time to refresh the hosts soon with some newer dual sockets 20 cores and more RAM.
The aim now is to go for a software defined storage, where we can use a regular 12bay server like a Dell R515 or R510 to start testing it. There are 3 paths from here:
I've been thinking for a long time to use ZFS. I tried Nexenta in the past but their HCL is rubbish, not a good support for hardware at all. Then recently ZFS on Linux has matured a lot, which has several companies offering their own version of Ubuntu/Debian with ZFS on top and management. Nothing new here as lots of players seem to be doing the same job, getting open source software and selling as their own. Lots of people talk highly of FreeeBSD + HAST + ZFS as way more mature than in Linux, but I'm not sure if it is still the case non late 2016. Lots of posts are from 2013/2014 and in 2-3 years a lot has changed. What commercial options are out there for FreeBSD? I know TrueNAS/FreeNAS and other do it, but considering my past experience with FreeNAS, I would not go there. I don't believe it is a good option, but you guys may think it is, and I'm happy to change my judgement.
Some solutions I've found/tested/equired:
There are some other thigns around like BTRFS which seems too experimental at the moment but for some reason Netgear is selling appliances based on it, lots of companies getting Ubuntu + ZFS and rebranding as their own stuff making claims is the holy grail. Also I have doubts about 10GB Ethernet, it has some latency so how bad would it be in my case with 10 hosts and about 100 VMs? I heard Melanox and their Infiniband 10/40GB cards have a lot lower latency. As we are moving to network storage instead of DAS I'm a bit concerned on the latency issues we can have over 10GB Ethernet, but it may be the case is negligible.
I'm also considering going all flash for the storage box. The Samsung SM863 SSDs that I've been using in RAID-5 have been fairly stable and good performer, so I may use them on a larger array, or perhaps Intel SSDs as they are only 10-20% more expensive than Samsung but have amazing track record.
What are your thoughts on this?
Thanks!
Currently I have a couple of Dell MD3200 (SAS) plugged to a few ESXi hosts, which has done the job back in the day but now is a performance bottleneck, plus it is connected through SAS, limiting the quantity of hosts we can connect to it, and also has the propriety rebranded disks, so I can't use what I feel like. My infra structure is all VMware ESXi and Windows guest, ranging from MS IIS, MS Exchange servers to file servers, RDS server and Citrix servers, and some SQL servers. I've got about 10 Hosts with a mix of 2 dual sockets 4 and 6 cores CPUs, so time to refresh the hosts soon with some newer dual sockets 20 cores and more RAM.
The aim now is to go for a software defined storage, where we can use a regular 12bay server like a Dell R515 or R510 to start testing it. There are 3 paths from here:
- convert existing regular servers to SDS, use internal storage on the box (2U 12bay R515/R510 servers) and share it using NFS or iSCSI over 10GB ethernet. I would like 2 boxes to begin so data is fully synced between the 2 boxes, so in case one goes down the other takes over with no downtime. I can do this in pfsense for example with virtual IPs, so one device has the virtual IP in standby and once the main pfsense fails, the virtual IP changes and it all works from the secondary box with no downtime
- use 1U servers as head node, connected via SAS cable to JBOD. I don't have any 1U server currently, but would most likely get Dell R630 or a supermicro equivalent with high memory module slots, use an HBA and connect to a JBOD (supermicro most likely). Some may say this approach is good because I can hook up 2x 1U servers to the same JBOD and work in high availability. So if one head fails, the other picks up the work. The downside is, you need to use SAS drives only as with SATA it will not work. My concern with this is, what happens if the backpannel of the JBOD fails? That is a single point of failure, and with one Dell MD3200, I had this problem in the past, we have dual controllers but one SAS backpane, if failed and the whole box was down.
- Hyperconverged, not entirely sure on this option yet, as you may have too many eggs in one basket. So there are a few different approaches like VMware VSAN where it seems to write parity to different nodes and pooling every single node as one single pool of storage. So if you lose one host you can rebuild the data parity to another host as the servers work in a RAID-5 like way? I may be wrong here, but that is what I understood. Nutanix and Solarwind work differently doing mirror, so one host has exactly same data as another host, and VMs work with "local" speed storage. What about when I want to expand beyond 3 nodes? So far I can see it replicates the data across 2 nodes, then mirrors up to 3 hosts. So every host I add I just keep replicating, adding more compute power but not actually adding storage?
I've been thinking for a long time to use ZFS. I tried Nexenta in the past but their HCL is rubbish, not a good support for hardware at all. Then recently ZFS on Linux has matured a lot, which has several companies offering their own version of Ubuntu/Debian with ZFS on top and management. Nothing new here as lots of players seem to be doing the same job, getting open source software and selling as their own. Lots of people talk highly of FreeeBSD + HAST + ZFS as way more mature than in Linux, but I'm not sure if it is still the case non late 2016. Lots of posts are from 2013/2014 and in 2-3 years a lot has changed. What commercial options are out there for FreeBSD? I know TrueNAS/FreeNAS and other do it, but considering my past experience with FreeNAS, I would not go there. I don't believe it is a good option, but you guys may think it is, and I'm happy to change my judgement.
Some solutions I've found/tested/equired:
- I've been testing Zetavault (they used to be called DNUK), which is just Ubuntu + ZFS with a average looking interface, but it works okay so far I can see on my test server.
- Open-E JovianDSS is Debian + ZFS, interface looks sleek but with less features. I've been battling with them to get a LSI2008 HBA (Dell Perc H200 flashed with IT bios) to work properly as their interface doesn't display my disks, so I can't use it. Their support is trying to get it to work for 5 days now.
- Starwind Virtual SAN - Seems more targeted towards Hyper-V than Vmware as it integrates nicely with Windows/Hyper-V for hyperconverged system, but with Vmware it would need to run as a barebone as I don't feel like running a virtual machine to store my data, which means standalone box with NFS/iSCSI. They make bold claims about re organising writes to disk in memory and then writing them sequentially to spinning disks, similar to what ZFS does every 5 seconds. They call this IO Blender, where several VMs IOps reach the storage as random IOps and they cache them to write as one single sequential write. I would need to use a barebone server for this, with Windows installed and then Solarwind softwrae on top. That means I need a hardwrae raid controller as I would not trust Windows software raid, it was always rubbish. I would rather use ZFS. Other claim by soalrwind is that you can use RAID-5/6 for storage and because of their way to cache write data to disk, you have better performance than using RAID-10. How do you guys see this? I think is a bold claim, to not say BS.
- Nutanix/Netapp/Tintri/Nimblestorage - all out of question, propriety hardware and expensive. Last time I checked in teh UK, Nutanix entry level single box would cost £25000, no a flipping way! It would need to take my kids to school and wash my dishes to justify this price tag.
- EMC ScaleIO - seems neat, I like they sell just the software without hardware. I'm waiting for quotes.
There are some other thigns around like BTRFS which seems too experimental at the moment but for some reason Netgear is selling appliances based on it, lots of companies getting Ubuntu + ZFS and rebranding as their own stuff making claims is the holy grail. Also I have doubts about 10GB Ethernet, it has some latency so how bad would it be in my case with 10 hosts and about 100 VMs? I heard Melanox and their Infiniband 10/40GB cards have a lot lower latency. As we are moving to network storage instead of DAS I'm a bit concerned on the latency issues we can have over 10GB Ethernet, but it may be the case is negligible.
I'm also considering going all flash for the storage box. The Samsung SM863 SSDs that I've been using in RAID-5 have been fairly stable and good performer, so I may use them on a larger array, or perhaps Intel SSDs as they are only 10-20% more expensive than Samsung but have amazing track record.
What are your thoughts on this?
Thanks!