Cluster Mirrored Storage Software (VSAN alternative?)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
H

hyltcasper

Member
May 1, 2020
45
4
8
Hi I am looking a
Storage Solution For No Downtime System

I need a storage solution for highly available virtual machines. I am developing a startup like UpCloud. I will offer all NVME virtual machines to customers. I target maximum IOPS.

Looking for 2 features. One is covered by ZFS, other is covered by Microsoft S2D. But I couldn't find both the same time.

It shouldn't duplicate the same data. For example operating system files must write once. All VM's must use the same files. Differences must be handled by copy on write. Because NVME's very expensive. ZFS is covering this situation.

3 or more times Live Data Copy like Microsoft S2D. When disk failure or machine failure happened, virtual machines must continue to run without any problem in seconds. This will be done by automated processes below:
------ Detect machine or disk failure
------ Boot failed virtual machines on a healthy machine. Thanks to duplicate data!
------ After each successful boot, assign machine's old ip address.

I am also sharing current hardware design. Machines haven't fibre cables. I can obtain if necessary but I don't want because they are expensive. Each machine have dual 1 gigabit ethernet port. Each NVME disk is 2TB. All HDD's are 8 TB. Virtual machines will run on NVME disks. HDD's are exist for only backup and they will sleep almost all time.

Machine_1:

- NVME_1
- NVME_2
- HDD_1

Machine_2:

- NVME_3
- NVME_4
- HDD_2

Machine_3:

- NVME_5
- NVME_6
- HDD_3

It does not appeal to me as the storage on the network is very expensive. The current hardware configuration is very cost effective. Because it using the same machines for compute and storage. First class communications between disks and cpu. Maximum IOPS. I highly recommended.

I need your solution suggestions that provide both of the features I mentioned.

There is a old thread but I couldn't find a fitted solution.

My budget is 200$ monthly for this storage software. But I highly prefer free open source solution.
 
A

amalurk

Active Member
Dec 16, 2016
312
116
43
102
So you want:
a) hyperconverged system that gives as good IOPS as local drives but tolerates failure of a node through shared storage
b) you want to do that over 2 x 1gb links per node.

A does not exist even without B (will be a lot slower than local drives, see VSAN, Proxmox w/ Ceph, you have to scale to large # nodes to get good IOPS) and with B it won't even come within a mile.
 
  • Like
Reactions: Blinky 42
H

hyltcasper

Member
May 1, 2020
45
4
8
amalurk said:
So you want:
a) hyperconverged system that gives as good IOPS as local drives but tolerates failure of a node through shared storage
b) you want to do that over 2 x 1gb links per node.

A does not exist even without B (will be a lot slower than local drives, see VSAN, Proxmox w/ Ceph, you have to scale to large # nodes to get good IOPS) and with B it won't even come within a mile.
Click to expand...
Yep I want A. If only problem is links I can get fibre ports. I need something boutique. I don't want more nodes. Instead of more nodes I prefer upgrade nodes to newer EPCY. Because these cases are really heavy and difficult to carry. More computers mean more waste of electricity while more effective computers are exist.

I am using all NVME disks for running virtual machines. For 3 machine setup; machine 1,2,3 will instantly mirror each other's data. If one of machines fail, virtual machines will boot on another machine without any data loss. Seconds are very important here. VMware Vsphere done this with it's own shadow VM features. There is some softwares like Microsoft S2D doing that but I need ZFS like features also. Because every single virtual machine runs ubuntu 20.04, duplicating ubuntu 20.04 operating system files for each virtual machine is totally disk wasting. Writing too much can cause discs to die too early. Also backups will huge and very slow.

An example for 3 virtual machines. In a real scenario there will be 400 vms.
Without copy on write
Ubuntu 20.04, 20GB
User1 Files, 5 GB
Ubuntu 20.04, 20GB
User2 Files, 5 GB
Ubuntu 20.04, 20GB
User3 Files, 5 GB
Total 75 GB
Monthly cost of daily backups: 2.2TB

With copy on write with ZFS like file systems.
Ubuntu 20.04, 20GB
User1 Files, 5 GB
User2 Files, 5 GB
User3 Files, 5 GB
Total 35 GB
Monthly cost of daily backups: 0.5 TB 4x faster!

I need an easy to configure software that does not need to be managed.
 
D

dswartz

Active Member
Jul 14, 2011
610
79
28
Have you looked at starwind's iSCSI 2-node cluster? Can be run with the 2 storage appliances under vmware, and can use NVME for the storage on each node.
 
E

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
And you want it cheap ?
Possibly the only thing with a small number of nodes that may work is this Virtualise Everything - Sunlight
But it won’t be free like KVM and or ‘simple’ (but expensive and not so performing even with all NVMe) like VMware.
UpCloud is KVM based as I know but not sure what the actual storage back end is... a quick Google didn’t really find the answer either. Anybody know ?
 
I

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
I was told once that a high availability san would costs more than $100k+ (including discounts) :confused:
 
E

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
i386 said:
I was told once that a high availability san would costs more than $100k+ (including discounts) :confused:
Click to expand...
well that entirely depends what level your playing at but if you call dual controller redundant something like a DellEMC Unity with all flash disks is way way way less than that. All depends on capacity.

if you talking very high end IBM, HDS, Hitachi, EMC, etc the yes it starts at 250k and up (but when you offer millions of IOPS and 100% data availability then that expected)

if you using fiber channel remember unless it’s only a couple of hosts the switches need to be added that costs $$ as well.
 
Last edited:
Blinky 42

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
If you are offering the VM's up to clients/users who will install their own stuff and manage the VMs themselves, then they will quickly diverge from the original virgin install image with actual user data. If you have more control over things and can force the config to store the actual unique content on a a different device then you can save some space that way (like put /var /tmp and /home on a 2nd lun, with / on the 1st).

If that isn't feasible - how much disk space are you giving to each VM? If it is for the general public / paying clients I wouldn't try and fight to over-subscribe your storage with the setup you describe above, because as son as you have a few rouge processes on a dozen VMs it will consume all of your fee space and then the entire storage system for all clients with be SOL.
At a medium scale, people tend to use a SAN with a good block-level de-dupe, but that still doesn't do awesome over time, and they are not inexpensive.

And seriously - don't even bother setting this up for anything other that fun home use with 1G networking and NVMe, the speed differential is absurd. If your i/o load of all the VMs can be sync'ed to the other servers in the cluster over just 2G of network then you can get away with cheap SSDs or even HDDs (considering you can saturate 10g networking no problem with a small HDD array). If you are trying to offer redundancy by sync across the network, then the speed of the speed of the network is your limiting factor.

If clients are paying for it:
- Make them cover the costs of a decent storage layer & supporting network

If it is for you:
- Lock it down and make static / OS file content served up by a single central filesystem, and then all transient & user data on different filesystems (local or remote). Linux and Windows labs across the planet have been doing that for decades to give performance to the files that matter and mimize the waste of storing redundant copies of OS files.
 
R

Rand__

Well-Known Member
Mar 6, 2014
6,634
1,767
113
Just stumbled over this, @hyltcasper - have you decided on a tool yet?
I very much doubt that it will work as described unfortunately (mabye we dont have all relevant information though), but if it does I'd be interested in what you came up with :)
 
You must log in or register to reply here.
Top Bottom