ZFS vCluster in a Box

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
Nearly eight years ago, I introduced All-In-One virtualized storage, first at STH
Time for a new idea:

napp-it Z-Raid HA vCluster
easy and cheap enough

I am on the way to finish my Z-Raid Cluster in a Box solution.
In the past a Cluster in a Box consists of two servers with a common pool of mpio SAS disks. One of the servers builds a pool from the disks and offers services like NFS or SMB. On a failure or for maintenance you can switch over to the second server within a few seconds. Management is done for example with RSF-1 from high-availability. SuperMicro offers special cases that can hold two mainboards

Such solutions are expensive and management is quite complex but offers high performance and high availability. To reduce costs, my solution is based on ESXi (any license) to virtualize the two nodes and a control instance. It uses the shared controller/ shared raw disk options of ESXi so my solution does not need multipath SAS but can work with any disks.

Setup see http://www.napp-it.org/doc/downloads/z-raid.pdf

If you want to try the preview state that allows a manual failover within 20s, you can update napp-it to the free 18.02 preview (or 18.12dev).

 
Last edited:

MiniKnight

Well-Known Member
Mar 30, 2012
2,987
892
113
NYC
I think you should try selling these to pay for napp-it development.

There are some Supermicro people on here that I have seen and even @Patrick knows people there. Maybe they could get a part made for you with this software installed?
 

Rand__

Well-Known Member
Mar 6, 2014
4,560
909
113
Very nice, looking forward to see more info on the „Twin vCluster“ :)
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
A „Twin vCluster“ is already inside (a howto and some tests are missing) .
You can set a second cluster control node in settings to manage a second vCluster. Each vCluster has a head and a control VM. Storage in this case is provided by iSCSI, either with dedicated iSCSI servers or a virtual SAN VM on each vCluster for network mirroring between them. This would not only allow a failover between heads but a full vCluster/storage failure.

@MiniKnight
No special hardware needed for a vCluster in a Box. A basic ESXi server with an LSI HBA (the typical AiO setup) is enough to setup a vCluster with shared raw disk mapping of SAS or Sata disks. Maybe you want a little more RAM.
 

Rand__

Well-Known Member
Mar 6, 2014
4,560
909
113
So now the only question remains how to get more than 10G sync speed out of my hw, but thats probably for another thread;)
 
  • Like
Reactions: T_Minus

tjk

Active Member
Mar 3, 2013
285
48
28
www.servercentral.com
Would be great to see this work on bare metal servers! Unless I'm not understanding, your HA functionality requires VMware VM's today, right?
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
Each head needs shared disk/Lun access. The cheapest and easiest way is ESXi. But as the Cluster Control software does not care about, you can basically use barebone server with mpio SAS or FC/iSCSI as well. Shared disk access can and must be configured outside my Cluster management.

At the moment I do all my tests with an ESXi vCluster as I see this as the typical use case (simple, fast and cheap HA)
 

azev

Active Member
Jan 18, 2013
737
207
43
So now the only question remains how to get more than 10G sync speed out of my hw, but thats probably for another thread;)
Is it even possible to get 10G sync speed on non specialized hardware ?? I am getting no where near that with optane zil and bunch of enterprise class ssd.
 

Rand__

Well-Known Member
Mar 6, 2014
4,560
909
113
I dont know, hence the question. But after @gea has found a solution for the HA problem this will be the next major hurdle I think (at least for my search for the ultimate high speed shared VM storage ;))
 

i386

Well-Known Member
Mar 18, 2016
1,995
517
113
31
Germany
Is it even possible to get 10G sync speed on non specialized hardware ??
The optane (900/905p) can do ~1.2 gbyte/s @ 1t/1qd with large io (1mb+) and sequential writes.

The question is what workload; how many io requests, which io sizes, sequential or random writes? :D
 

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
I think you should try selling these to pay for napp-it development.

There are some Supermicro people on here that I have seen and even @Patrick knows people there. Maybe they could get a part made for you with this software installed?
If I would have a wish free from SuperMicro
I would ask for something like Zstor GS-Cube8 Mini Cube Storage Server | Zstor Storage | Produkte-Zstor GmbH - Open Storage

but
- Flex ATX (not ITX) with the option to use uATX maybe with only two Slots out of three to outside
- Case with integrated PSU with one or two silent 120 mm fans for the whole case incl PSU.

Optionally with 3,5". 2,5" or U.2 , full or shared
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
These are the classic hardware solutions for "Cluster in a Box". SM combines two mainboards with shared disk access ex over redundant multipath SAS for both nodes in a single case.

Napp-it would be basically able to use such a hardare setup to manage the node and service failover with an additional control node. My vCluster concept follows the same idea but intends to reduce costs and complexity but maintain most of its performance and options.

My vCluster use virtualized nodes instead of two mainboards and the shared disk options of esxi. This allows the same but supports SAS, Sata and NVMe out of the box on a single server (the usual AiO server).
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
2,502
842
113
DE
vCluster Beta2 is available (napp-it 18.12g dev)

Current state:
Manual Failover between nodes working for NFS and SMB (failover in around 20s)
For SMB it supports failover for local user and AD users connected to the share during failover.

Todo
auto-failover (on tests)

Expect vCluster to next Pro (Jan 2019)
 
Last edited:
  • Like
Reactions: Rand__