Alternatives to vmWare vSan for hyperconverged environment (Home)

Rand__

Well-Known Member
Mar 6, 2014
4,491
877
113
Hi,
I am currently running a vmWare environment at home using vSan for shared storage and Horizon for VDI. I also have a bunch of VMs running (<20).

This is licensed via VMUG so 200 bucks a year, my vsan is based on All-Flash (P3700 & P320h as cache, 750's as storage), currently 3, soon to be 4 nodes.

Now I know that hyperconverged is primarily targeting enterprise users where its more important to have a consistent and equal distribution of performance to a larger user base opposed to peak performance of a few users.


I am aware that given the low node &disk amount (1 cache+1 capacity/node) the most I can get is 1 disk's worth of performance, but at least for write I am not even getting that.


I have run a few tests (I know CDM is not really the best tool for this, its just for illustrating the difference between a local ssd and the vsan).

Its the same VM moved to different hosts (with different cpus and cache drives)

The local drive on a 26667 (2 cores):
38_local_s3510.PNG

Vsan based vm on the same box, with a P320h:
38_vsan_p320.PNG

Vsan based vm on a 2630L with a P3700
37_vsan_p3700.PNG

And lastly, to test whether this really might be due to the multi user design, 8 concurrent CDM runs on the 2667/P320h (with 8 cores). Note that CDM did not run all in parallel, it grouped as marked with a delay of maybe 15 secs total. That means the last CDM run finished 15 secs after the first one, so some tests were not running in parallel, at some point there was mixed read/write going on.
Yes I know, "run a proper test using fio" or such :p
38_vsan_concurrent.PNG



I can live with the current setup, but I wonder whether one of the alternative options would provide more individual performance than vmWare/vsan. O/C its totally possible that my (unsupported HW) setup is just not well built and this is just a config issue.
 
  • Like
Reactions: james23

cheezehead

Active Member
Sep 23, 2012
717
174
43
WI
VMware/VSAN is one model

FOSS would be a Proxmox VE/Ceph build

Windows-centric would be Hyper-V/Storage Spaces

FreeNAS might be there at some point by Coral got axed and the current Bhyve implementation is more a POC currently IMO.

Also with VMware, there are other storage platforms available...if it's Proliant gear then you should have the free licensing for their StoreVirtual (aka Lefthand rebanded product).
 
  • Like
Reactions: james23 and Evan

Rand__

Well-Known Member
Mar 6, 2014
4,491
877
113
Yes - is any of these alternatives configurable for "few users / high performance" opposed to enterprise "many users/throw more hardware at me" ? :)
 
Last edited:
  • Like
Reactions: dicecca112

Evan

Well-Known Member
Jan 6, 2016
3,065
512
113
VMware/VSAN is one model

FOSS would be a Proxmox VE/Ceph build

Windows-centric would be Hyper-V/Storage Spaces

FreeNAS might be there at some point by Coral got axed and the current Bhyve implementation is more a POC currently IMO.

Also with VMware, there are other storage platforms available...if it's Proliant gear then you should have the free licensing for their StoreVirtual (aka Lefthand rebanded product).
Other vSAN's or similar commercial products such as scaleio, starwind, nutanix, simplivity, storpool, hedvig, whatever IBM call gpfs these days.

I am disappointed that FOSS options (ceph) are so limited at the small end, at the high end number of nodes it's more suited.

Of course there is many "failover" options available as well which for home use situations maybe easier to manage as well.
 
  • Like
Reactions: NISMO1968

dicecca112

Active Member
Feb 10, 2016
300
35
28
35
Yes - is any of these alternatives configurable for "few users high/performance" opposed to enterprise "many users/throw more hardware at me" ? :)
Bingo, been hunting that too. I'll be watching this thread carefully.
 

Rand__

Well-Known Member
Mar 6, 2014
4,491
877
113
Yes, I am aware there a lot of similar solutions :)
But which one of these caters for the requirement of "great performance for the few" ?
I can't believe that all except vmware/vsan are top performers for small work-groups :D
 

Evan

Well-Known Member
Jan 6, 2016
3,065
512
113
I would say none of them.
Some of them can perform better if you put lots of disks on a small number of nodes but generally nothing works that well at 2 or 3 nodes from what I see, of course that's not to say it's terrible but simple fact is 2 or 3 nodes will never perform better than 1
 

Rand__

Well-Known Member
Mar 6, 2014
4,491
877
113
And thats perfectly fine - what I am looking for is a solution that offers vSan like capabilities (real time synchronisation of vm data, HA/DRS, dedup) with at least 70% of the performance of a single node with a 3 or 4 node setup.

I totally understand that with the low number of nodes only resilience will be improved and not speed because data cannot be striped and mirrored at the same time with these few nodes/disk groups.

But I do expect to get the better part of that single node's performance - ie I'd hope to get (70%) Cache disk write speed and Capacity disk read speed

This is a 750 CDM from another box (on a 400GB model) - thats significantly better than I get from the vSan datastore (especially 4K)
750_2_CDM.JPG

So either my vsan config is crap and I should get better speeds with what I have
or vSan is not cut for "great performance for the few" and limits the resources to a single client to be able to serve many

Edit:
Hm it might be my box after all - its difficult to find some small setup comparison values but this guy here gets ~ similar results with much cheaper hardware (1 ssd, 2 spinners)

Edit2:

Ran a Storage test and that looks fine... cant copy/export it for whatever reason, so partial screenshot only:
vSan7030.PNG

30 vmdks total (10 per host), so conservatively its 30x50 MB/s which is 1500MB which is totally ok
 
Last edited:

NISMO1968

[ ... ]
Oct 19, 2013
78
13
8
San Antonio, TX
www.vmware.com
ScaleIO is extremely slow unless you use 10+ nodes, and it isn't free for production use.

https://www.emc.com/content/terms/eula-scaleio.htm

"E. “Internal Business Purposes” means an internal (non-commercial) Use for the purpose(s) of testing and demonstrating the features of the Software, and not for Customer product development, product testing, or other Customer research and development or commercial purposes."

I know it's a home lab but why would you want to learn anything you can't use @ long run?

Scale-io, nutanix, proxmox
 

NISMO1968

[ ... ]
Oct 19, 2013
78
13
8
San Antonio, TX
www.vmware.com
Did you get a chance to get hands dirty with SimpliVity VSA? I only heard it exists, but it was never ever made public AFAIK.

P.S. Nutanix software-only comes with their own KVM fork-out, so it isn't a SDS either. FYI.

Other vSAN's or similar commercial products such as scaleio, starwind, nutanix, simplivity, storpool, hedvig, whatever IBM call gpfs these days.

I am disappointed that FOSS options (ceph) are so limited at the small end, at the high end number of nodes it's more suited.

Of course there is many "failover" options available as well which for home use situations maybe easier to manage as well.
 

Evan

Well-Known Member
Jan 6, 2016
3,065
512
113
Only seen simplivity with demo done by their staff, could tell it did not really bring anything for the company I work for as it would mean more products and processes rather than every site and install being same.
 

Rand__

Well-Known Member
Mar 6, 2014
4,491
877
113
So from the other thread it shows Nutanix won't cut it (for my specific use case).
So how about Promoxx? How is the performance with few fast hosts?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
You can't do HA on ProxMox without shared storage :(

If it let you do it with localized it would be very fast even with ZFS since it's baked in, and no network sharing.

I hope they offer HA with local storage soon, I would configure for that! For now I'm just using ZFS+Proxmox and doing backups.
 

Rand__

Well-Known Member
Mar 6, 2014
4,491
877
113
Ah thats why its always ProxMox+Ceph
So how is Ceph then in terms of performance?
 

NISMO1968

[ ... ]
Oct 19, 2013
78
13
8
San Antonio, TX
www.vmware.com
You need quite many nodes to assemble a reasonable I/O bandwidth. Replication is OK-ish but you need erasure coding to gain proper fault tolerance and space savings and E/C is... slow :( I'd say Ceph is OK for a big data dump (CERN and NSA use it for exactly this purpose) but for VM storage it misses quite a lot. IMHO.

Ah thats why its always ProxMox+Ceph
So how is Ceph then in terms of performance?
 

Evan

Well-Known Member
Jan 6, 2016
3,065
512
113
Hyper converged sounds like the perfect solution for small installs but in reality it seems to fall short unless you actually have bigger installs of at least 4+ nodes which for most people means license costs that don't make all that much sense.

2 nodes solutions (or 3 if the 3rd node can run on a really skinny config without storage) that actually work seem to be few any far between.
 

Rand__

Well-Known Member
Mar 6, 2014
4,491
877
113
Ok, so looks like I better try to coax some more performance out of my vSan setup then ...
Thanks
 

NashBrydges

Member
Apr 30, 2015
85
24
8
53
Starwind has recently removed all restrictions on their free version so this might be something worth investigating. You can now scale up or scale out without cost.