Hyperconverged Home Lab 1.0

[msvirtualguy]

Disclaimer: Why Nutanix? I work for Nutanix as an SE.

Build’s Name: SPARTA
Operating System/ Storage Platform: Nutanix Community Edition ce-2015.11.05-stable (Acropolis 4.5)
CPU: 6 x Intel Xeon 5620 Westmeres (important to include AES-NI Support with NutanixCE)
Motherboard: Supermicro X8DTU-6TF+-LR x 3
Chassis: Supermicro CSE-119TQ-R700UB
Drives: 6 x Samsung 480GB 843T's/18 x 1TB Seagate 2.5" Hybrids (Leftover from other project)
RAM: 288GB DDR3 ECC Registered (36 x 8GB)
Add-in Cards: NONE - Built in LSI2108 and 2 x Intel 10Gb SFP+
Power Supply: Dual Supermicro 700W Gold per chassis x 3
Other Bits: Dell X1052 Switch (on order, replacing dual HP 1810-24G's), Cisco TwinAX, Supermicro 19u RACK, UPS - Haven't decided yet, Monster PDU, Ubiquiti Edgemax Router

Usage Profile: This will be for home lab usage to continue my Linux and OpenSource OS learning as well as hosting some prod VMs like the Ubiquity Controller, Blue Iris Video Surveillance software, etc.

Other information… Pics to come soon as I build out the solution and as the parts arrive.

Progress:

12/5/15-Rack and Nodes arrived. Still waiting on :
Dell x1052
2 x TwinAX Cables
12 x 8GB DDR3 ECC 1333 - 1 node missing memory, have temporary in there to test and burn in
1 x USB 3.0 16GB Sandisk

Need to purchase:
UPS
Another node in remote location for backups (Will run in guest backups to NAS temporarily)

12/11/15- Everything has arrived. Just unboxed the Dell X1052 and getting ready to configure.

This weekend will be cabling, and the NutanixCE install which I will outline. If you were wondering about the NUC, that's actually running 2012 Server Core for Domain/DNS services to the lab.

12/14/2015: The experience with the Dell X1052 has not been good. More than likely I have a defective switch. The switch reboots on it's own out of the blue sporadically. The interface is slow, etc, but not sure if that's related to the actual issues i'm having. Right now, the jury is still out but will update this post after my tech support call. On the phone right now.

Update: Switch is defective. Dell is shipping me a new one, however I have to wait a week for new and not refurb. No biggie, these things happen. Will update the post probably on Christmas Vacation.

12/18/2015: New replacement switch was actually overnighted and i'm installing now. I'll update the post soon with the progress but things are looking much better so far.

1/13/2016: I've added more build log info to my Blog. You can view here.
MSVIRTUALGUY

 

[Lost-Benji]

CPU: 6 x Intel Xeon 5620 Westmeres

Might be a tight fit in a 2P system.

 

[CreoleLakerFan]

CPU: 6 x Intel Xeon 5620 Westmeres

Might be a tight fit in a 2P system.

I got the distinct sense he is planning on three of these.

 

[msvirtualguy]

 

[snazy2000]

Would love to have a nutanix lab !!

 

[msvirtualguy]

You can, Community Edition is available for free.

Community Edition - Nutanix

 

[whitey]

You can, Community Edition is available for free.

Community Edition - Nutanix

Will the community edition run a 'nested' infrastructure? Assuming yes since vESXi is easy-peasy and Nutanix would be mgmt-plane correct?

Can you speak to the limitations or feature set availability w/ the community edition say compared to whatever is the crown jewel in terms of licensing or a license comparison chart would be great.

EDIT: NM just found it, time to RTFM

Software Defined Storage | Hypeconverged Software

Now I just need some more resources, sonofa...where's my 3-node Xeon-D setup when I need it...I am hampered by socket 1155/32GB max memory per node setup currently. BOOO

 

[whitey]

Just read up a bit on Nutanix licensing and I ain't gonna lie...that made me cringe a bit. The editions are nice, cool to roll in DR/Compresison licenses to pro but term-based v.s. perpetual licensing 'tied to phys HW' gives me the heebie jeebies :-D

I bought it, I own it, non-expiring :-D, once my HW sunsets or the release of the major platforms moves on (and I inherenelty want/need newer HW to run the updated stack on) as long as I am paying for support/entitlements I should be allowed to use my lics 'however' I please.

We saw the vTax revolt on memory limits in vSphere...this should be fun :-D

2cents

 

[msvirtualguy]

Community Edition essentially has most of the features of the Production software, however, it's fundamentally packaged differently meaning it's a single install with our Acropolis Hypervisor (no support for vSphere or Hyper-V like Prod) and it uses LUN passthrough instead of PCI passthrough so performance will not be as good, queue depths will be much less. They say 4 drives is the sweet spot per node, but I run 6, 2 x SSD and 4 Spindles. This is done to accommodate much more hardware compatibility. I also ran a single node "cluster" all flash and it performed very well, which leads me to the next difference, Community Edition only supports 1,3 or 4 node clusters.

The goal is essentially to provide an easy way to show you the features and functionality, not so much about performance.

This can be run nested, on Ravello, non-enterprise grade hardware, or enterprise grade hardware like I'm using. We have users using Mac Mini, etc.

Honestly I don't run into complaints about our licensing model, really because look at what you are getting, even in Starter.

Nutanix is not just a Hyperconverged company. We aim to tackle 3 main data center pain points:

1. Legacy three-tier mgmt, forklift, simplicity, etc...FULLY integrated and dead Simple
2. Hypervisor as a feature not a separate product (no shelfware, vTax, etc)
3. SPoG MGMT, no "bolt on" from aquisition, just teams of talented developers that truely integrate into the platform.

Our vision is to provide "Invisible Infrastructure" using intelligent software to do a ton of work behind the scenes so you don't have too. Some of the things on the roadmap that I wish I could talk about are simply amazing, especially around application mobility.

 

[snazy2000]

Can you use it on just two physical nodes? or does it require 3 physical nodes? Shame it doesn't work with Hyper-v

 

[msvirtualguy]

Community Edition (Non-production for PoC) only supports our Acropolis Hypervisor, and believe me, you will be impressed. You can have a 1, 3 or 4 node cluster. The single node provides no Data Resiliency. You can run this on just about any hardware within reason, nested or Ravello. The hardware requirements are listed in the link I provided.

Production Nutanix Software (XCP - Xtreme Computing Platform) supports vSphere, Hyper-V and Acropolis on prem. and must start in a 3-node config because we default Data Resiliency (Replication Factor 2) for all storage. From there you can scale any node count from one to many at a time, linearly.

 

[whitey]

Is Acropolis KVM re-spun or a complete hypervisor re-write/in house product? I keep seeing video's saying vSphere, Hyper-V, KVM support and then you hear about this Acropolis which threw me for a loop.

 

[msvirtualguy]

Acroplis is our turnkey hypervisor based on Open Source KVM, integrated into our platform, and managed through our mgmt tool, Prism. We offer full REST API and CLI support but if you want to use VIRSH outside of those things, have at it.

It's fully supported by Nutanix (on hand to shake), zero licensing overhead, and since we own the entire stack, it's fully secure using our custom STIG, and FIPS compliant which is also hardened by SaltStack orchestration and automation.

Within the same interface, Prism, you also have Image Services, VM mgmt, Distributed Switch with OVS with advanced features like IP pooling..etc.

http://go.nutanix.com/rs/nutanix/images/nutanix-acropolis-virtualization-datasheet.pdf

 

[TuxDude]

... zero licensing overhead ...

Nutanix can do some pretty cool things, but the production version might be the single most overpriced piece of software I've ever seen. The last quote I saw (list prices, not what we will end up paying) wanted just shy of $300000 per node, while I could purchase the identical hardware for a tiny bit over $60000 (also list price) - there is definitely some licensing overhead in there adding $240000 per node and it didn't even include the hypervisor/guest OS licenses. And it gets worse when you consider that in 3 to 5 years when the hardware needs upgrading/replacing there is no upgrade path for that license - you throw it away with the node it was purchased with and buy a new one.

 

[msvirtualguy]

Nutanix can do some pretty cool things, but the production version might be the single most overpriced piece of software I've ever seen. The last quote I saw (list prices, not what we will end up paying) wanted just shy of $300000 per node, while I could purchase the identical hardware for a tiny bit over $60000 (also list price) - there is definitely some licensing overhead in there adding $240000 per node and it didn't even include the hypervisor/guest OS licenses. And it gets worse when you consider that in 3 to 5 years when the hardware needs upgrading/replacing there is no upgrade path for that license - you throw it away with the node it was purchased with and buy a new one.

I would be very surprised to see $300k per node..even at list. Of course there are licensing costs, Nutanix is a software company. The licisnsing I was referring too was the Hypervisor. The introduction of Acropolis Hypervisor has not increased the cost.

I would love to see that config at $300k per node list. We have solutions ranging sub $100k on up and remember you're getting three nodes to start.

On top of that there are operational savings that make our solution very competitive.

 

[TuxDude]

I would love to see that config at $300k per node list.

The nodes were Dell XC730's, configured with 768GB RAM, dual E5-2690v3 CPUs, 2x 400GB SSDs plus 12x 2TB 7K drives, and a pair of grid cards. And the numbers are in $CAD which is normal for me, but makes numbers look bigger for people south of the border.

 

[msvirtualguy]

That can't be right..if you want me to look into it, let me know. I just put this together without the Grid cards and single node price is no where near $300k. Please keep in mind that standard deal reg pricing is much less than list. Heck when I worked in the partner space the deal reg delta between list was upwards of 45%, etc for some vendors.

I'd like to not use this forum as a place to do Nutanix business as it's not only inappropriate, it's probably against the rules, so if you wanna email me and look into it, please let me know:

msheehy@nutanix.com

Apologize in advance to the admins. All future responses to this post will be Home Lab related.

Thanks

 

[TuxDude]

I'm not concerned about it - most of the price is due to the industry-wide tendency to publish stupidly-bloated list prices that no-one ever pays. Maybe if I ever build an enterprise IT product I'll list it at 10 billion dollars, so then I can have an even bigger discount % than everyone else and still sell it for 10x what its worth.

Sorry for derailing the thread a bit - software licensing in general tends to be a topic that makes me mad. Please return to your regularly-scheduled home lab.

 

[snazy2000]

Shame you cant do two nodes so that out of the question lol

 

[msvirtualguy]

Shame you cant do two nodes so that out of the question lol

We are listening, and understand. I won't go all out and say anything on this yet.