ServeTheHome 2013 Architecture Discussion

[Patrick]

Single L5520 for pfsense(or similar) load balances?

The original plan was to use two Atom based pfsense boxes. The S1260 based Atom machines I can't get in time. Also I need two NICs (one to each switch) for data and one for heartbeat. The new SM offering has a dedicated IPMI port but you need to use PCI for the third NIC. Buying the D525 based machines would work well, but at around $350 with tax and ship each + memory + a 64GB SSD + shipping to the DC they are at minimum a $450/ea proposition. Conversely, my hardware cost for adding 2x single L5520 Dell C6100 nodes is about $0. Bringing the second chassis online would cost around $50/mo to be safe power wise. I think there will be a lot of idle time so probably do not even need this, but just in case. $900/50 = 18 mo payback plus I also have the capability to have a second hot swap chassis online or have a hot spare node if necessary. Space wise both solutions take 2U (2x 1U Atom machines or a second C6100 node.)

Still mulling it a bit, but that may end up being the pfsense/ HAproxy solution.

 

[cactus]

You can also split your HA nodes between chassis. Have you dont any power testing with <4 active nodes?

 

[Patrick]

You can also split your HA nodes between chassis. Have you dont any power testing with <4 active nodes?

Was thinking the same thing. Even with the single L5520 should lower power consumption. May try two shells with two nodes each. The only thing there is that you are still running fans regardless so not sure how much it will help.

Even with 3x C6100 (two online shells and one cold offline) and two HP switches I'm still at 8U which is not bad.

 

[Patrick]

Another thing worth mentioning is that cables for this actually are a decently large cost.

$5/ cable * 25 cables = $125 + tax.
Also bought 4 power cables for $25 shipped.

Making final preparations to ship. Need to get out by Wednesday in order to make it to the DC by Friday. Then hopefully Saturday installation.

 

[PigLover]

You know we all expect pictures! And yes - will will judge you for how you dress out the cables

 

[Patrick]

You know we all expect pictures! And yes - will will judge you for how you dress out the cables

That's the goal. If all goes according to plan. Wednesday is UPS/ FedEx day. Friday the boxes are delivered. Saturday I should be there working on everything. I may have the team at the DC help get things setup until I get there.

Oh and /afk superbowl Go 49ers.

 

[Patrick]

Well... the superbowl thing did not go as well as planned.

Tickets booked for Saturday to Las Vegas. Did burn-in this weekend. Very excited.

 

[cactus]

Well... the superbowl thing did not go as well as planned.

49'ers secondary could have shown up.

 

[PigLover]

With 48GB (or more with your recent purchase) how are you dealing with the 32GB limit for the ESXi nodes? Did you go to a licensed configuration?

 

[Patrick]

With 48GB (or more with your recent purchase) how are you dealing with the 32GB limit for the ESXi nodes? Did you go to a licensed configuration?

Absolutely great question that I'm still not 100% sure on. The current thought is to use HAproxy/ pfsense nodes in front of the servers. Good chance we will use something like Proxmox to manage the VMs.

The cloud architecture is a bit of a tradeoff. They are still all much more complex and each component/ service is something else that can fail. It may be the case where we have the hardware but the software is an issue.

ESXi is likely out at this point. 48GB is beyond the ESXi 5.1 free mode and VMware licensing costs too much for this type of installation unfortunately. The basic tenant of the build is that the hardware needs to be a commodity and the software needs to run on anything I add into the mix.

 

[nitrobass24]

Have you considered Citrix & CloudStack?

 

[MiniKnight]

Have you considered Citrix & CloudStack?

What about stackops? and opennebula?

Found this - http://talkincloud.com/cloud-computing-management/openstack-vs-cloudstack-latest-score

 

[PigLover]

VMware is such a well designed and simple hypervisor. It's a great tool. Its really too bad that they have priced themselves out of the SOHO and Small/Medium business segments. I'm sure it would be possible to find a pricing model that - for a modest license charge - lifts the memory cap. I know they are afraid of revenue dilution if too many of their enterprise customers go down that road, but they could solve that by not offering any materiel support at that price.

Heck, if they do it right they could even make it revenue accretive. I know I'd pay them $100 to get over that cap. And with memory getting cheaper and large-scale systems getting more common in SOHO/SMB environments their potential marketplace grows over time if they hold the entry-line at 32Gb.

 

[Jeggs101]

VMware is such a well designed and simple hypervisor. It's a great tool. Its really too bad that they have priced themselves out of the SOHO and Small/Medium business segments. I'm sure it would be possible to find a pricing model that - for a modest license charge - lifts the memory cap. I know they are afraid of revenue dilution if too many of their enterprise customers go down that road, but they could solve that by not offering any materiel support at that price.

Heck, if they do it right they could even make it revenue accretive. I know I'd pay them $100 to get over that cap. And with memory getting cheaper and large-scale systems getting more common in SOHO/SMB environments their potential marketplace grows over time if they hold the entry-line at 32Gb.

I would like to see a la carte. Want a 33-256GB server? $50. Want HA? $50.

Let us pick features and use ESXi.

 

[PigLover]

I agree that full ala-cart is probably what you and I would want, but that is a model the existing enterprise customers could use to slice off pieces that they don't use. As a business with an existing revenue stream VMware needs to be careful that they don't cut their own arteries in order to open a new market.

Sorta like airline super-savers - they need to find a way to sell the "unused seats" (customers like Patrick, you and me) who just want the hypervisor without making that an attractive way for their existing customers to by the product.

 

[nitrobass24]

Problem w/ VMware and other "cloud" solutions is that they rely on the vCenter Server, which you cannot get right now without an Enterprise License. So even if ESXi had a 64gb cap, it wouldnt do you any good, because you still don't have vCenter.

 

[PigLover]

Problem w/ VMware and other "cloud" solutions is that they rely on the vCenter Server, which you cannot get right now without an Enterprise License. So even if ESXi had a 64gb cap, it wouldnt do you any good, because you still don't have vCenter.

Only partly true. If all you are trying to do is manage a small number of hosts (say 4, like a single instance of a C6100/6220) and a few VMs on them vCenter is not all that important. You can certainly manage a simple, stable environment through vSphere client.

Yes - I agree - if you want to use vMotion, HA & the other orchestration tools, then its is unmanageable without vCenter. But a simple SMB who just wants to manage a few apps onto a small number of host instances is completely workable without it. Patrick's application is a perfect example.

And - if you are VMware - that simple environment is a marketmaker for you. Those SMBs that grow will discover the benefits of orchestration. And with those benefits comes license revenue to VMware.

To Patrick: sorry for the threadjack. If this dialog continues I'll open a new thread for it so that this one can stay focused on your architecture for the site.

 

[nitrobass24]

Only partly true. If all you are trying to do is manage a small number of hosts (say 4, like a single instance of a C6100/6220) and a few VMs on them vCenter is not all that important. You can certainly manage a simple, stable environment through vSphere client.

Yes you can manually manage it, but thats not what were talking about. A cloud product like cloudstack, xendesktop, etc. has to have an interface to communicate with the underlying hypervisor on the physical host. In the case of VMware, this is vCenter. If you try to manually use the v-sphere/host address it will not work. Same goes for Hyper-V, in a lot of cases you need to have SC2012 VMM running for something else to manage the hosts.

 

[Patrick]

Yes you can manually manage it, but thats not what were talking about. A cloud product like cloudstack, xendesktop, etc. has to have an interface to communicate with the underlying hypervisor on the physical host. In the case of VMware, this is vCenter. If you try to manually use the v-sphere/host address it will not work. Same goes for Hyper-V, in a lot of cases you need to have SC2012 VMM running for something else to manage the hosts.

This is true.

Longer term, the site is going to have 8 nodes, with 2x UP pfsense nodes that should probably be excluded (12GB single L5520 right now).

Going to try StackOps one more time tonight, but still a painful installation and setup.

The other issue is if you look at StackOps most people doing a "fast" installation are using a single node server. Here's a YouTube search on StackOps. That's supposed to be the easy version of the OpenStack install so nobody seems to be using multiple nodes.

 

[nitrobass24]

Yea if you want open stack, stackops is the best choice, but its still a PITA. And I dont think you really get much advantages. NAS/SAN integration is difficult at best.

Eucalyptus - Havent Tried
CloudStack - Relatively Easy to Install - I got lost in the networking though. Requires either Citrix, KVM, or ESX w/ vCenter
OnApp - Love the interface and iphone apps, really cool product, but it is expensive. Min $500/mo 6k/yr.
ProxMox - Havent Tried