Falling From the Sky Part 4 Leaving the Cloud 5 Years Later

[Patrick Kennedy]

Years after leaving the cloud, we share insights into our budgeting and planning for self-hosting and show our cost savings

The post Falling From the Sky Part 4 Leaving the Cloud 5 Years Later appeared first on ServeTheHome.

Continue reading...

 

[gigatexal]

The transparency in that article is legit. Real numbers countering cloud FUD is why STH is great. Keep it up guys!

 

[PigLover]

The analysis and experience here is valid - and should be modeled by others for sure.

But you can't lose sight of the fact that this is possible is due in large part to the very subject matter of STH (and your team's particular expertise). You are able to make rather optimal equipment and deployment choices without much challenge, you can test assumptions about the use of used/recycled equipment in ways that add publication value to the site itself, etc.

Realistically, assuming you are dealing with non-technical subject matter and didn't have this built-in expertise, your likely equipment costs might be double, you'd be exposed to "experts" and salecritters whose interests don't match yours, etc. Or, alternatively, you'd have to pay consultants or hire experts to match your expertise (very likely a dear price).

You've shown what can be done if you know what you are doing - but you don't cover much about the cost of "knowing", which for your special case is not just "free" but in material ways accretive to the value of the STH itself.

 

[zedascuras]

Very good info Patrick, thanks for the article!

 

[gigatexal]

The analysis and experience here is valid - and should be modeled by others for sure.

But you can't lose sight of the fact that this is possible is due in large part to the very subject matter of STH (and your team's particular expertise). You are able to make rather optimal equipment and deployment choices without much challenge, you can test assumptions about the use of used/recycled equipment in ways that add publication value to the site itself, etc.

Realistically, assuming you are dealing with non-technical subject matter and didn't have this built-in expertise, your likely equipment costs might be double, you'd be exposed to "experts" and salecritters whose interests don't match yours, etc. Or, alternatively, you'd have to pay consultants or hire experts to match your expertise (very likely a dear price).

You've shown what can be done if you know what you are doing - but you don't cover much about the cost of "knowing", which for your special case is not just "free" but in material ways accretive to the value of the STH itself.

Agreed. If your core competency isn’t servers and managing and building and maintaining them then perhaps cloud and it’s premium is worth it.


Sent from my iPhone using Tapatalk

 

[msvirtualguy]

Great article Patrick, and you touch on a lot of reasons why companies are moving to hybrid approach, for most it just makes sense.

What we are finding is that traditional steady state workloads encompass about 75% of the workloads out there which are expensive when running those in the public cloud. With these workloads, Public Cloud providers try to alleviate the cost burden with longer contractual obligations such as 3 years instead of a "pay what you use" model. More and more we are seeing customers that have gone "all-in" with cloud are pulling out due to the high cost of running these types of workloads.

The key is to provide a cloud like experience on prem for these types of workloads, with rapid deployment, fractional consumption with all the simplicity and rapid innovation that comes from Public Clouds but still provide the controls of running these workloads on prem, with SLA's, Governance and choice. It's a balance of owning vs renting, same idea when renting/leasing/owning a car or a home.

This is why software is eating the world in the IT Infrastructure space because it can bring these promises to Data Centers, much like Google, Facebook and other "web-scale" companies deliver.

Having said that, of course there are the other workloads that are elastic, unpredictable where the cloud fits very well, spinning resources up and down as needed whether that's a seasonal requirement such as an online retailer during the holidays..etc....or even a real/rapid change in resources, cloud makes a lot of sense.

The key, however, is building a single construct or Operating System in this model that stretches from edge to core to cloud, while providing all the above tenants but making it dead simple, while providing choice both on prem and cloud and making that experience seamless.

 

[Patrick]

Sorry guys. I am in Park City today... learning double black diamonds has yielded a bit of pain and not a lot of STH.

@PigLover - valid, and that is why the IT / VC interaction was mentioned.

@msvirtualguy certainly agree with those points.

BTW - we always are way under budget on what we actually spend when we add recycled gear into the mix. Next will be adding AMD EPYC. The numbers published were extremely conservative.

Also noted in the article, there is still some AWS service usage (e.g. Route53).

 

[MiniKnight]

23-30 VMs is easy for a single server. You aren't using that much storage and BW if you don't do video so your Colo costs are overstated.

You also should have done a snapshot of on demand Instances because your not capturing that in the analysis.

 

[TangoWhiskey9]

More of these articles.

You've got great comments on that one too.

 

[mstone]

But you can't lose sight of the fact that this is possible is due in large part to the very subject matter of STH (and your team's particular expertise). You are able to make rather optimal equipment and deployment choices without much challenge, you can test assumptions about the use of used/recycled equipment in ways that add publication value to the site itself, etc.

Realistically, assuming you are dealing with non-technical subject matter and didn't have this built-in expertise, your likely equipment costs might be double, you'd be exposed to "experts" and salecritters whose interests don't match yours, etc. Or, alternatively, you'd have to pay consultants or hire experts to match your expertise (very likely a dear price).

If you're using EC2 you already need to be paying someone to manage the software--even if the hardware is the cloud. So the cost difference isn't "magical cloud that does everything" vs "people", it's basically "do my people have the cycles and basic competence to rack a server under 'other duties as assigned' or do I need to hire more people to manage equipment". There are a bunch of crossover points where one solution or the other might make sense. But the bottom line is that it doesn't really cost that much to buy a server, and the choice of server really doesn't matter for basic workloads.

Now if you want to talk about self-hosted vs cloud services, that's a whole different thing--and this post doesn't address that at all. In general, I advise most small firms to just do the cloud services: most small shops really can't manage email and even basic MS stuff that well, and the outcomes with a cloud based office are much better than that server in the closet that Bob the consultant set up a couple of years ago and never touched again. For a simple web site it's a no-brainer for a small company to just outsource the thing.

 

[Patrick]

@mstone I tell people that I have more server capacity than most businesses our size, but I will not run a mail server. Google and Microsoft have compelling offerings. I think you have a good point that there are many services that are worthwhile to buy.

 

[Evan]

Yes services are often a good deal, I would it be running mail servers unless you had thousands of users, until that point just outsource it to the experts, eg O365.
$12.5 a month including all the office software is a really competitive price compared to owning software and running a mail service.

Online service I see a big advantage to the cloud, mainly related to security and scalability (if it’s designed well).

Otherwise if you want you own stuff throw a couple of boxes into a colo and just run it as you would on premises.

One word of warning for business is the internet connections then are going to be much more utilized and much more dependent on. Having said this even an office outage can for a short time be worked around by 4g/lte thethering or work from home etc.

 

[eva2000]

@mstone I tell people that I have more server capacity than most businesses our size, but I will not run a mail server. Google and Microsoft have compelling offerings. I think you have a good point that there are many services that are worthwhile to buy.

+1 on 3rd party mail services Google Gsuite is one worth paying for

Thanks @Patrick for the write up. It's always interesting to see your comparisons and the concrete numbers. Even some of the big buys like Dropbox see the light in moving off AWS cloud - $75 million saved over 2 yrs is no small amount Dropbox saved almost $75 million over two years by building its own tech infrastructure !

I guess one thing you might not have touched on would be server performance efficiency and native hypervisor performance over heads. Older AWS EC2 instances used older hypervisors with overhead as high as 50% so would take 2 servers to do the work of 1. Newer AWS EC2 instances like c5 and m5 use newer Nitro hypervisor with overhead as low as 1.5% so near bare metal according to Brendan Gregg from Netflix AWS EC2 Virtualization 2017: Introducing Nitro. The choice of AWS EC2 instance types would factor into your running costs due to instance type's efficiency. I prefer to squeeze out more performance per server so if each server can do 50% more work than default, that means on a 60 server cluster, I can reduce reduce the number of servers by 20 and only use 40 servers instead of 60. With 50% overhead, I would need 120 servers to do the work of 60 default or 40 optimised servers !

Also if you use AWS T2 instances also have to deal with burst vs baseline performance CPU Credits and Baseline Performance - Amazon Elastic Compute Cloud

I'm surprised STH only uses 4TB/month so that certain saves alot of AWS's bandwidth costs at US$90-120/TB per month !

I mainly use AWS for Route53 GeoDNS and S3 storage for backups (2.5TB backups at US$24/month is hard to beat !)

 

[compuwizz]

@Patrick can you elaborate on what all you get for the $600 /mo for the power and bandwidth? Does that include UPS A/B power with generator backup? Do you have more than just Hurricane Electric for bandwidth? How many racks does that include?

I mention this because realistically, most companies will go with at least 2 power feeds, and 2 bandwidth providers to ensure higher uptime.

For those people saying bandwidth is more like $0.20 per megabit, thats not the case unless you're buying it on 10G or higher circuits and committing to the full circuit. Typically providers require a 10% commit on the port that they are buying. We are seeing more like $1.50 - $3 /meg for 100 megabit on a 1G circuit plus cross connect fees.

 

[Patrick]

That is a good question. Realistically, you can use a decent low-cost hosting provider (e.g. HE) and get ~4 nines. So you are correct that the number can be lower and we do not use all of the space/ power/ bandwidth in a $400 HE rack for hosting two dozen VMs and we should instead us a fractional portion thereof. Fair point.

Instead of using an older A+B model which does not help if a facility fails, we have a backup at a different facility with a different mix of bandwidth providers and redundant facility power UPS and etc. This is part of why we are at a 4x overbuild in terms of hardware.

A valid point someone made was that realistically we could instead gain redundancy by simply colo'ing 3x 512GB RAM boxes as POPs at different facilities, spend $100-150/ mo each for 2-3 boxes and manage less infrastructure (e.g. not paying for switches.) That is certainly something we have explored and can do in the future and it would drastically lower costs.

On the $1.50-$3 it may be dependent on where you are. In the Silicon Valley, it is nowhere near that going directly to providers. That is not even getting into hosting companies that are selling oversubscribed bandwidth at a fraction of that. I had a quote a year ago at a quarter end for a large provider that was $650/mo for 1GbE on a 10GbE port billed on 90th percentile.

 

[compuwizz]

Thanks for addressing my questions. It provides a better view of what you're actually getting.

At our colo provider, we've had A power fail for close to an hour but B power remain up during that time. Cooling also partially failed during that power issue. This was in Coresite Reston VA a couple years ago.

I know HE's datacenters tend to go dark every few years. That just scares me since when devices lose power, things tend to have a higher probability of breaking or RAIDs going out of sync.

The bandwidth cost you mentioned is what we see for a 10G circuit with a 1G commit. I was mentioning pricing for a 100M commit on a 1GE pipe 95% billed.

We have thought about doing quarter or half rack colocation and hosting 3 dense servers but then we also lose out on A/B power in most buildings.

You definitely got the wheels turning and I'm thinking about things from a new viewpoint. Thanks!

 

[Evan]

As an enterprise let’s just say I would design redundant systems across 2 tier 3 DC’s over a single tier 4 room !