STH 2013 - Infrastructure Planning

Patrick

Administrator
Staff member
Dec 21, 2010
12,249
5,350
113
Quick request for feedback.

Right now the main site is running WordPress which I really like. Forums are still vBulletin 4. At some point in 2013 we are going to need to move to newer infrastructure. The forums need to move either to Amazon EC2, or a colocated solution in 2013. The plan was upgrading to a vBulletin 5 Connect license.

One other thing I am tossing around is the idea of moving everything to a BuddyPress infrastructure. Social + the main site in one. If I do that, I would likely hire a main site designer and put some resources into the redesign. Also, servethebiz.com may make a comeback.

We have a bunch of cool other projects brewing, but I wanted to see what folks here thought. 2011 was house, 2012 was car, 2013 should free up money for developing STH. Feedback much appreciated.

All of this entails a bit of pain, but I need to keep the thing scalable.
 

gigatexal

I'm here to learn
Nov 25, 2012
2,825
549
113
Portland, Oregon
alexandarnarayan.com
EC2 all the cool kids are doing it and then you can do a writeup of how a successful site is using it. :p

but seriously I'd be interested in the financial side of it as I am keenly interested in the ability to be billed on only what you use.
 

Mike

Member
May 29, 2012
482
16
18
EU
This being a hardware site i wouldn't even think about Amazon. A small 2 server cluster is fun, flexible and pretty safe. Uptime will always be a guess, amazons refunds sure shouldnt keep you with em.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,249
5,350
113
EC2 ....your going to either outgrow your hardware or your colo space.
I've "outgrown" EC2 instances twice in the last 18 months. Colo would be a jump to something 10x faster. Next step is basically moving to a cluster with a HA database and then spinning up instances to meet demand as needed. That doesn't necessarily get better performance since it introduces more latency in the equation versus going with a faster box. Also it is a total PITA to manage.

Gigatexal - one thing I have learned is that with EC2 paying for what you use is tricky with a website. There is a minimum level of resources you need to be online at any given time.

Mike - Amazon's uptime plus not having to deal with that piece saves me a lot of time.

Spinning up a medium instance for the forums would be an option, but that plus main site hosting will be more than 60% monthly of a 1/4 cab even without growing the main site instance.

Probably going to do some testing to see what I can get away with. Moving to BuddyPress would lead me either to Amazon bigger instances with HA RDS or colo as that would be one set of resources to manage.
 

dba

Moderator
Feb 20, 2012
1,478
183
63
San Francisco Bay Area, California, USA
For a very small installation that does not have large usage "peaks" that would make dynamic provisioning worthwhile, I begin to doubt that EC2 would make financial sense - especially for someone whom I suspect has enough hardware laying around that their next few servers have near zero incemental cost. It might be educational to turn your infrastructure into a clustered setup utilizing a number of different EC2 instances, but that is probably even less cost effective at small scale.

How about a colo, with a pair of VM servers? You'd really only need one VM server, but a pair would provide some resilience. Bonus points for using some type of shared storage and creating a full VM cluster. If LSI won't let you beta test their Synchro solution for HA-DAS, then you could build your own shared storage based on iSCSI or SRP (Infiniband or 10Gbe Ethernet hardware). I'm thinking something like two 1U VM servers connected to a 2U DIY storage server stuffed with 12 disks plus a few SSD cache devices. Total rack space is 4U and total power is probably between 7 and 10 amps @ 120V. Granted this is far more horsepower than you actually need...

I've "outgrown" EC2 instances twice in the last 18 months. Colo would be a jump to something 10x faster. Next step is basically moving to a cluster with a HA database and then spinning up instances to meet demand as needed. That doesn't necessarily get better performance since it introduces more latency in the equation versus going with a faster box. Also it is a total PITA to manage.

Gigatexal - one thing I have learned is that with EC2 paying for what you use is tricky with a website. There is a minimum level of resources you need to be online at any given time.

Mike - Amazon's uptime plus not having to deal with that piece saves me a lot of time.

Spinning up a medium instance for the forums would be an option, but that plus main site hosting will be more than 60% monthly of a 1/4 cab even without growing the main site instance.

Probably going to do some testing to see what I can get away with. Moving to BuddyPress would lead me either to Amazon bigger instances with HA RDS or colo as that would be one set of resources to manage.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,506
233
63
It depends on what you want to do. How much storage do you need? How much bw? How much CPU?

Other items to stoke thoughts --- Are you going to re-design STH? I see the new logo and notice tweaks, but is there any notion of moving to non-WP? What about a new theme? How well integrated is everything? Do you have lots of miscellaneous stuff around? Have you done basic stuff like nginx and varnish or just Apache? How fast are things running?

Biggest question --- Are you still servethehome? You mention servethebiz.com which is a better domain name for you at this point. How expensive is that Supermicro server reviewed this week?

If it were me, I would decide BuddyPress or vB or something else. I would then decide whether I wanted to create ServeTheBiz.com on the new platform and integrate there. Seems like the site has good Google rankings, but might be the case where in 3 years servethehome.com is mostly archive and forums for IT professionals and developers building home systems while servethebiz.com is considered the main site.
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
12,249
5,350
113
Great questions.

dba - Assuming hardware is "free" I think you are right. Right now hosting costs are about the price of colo-ing two 1U machines. Moving forums to AWS basically is worth about 2.5U and 2.5A. Actually was considering a 5A max setup. Storage needs are actually very low. So 2x 256GB Samsung 840 Pro or 830 SSDs would be fine. The big thing there is the server goes from <150 IOPS disk to much more than that. I also get more CPU and more memory. Right now 5-6GB RAM footprint could go much higher with stuff lying around. Was thinking 2x 256GB SSD RAID 1 and 2x spindle disks (backup only) RAID 1 in each box.

Jeggs101 - Forums are on Apache but the main site is nginx no varnish. Right now in terms of architecture, Wordpress/ BuddyPress would be OK. vBulletin I am fine with for forums, but not sure I want to invest in making a full vB Connect site at this point and migrating current data. As far as ServeTheBiz.com is concerned, actually odditory had a similar thought two years ago almost to the day on the old phpBB test forum. It lasted a day or two then I bought the vB 4 license and made these forums. As far as target audience goes, you may be correct though. Maybe the answer is to do a BuddyPress test installation on ServeTheBiz and see what that looks like. May just build, stick a bit of content up and see what happens.
 

Tim

Member
Nov 7, 2012
105
6
18
Speed, for all parts of this site is good (frontpage, articles and forum).
As an enduser I don't experience any speed related (or other) faults with servethehome.
So any upgrade needed based on speed must be because you see that todays traffic is near a limit of your resources.

I see this site and forum as a place for the high-end home users with an interrest in servers (both hardware and software).
And not the regular user. They've got lots of forums and sites to choose from.
This means that hardware and software come at a cost, but there's alwas a sweet spot when the price goes from (even pro) home setups to become something for a company budget.
So if the reviews are going to stretch the budgets too far it would stop beeing interresting for the high-end home users. (at least to buy it, we can always do some window shopping and dream).

I don't think we're many at this high-end home usergroup, and that's why there's so few of this good forums like servethehome (I don't know of any other this good and friendly).
I think it's important to continue the servetheHOME but I do see the points of starting up a servetheBIZ site/forum as well.
And it might be easier to be a biz site, with room for high-end home users, then a high-end home user site with room for the biz part.

All I can say is thank you for this site, the reviews and articles and the great forum.
And I hope it continues.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,249
5,350
113
Thanks Tim! Feel free to bring others around. Great feedback. It is working "OK" now but after a few, incidents, this year. And those were the ones that I didn't catch quickly/ have an easy fix for. Trying to get this to scale so 2013 is better. Forums had three major incidents also.

http://www.ServeTheBiz.com is officially in exploration status (right now has BuddyPress installed.)
 

cactus

Moderator
Jan 25, 2011
830
77
28
CA
I agree with dba, colo-ing two boxes sounds like a good path forward. Even if you must buy new, a couple 1u servers are not hugely expensive. It gives you a lot of room to expand and HA to an extent. If you are concerned about a regional blackout, put the second box in a different region.

I don't feel the main site has content that will drive away readers, even if you are doing more business targeted articles. The businesses I assume servethebiz would target are on the smaller side anyway and, from my experience, small business goals are closer to prosumer or enthusiast than enterprise. Is maintaining both sites out of the question? I realize much more content is needed, especially to hit critical mass to keep people returning to the new site. I would think having the sites linked would, at least at the onset, add some credibility to the new site.

If you are sticking to vB, I say keep a single forum, sub-forums will keep it segmented. This goes back to getting to a point where there is enough content to keep people coming back. I have visited enough niche forums to know they are hard to keep alive.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,249
5,350
113
So minor development:
1. http://www.ServeMy.Biz has a development site now. Much shorter URL and instead of STB it will be SMB :)
2. cactus totally right on keeping STH going. The re-design I'm working on might help a bit. The big thing is that it requires a lot more server resources from what I am seeing. Good thing is that it will make for a functional upgrade and hopefully add more users to the forum.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,249
5,350
113
Overall... this turned into one of those days where I was unhappy.

Looks like the main site EC2 did a reset thanks to a backup. When the server came back up, nginx was unhappy. Still working on thumbnails.

Here we go colo.

Also did a bit of work on the new STH look: http://www.servethebiz.com (dev server)
And the new site http://www.servemybiz.com

Update: Steven at Rack911 is a miracle worker for stuff like this.