ServeTheHome 2013 Architecture Discussion

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
For those wondering here is the current thinking in terms of STH architecture for 2013. The big reason for doing this is to improve performance and build things like high-availability.

Colocation
1/4 Cab - 5A power 30-100mbps

Components
Dell PowerEdge C6100 (these are rebranded Supermicro 2U Twin2 machines) Here is an example.
  • Four nodes
  • Dual L5520 in each node
  • 24GB or 48GB RAM for each node
  • 2x Samsung 830, 840 Pro or Intel 710 / 320 SSDs in each node
  • 1x 2TB SATA disk per node* (Optional)
Switch
iSCSI server (if I go with OnApp) - 3-4 bays max. Possibly would allocate one of the Dell C6100 nodes for this and remove a CPU from that node.

Overall Environment
Proxmox or OnApp "free" edition

Will post diagrams and such soon. Open to thoughts/ suggestions.

Update 24 Jan 2013
Here is the current architecture and hardware notes. Very close to final.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
I like the C6100 idea, and it looks like you found a great deal on eBay. With so many C6100s for sale, my guess is that these were popular last-generation VM servers, which is a vote of confidence for your use of them as a micro-cloud.

What do you plan to do for backups? Can you add amps in small increments should you wish to grow? With a C6100 plus a 1U storage server you'll have plenty of extra physical space but not that much reserve power.

...$1K for four ready-to-run dual-CPU servers with reasonable power consumption? Must resist...
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
The C6100/ 2U Twin2 from what I understand was a very popular platform for cloud providers so these were purchased in big quantities. The L5420 generation is very inexpensive also. Mr Rackables have a 64GB version that is sub 400w for 8x CPU under load. One of the nice things is that instead of four PSUs in 1U servers that can go down, I can have two redundant PSUs for the four servers. It also cuts power consumption a bit since you get higher PSU utilization and fewer fans spinning.

I can add power in 1A increments. The total size of the site and forums is under 100GB so a few 3.5" disks would be more than is needed.

One of the other thoughts I had was to take one or two of the servers and turn it into storage. I could pull one CPU and memory bank to save 0.4A per server and it would provide spares. For example for OnApp it would have two hypervisors (dual CPU), one control server (only one CPU), and one storage/ SAN server (one CPU).
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Now on the networking portion. If anyone has thoughts about the Fortinet FortiGate 60C I started a thread. Since I have the space, thinking I could even do an active-active setup. Goal is to minimize how many things wake me up at night.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA

nitrobass24

Moderator
Dec 26, 2010
1,087
131
63
TX
Colo probably already includes a remote reboot PDU. Might want to ask about that prior to buying one.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Current Components List:


Current Network Diagram


Ordered the HP switch today. Thoughts appreciated. Also thinking that I need a lot less storage than I originally thought.
 

gigatexal

I'm here to learn
Nov 25, 2012
2,913
607
113
Portland, Oregon
alexandarnarayan.com
um kudos to the confusing networking diagram haha

my vote is to go with enterprise ssd, maybe even the sd3700 which is better than the 710, either would be better suited for a server use case I would think

then again the samsung's are pretty reliable...

does one need a vlan capable switch if the hypervisor is creating the emulated vlan switch?
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
does one need a vlan capable switch if the hypervisor is creating the emulated vlan switch?
He is using 4 physical nodes each connected to both vlans.

@Patrick, 5A at 220v?

Only going with single network pathway?

Also, what is your plan for management? VPN to vlan3?
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Scary thing is 5A @ 120v. Cutting it close yes, but it is unlikely I'm going to be 100% utilized on the boxes anytime soon. Also each physical node can be powered on separately. The L5420 4 node version is quoted by MrRackables as:
Watt Power Readings Energy Saver

57W Per mode at idle

85W Per node at High Load

228W Whole System at Idle

338W Whole System at High Load
10w lower TDP per CPU, but FB DIMMS vs DDR3. Should be "interesting" to say the least. What I do know is that I'm not using 12 rotating disks.

One other thing I was toying with is pulling one CPU/ set of RAM in the backup node. Worst comes to worst 1A is ~$20/mo so I'm leaning to seeing how much it actually pulls.

On the single network pathway it is single to the colo provider's switch. Not much more to add a second one and the FortiGate. One thought I have been toying with is putting the IPMI ports on the FortiGate.

This is certainly something where there is a bit of guessing going on.
 

nitrobass24

Moderator
Dec 26, 2010
1,087
131
63
TX
Just for consistency sake I might leave the IPMI ports on the switch. Set up subinterfaces on the fortigate and setup vlans on the switch.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Just for consistency sake I might leave the IPMI ports on the switch. Set up subinterfaces on the fortigate and setup vlans on the switch.
That make sense especially if the C6100 ends up being a bit better on power and I can colo a low power Xeon E3 or Atom machine also.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Have not received yet.

Did learn something interesting. Dell basically sells these in 500 unit quantities. Somewhere in the $700-800/ unit range but you have to buy 500.
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Have not received yet.

Did learn something interesting. Dell basically sells these in 500 unit quantities. Somewhere in the $700-800/ unit range but you have to buy 500.
What, the site doesn't need 2k dual cpu nodes yet?