Suggestions for my first real server

legen

Active Member
Mar 6, 2013
208
35
28
Sweden
Long time reader of STH here, this might be a long post so skip to the end if tl;dr:cool:.

I have been running various small workstation PCs and called them servers for many years now. I will soon get my degree in master of computer science here in Sweden and start working at my first job. I thought that now that I have a stable income I could finally buy a real server for my home lab!

Currently running Esxi 5.1 on my whitebox (AMDx4, 16gb ECC), in the future i would like to test proxmox, xen etc. The issue is weird random freezes during heavy load, lacking support for VT-d/IOMMU etc.

For the past few days I have been reading on STH to update myself on the server scene. I have come up with basically three different alternatives.

1. The C6100 from ebay
Pros: 4 nodes/servers, many cores, much ram, a really good price for the unit
Cons: Shipping cost is almost the same as for the server (to Sweden), it would probably break when shipped here (the shipping people are not that gentle), very loud (I live in a 3 room apartment)

Still this would be the real deal since I get 4 real servers to work with. However it does not seem possible to get one to Sweden.

2. Build a new machine using xeon E3 and LGA1155
Pros: Cheap alternative to E5, 3 Ghz, cheaper RAM, more low-power than E5, Ivy bridge
Cons: Limit to 4 cores, 32 GB ram, feels more like a common workstation

This is the solution im looking at now since it seems the best i can get for a decent cost. I read here on STH and thinking to use the same motherboard.

3. Build a new machine using xeon E5 and LGA2011.
Pros: More powerful hardware, support for more memory, more cores, more sockets
Cons: Sandy bridge (Q3 ivy bridge), costs more, much lower clock freq, high power consumption

Should one wait for Q3 and e5 ivy?

Goal
I want a system with IMPI, at least 32 GB ram and good performance (would be nice with more than my current 4 cores) with decent power consumption. I aim to use ZFS with VT-d on either a built in sata-controller or a M1015 i own for storage while i plan to buy SSD's for esxi datastore and maybe for ZFS ZIL and L2ARC. Other than that i run two Windows 2008, some linux (debian mostly), mineOs (for minecraft), 2 pfsense boxes. This pretty much maxes the current 16 gb memory and also the CPU starts to bottleneck during load.

Any comments are welcome. I feel like the E3 alternative does not give me that big boost compared to my current whitebox, should i wait for the E5 v2 cpus?
 
Last edited by a moderator:

Patrick

Administrator
Staff member
Dec 21, 2010
11,966
4,931
113
I probably have "decent" access to hardware and I still have two C6100's. Here's the reason - fully loaded, like Xeon E3 power/ speed per node. You do get 4x nodes though to play with which means you can do a lot of experimentation. Once you get into tinkering, you will want a second dev/ sandbox server. The fully populated ones at around $900 are a good value.
 

s0lid

Active Member
Feb 25, 2013
259
34
28
Tampere, Finland
Well you can get the C6100 fairly cheap to skandinavia, you just ought to split the shipment into 2 as the fully loaded server weights more than 70lbs which is USPS's limit for international shipping.
My 2 noder's shipping fees are goin to be ~160usd and some of the parts will be sent in another shipment with msa70.

And yeah, ups does shipping for those around 800usd and fedex was something around 500usd.
 

legen

Active Member
Mar 6, 2013
208
35
28
Sweden
Thats some fast answers, i didn't expect any answers until tomorrow.

I see you guys are pushing for C6100 and it is indeed a gem. However the shipping fee is not $200 dollars but $800 (UPS), and this price does not include taxes that i guess i have to pay. I will check Vista Computer, never heard of them before.

Also, what happened with my first post :p?

I probably have "decent" access to hardware and I still have two C6100's. Here's the reason - fully loaded, like Xeon E3 power/ speed per node. You do get 4x nodes though to play with which means you can do a lot of experimentation. Once you get into tinkering, you will want a second dev/ sandbox server. The fully populated ones at around $900 are a good value.
Yeah it sounds great, i will try to PM some ebay sellers and see if i can get the shipping fee down.

Well you can get the C6100 fairly cheap to skandinavia, you just ought to split the shipment into 2 as the fully loaded server weights more than 70lbs which is USPS's limit for international shipping.
My 2 noder's shipping fees are goin to be ~160usd and some of the parts will be sent in another shipment with msa70.

And yeah, ups does shipping for those around 800usd and fedex was something around 500usd.
May i ask in what country you are located? If i manage to get the shipping down to $160 it certainly starts to get interesting.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
You said the word "lab", which in itself answers your question: Get a C6100. Even if it costs $200 to ship to Sweden, you still have a four-node, eight CPU chassis for $450 delivered. That's cheap! It won't be as fast as a Xeon E5, but you are looking for a lab, not a production machine. Yes it is much louder than a workstation, but stuff it with low-power processors like the Xeon 5520 and the fans slow down greatly, plus you can always look at the fan modifications posted on this site to quiet it further.

By the way, if you order from Vista Computer (see other posts), they do excellent packing using foam-in-place. If you order a pre-configured machine from eBay, which is the cheapest way to get a fully stuffed machine, then ask lots of questions about how it will be packed.

(legen: Originally, I somehow edited your original post instead of replying. Now fixed)
 

s0lid

Active Member
Feb 25, 2013
259
34
28
Tampere, Finland
May i ask in what country you are located? If i manage to get the shipping down to $160 it certainly starts to get interesting.
Finland, and what i meant was that i'll get the C6100 chassis and 2 nodes shipped in one box that'll cost me 160usd. C6100's rack rails and psu will be sent with MSA70 that'll cost me ~140usd.
So around 300usd shipping fees for one dual noder and one 25x2.5" sas DAS :D
 

Dragon

Banned
Feb 12, 2013
77
0
0
Lessons learned the hard way over the years:

1. Resist the temptation to go for the newest and fastest, that's just adrenaline.
2. ALL new shiny CPU will become outdated crap within 3 years.
3. Always go for the highest cost/performance ratio base on your actual need.
4. Also take hardware resale value into account, thus the formula: ($power-cost + $hardware_cost - $resale_value) / $performance <= $actual_performance_need
5. As long as money is in your hands, you can use it to make more money and buy newer hardware later, when you actually need them.
6. YANG/YANF (You are not google/facebook), you don't need the best of the best, and even they use commodity hardware.

For development use, E5 is way overkill, these 130W TDP electric heaters will be replaced by Ivy-Bridge-EP Q4 this year, then by Broadwell-EP the year or two after that.

C6100 is all you need for now. They come with more/cheaper ram than a new E3, and when Boardwell chips start floating around later this year, you'll know you made the right choice not wasting money on the 130W E5 :cool:
 

Dragon

Banned
Feb 12, 2013
77
0
0
I see you guys are pushing for C6100 and it is indeed a gem. However the shipping fee is not $200 dollars but $800 (UPS)
Holy shit, are you sure? I am on the other side of earth and the shipping fee is still only $300.
 

RimBlock

Member
Sep 18, 2011
788
8
18
Singapore
Shipping can be a big issue out of the US for international buyers.

I am getting a unit shipped to Singapore for around US$200 but I am using my own DHL account tied to my company (this will be the first shipment).

If you can sign up with corporate shipping rates, even with low volumes, you may be able to get much better rates. Shipping companies generally have a threshold level (weight or volumetric weight) and they can usually quote one-off spot rates above this level. The levels are not usually advertised as they make most of their money by not quoting spot rates. Luckily I can get spot rates for the C6100 or it would be much harder to justify.

The C6100 is, without a doubt, the best bang for your buck but the E3 option is not a bad choice either especially for a home environment. It would also be current tech so easier to source local parts for upgrading rather than having to rely on EBay and international shipping.

RB
 

RimBlock

Member
Sep 18, 2011
788
8
18
Singapore
Checkout my buddy Peter on Ebay. He ships servers all over the world and has the C6100 in stock. I am not sure what freight would be but he gets some deep freight discounts and passes the on to the customer. Let him know Chris sent you...
Dell PowerEdge C6100 XS23 TY3 Server 8x L5520 Quad Core CPU's 96GB Mem 4X Trays | eBay C6100 Link
Yep getting one from Peter myself. Accepts international Paypal and I can arrange my own shipping. Doesn't have as many other items as some other suppliers but has been helpful so far and I am now just waiting for my total size / weight before arranging a pickup.

I am also buying a second unit from another supplier mainly as they have the US$75 DDR Infiniband cards I will use for 3 nodes and a US$200 LSI 9202-16e.

RB
 

legen

Active Member
Mar 6, 2013
208
35
28
Sweden
Holy shit, are you sure? I am on the other side of earth and the shipping fee is still only $300.
I have not contacted any seller but on the ebay page for the auction it says $800+. I have contacted Peter on ebay to get some more accurate numbers.

Checkout my buddy Peter on Ebay. He ships servers all over the world and has the C6100 in stock. I am not sure what freight would be but he gets some deep freight discounts and passes the on to the customer. Let him know Chris sent you...
Dell PowerEdge C6100 XS23 TY3 Server 8x L5520 Quad Core CPU's 96GB Mem 4X Trays | eBay C6100 Link
I sent a message to him this morning, thanks for the tip! One of the big issues with buying this stuff is finding a good seller, so thanks!

Shipping can be a big issue out of the US for international buyers.

I am getting a unit shipped to Singapore for around US$200 but I am using my own DHL account tied to my company (this will be the first shipment).

If you can sign up with corporate shipping rates, even with low volumes, you may be able to get much better rates. Shipping companies generally have a threshold level (weight or volumetric weight) and they can usually quote one-off spot rates above this level. The levels are not usually advertised as they make most of their money by not quoting spot rates. Luckily I can get spot rates for the C6100 or it would be much harder to justify.

The C6100 is, without a doubt, the best bang for your buck but the E3 option is not a bad choice either especially for a home environment. It would also be current tech so easier to source local parts for upgrading rather than having to rely on EBay and international shipping.

RB
I might check into the corporate shipping stuff if Peters cost for shipping is very high, otherwise it might not be worth the extra work.

Yep getting one from Peter myself. Accepts international Paypal and I can arrange my own shipping. Doesn't have as many other items as some other suppliers but has been helpful so far and I am now just waiting for my total size / weight before arranging a pickup.

I am also buying a second unit from another supplier mainly as they have the US$75 DDR Infiniband cards I will use for 3 nodes and a US$200 LSI 9202-16e.

RB
Sounds good, i would love to get some infinibands or 10GbE cards later, but thats way into the future for me.

I might post back here with several more small questions when Peter replies.

Until then thanks again, you guys are really friendly towards new guys :)

EDIT1: one direct question. Does the power supply that comes shipped with the device support the 240V (europe) or only the American 110V? I found "Hot-Swap Power Supply 1100W 100-240V PSU" so i guess it should work!
 
Last edited:

legen

Active Member
Mar 6, 2013
208
35
28
Sweden
Lessons learned the hard way over the years:

1. Resist the temptation to go for the newest and fastest, that's just adrenaline.
2. ALL new shiny CPU will become outdated crap within 3 years.
3. Always go for the highest cost/performance ratio base on your actual need.
4. Also take hardware resale value into account, thus the formula: ($power-cost + $hardware_cost - $resale_value) / $performance <= $actual_performance_need
5. As long as money is in your hands, you can use it to make more money and buy newer hardware later, when you actually need them.
6. YANG/YANF (You are not google/facebook), you don't need the best of the best, and even they use commodity hardware.

For development use, E5 is way overkill, these 130W TDP electric heaters will be replaced by Ivy-Bridge-EP Q4 this year, then by Broadwell-EP the year or two after that.

C6100 is all you need for now. They come with more/cheaper ram than a new E3, and when Boardwell chips start floating around later this year, you'll know you made the right choice not wasting money on the 130W E5 :cool:
Those are valid points, the E5 is way to power hungry but i was looking at their low-power variants.

However the E5 alternative probably lay outside my ~$1000 budget.
 

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
I think your choice depends on what you really want to end up with. They are really different outcomes...

If you want a "practical" server to run "home production" kinds of VMs (simple web server, SIP-PBX, TV tuner, etc) go with the E3-based server. You can afford a faster CPU, it will run lower power, you can make it quieter, etc.

If you want a "real" lab and want to play with VM alternatives (ESXi vs KVM) and want maximum capability per $ (or Euro) go with the C6100. However, do understand that this will end up with a system that is power hungry, hot and loud. Without mods it is not very friendly in a home/apartment environment. Don't forget its also one to two generations behind in technology - its cheap because its getting a bit old. But it is the best deal on the market right now for a small server cluster.

If you want something to play with higher-power solutions go with the E5 (though I'd advise waiting a few months for IB). Just understand that this is probably a choice focused on future upgrades and growth. Unless you are made of money you'll have to settle for a slower CPU than you get for the some money in the E3 lineup.
 

zer0sum

Active Member
Mar 8, 2013
389
144
43
I have been looking at the C6100's and other cheap options for my home lab as I have outgrown my 32GB, E3 1230V2 box, and thought I would throw in an AMD build :)

Asus KGPE-D16 motherboard - dual socket G34 with 16 dimm slots (ASUS - KGPE-D16)
They seem to go for $350 to $450 depending on if you go new or used

2 x AMD opteron 6128 - 8 core processors - $30-50 each on ebay
16 x 4GB SDRAM - $450 or so brand new, but I am sure you can find cheaper on ebay

RAM is limited to 256GB RDIMM or 64GB with standard UDIMM's
CPU's can be upgraded right through to the latest Opteron 6386SE which is a 16 core 140W monster

The good thing for you is the shipping costs will be low as these are all small packages, and then you can pick up most of the parts locally.
The motherboard is SSIEEB which is EATX size so it will fit in a lot of mid/full tower cases.
PSU can just be a high quality unit like Seasonic etc.

Thought I would throw it out there for consideration :)
 
Last edited:

cactus

Moderator
Jan 25, 2011
829
77
28
CA
$35 for 6128!!! There have been some dodecas in the $100 range also.

I have two KGPE-D16 systems and like them. Unless you are reusing ram, registered is less expensive for high density DIMMs. Also remember you cant mix the two. You should be able to get 4GB reg/ECC sticks for less than $20 each on ebay. Watch newegg openbox for the motherboard, they come up on there for ~$200. If going with a new board, look at the supermicros because they have more PCI-E.
 
Last edited:

Dragon

Banned
Feb 12, 2013
77
0
0
I have been looking at the C6100's and other cheap options for my home lab as I have outgrown my 32GB, E3 1230V2 box, and thought I would throw in an AMD build :)

Asus KGPE-D16 motherboard - dual socket G34 with 16 dimm slots (ASUS - KGPE-D16)
They seem to go for $350 to $450 depending on if you go new or used

2 x AMD opteron 6128 - 8 core processors - $30-50 each on ebay
16 x 4GB SDRAM - $450 or so brand new, but I am sure you can find cheaper on ebay

RAM is limited to 256GB RDIMM or 64GB with standard UDIMM's
CPU's can be upgraded right through to the latest Opteron 6386SE which is a 16 core 140W monster

The good thing for you is the shipping costs will be low as these are all small packages, and then you can pick up most of the parts locally.
The motherboard is SSIEEB which is EATX size so it will fit in a lot of mid/full tower cases.
PSU can just be a high quality unit like Seasonic etc.

Thought I would throw it out there for consideration :)
Yes, Opterons are still unbeatable at building cheap memory cache servers.

In fact if AMD would just get their act together in power consumption, I wouldn't consider Intel at all, even if they are twice as fast.

When you have everything in ram, 0.01ms and 0.02ms makes no difference, and the high throughput of Opterons make up for their slower single threaded processes.
 

GaryM

New Member
Feb 27, 2013
8
1
3
Belleville, ON
Yes, Opterons are still unbeatable at building cheap memory cache servers.

In fact if AMD would just get their act together in power consumption, I wouldn't consider Intel at all, even if they are twice as fast.
I run an Opteron 4170HE in a Supermicro board, H8scm-f with 12 hard drives powered by a Seasonic gold X-560 PS. PS fan never runs, not enough power being drawn by the system. My APC 1500 UPS, on its own, draws more power than this server running 6-8 VMs. I looked at Intels offerings and came to the conclusion that the Opteron 4XXX series was more than what I needed, not more than what I would like though. ESXi is what makes my server sing and it brings out what's best in the hardware.
I too built this server with long term power use in mind.