Next cheap cluster... What to get?

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
So I have monies to build my next cluster for work. Last time I built it on the C6100 with X5660 chips (or similar). What is the trend in a month or two? Would they be the C6220s with the E5-26XX chips? Trying to figure out sweet spot. Also, is there a SuperMicro equivalent?

Standard questions, use case etc. CFD simulations mostly. Will use Infiniband interconnects, and aiming at 4GB/core.

Thoughts and discussions!
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
The v3 CPU use significantly less power than the E5 v1 in my experience. DDR4 prices literally dropped 30-50$ (where you shop) overnight. The reason I went with a lot of E5 v1/v2 was DDR3 prices were DIRT compared to DDR4, now they're much closer, and "new" supermicro motherboards v1/v2 or v3 are near same price for chipset. Now, if you're buying used then the E5 v1/v2 stuff is def. cheaper and there is still more of the 8 node 1P E5 v1/v2 systems that's in the 'deals' section.

Some of the cheap E5s though will have < compute power than say an x5670 or x5680 so watch out.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
I plan on comparing cost/performance between v3 and v2. Not too concerned about power usage, but will take into consideration. Used market is what I am looking towards. Not sure which chips to go for in E5 line yet. All interesting.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,966
4,931
113
I have been watching these with significant interest:
$1000 obo: Intel System Cabinet H2216WPJR Server 735858243308 | eBay
$1000 obo: Intel H2216JFJR 2U Rack Server Barebone Four Nodes Dual LGA 2011 | eBay
$1500 obo x2: Intel H2216WPJR Socket R 2U Rack Server System New not in Original Packaging | eBay

But those old X5650's are hard to beat esp when they come with IB: Supermicro 6016TT Ibqf Twin Node 4X X5650 Six Core 48GB Mem 2X 1TB | eBay

I think the hard bit is that you need to get well into the Xeon E5 8 core parts to beat the X5650's and the C6220's are expensive: Dell PowerEdge C6220 4X Node 4X E5 2620 Six Core 128GB RAM 8x 1TB HDD Microcloud | eBay since the processors cost more now.
 
  • Like
Reactions: ultradense

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
Yeah, I sold some x5670 and then realized it was a better deal to keep them with the DDR3 than go for an E5 of equal processing power. Ended up scoring an x5675 for about the same anyway... and for <200 chassis/mobo/32gb for some of these deals the x58 really can't be beat still IMHO for the bargain of CPU power.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
I have been watching these with significant interest:
$1000 obo: Intel System Cabinet H2216WPJR Server 735858243308 | eBay
$1000 obo: Intel H2216JFJR 2U Rack Server Barebone Four Nodes Dual LGA 2011 | eBay
$1500 obo x2: Intel H2216WPJR Socket R 2U Rack Server System New not in Original Packaging | eBay

But those old X5650's are hard to beat esp when they come with IB: Supermicro 6016TT Ibqf Twin Node 4X X5650 Six Core 48GB Mem 2X 1TB | eBay

I think the hard bit is that you need to get well into the Xeon E5 8 core parts to beat the X5650's and the C6220's are expensive: Dell PowerEdge C6220 4X Node 4X E5 2620 Six Core 128GB RAM 8x 1TB HDD Microcloud | eBay since the processors cost more now.
They accepted 700$ offer.

They're also in Fremont... too bad I can't drive to get it, save probably $50 on shipping.

EDIT: Was told it's coming from TX.
 
Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
I plan on comparing cost/performance between v3 and v2. Not too concerned about power usage, but will take into consideration. Used market is what I am looking towards. Not sure which chips to go for in E5 line yet. All interesting.
There is such a HUGE range of chips in the E5 line-up, even some nice unique ones between the v1/v2 line-up. And, if you get the single CPU system(s) the E5-16xx has some nice low power, and some nice high power chips :) I'd love to stick a E5-1660 v2 in for playing around on.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
So what are you leaning towards :) Purchased?

It seems CPUs are harder to come by lately. At-least reasonably priced for 3 year old chips.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
Some of us just know those are big words... and not specifics... LOL :)
 

Marsh

Moderator
May 12, 2013
2,292
1,108
113
It seems CPUs are harder to come by lately. At-least reasonably priced for 3 year old chips.
Strange, few months ago, E5-26xx v1 chips were cheap, i.e. got E5-2660 for $260. a pair of E5-2650L C1 es for $300.
E5-1607 v1 quantity 10 went for $90 each.
Lately, no deals. That's the reason for me not to pick up a second Intel H2216 4 nodes E5 server.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
Yeah, the 2660's are $600 now :( 2620 v1s are still 200-250ish, which is what I've seen them stay around. I have 2 nodes with those CPUs just because I had them.

My best deal as of lately... I was lucky to score 3x E5-2670 v2 for $400/each and 2 more for 600/each though... wish I could get about 8 more v2s and I'd be set :(

The v2 of these are now going for $900-1200 :-X WOAH! You don't even pay half that for v3 ES....
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
2U Supermicro 6026TT HTRF Server 4 Swap Node 8x Intel X5650 Six Core 192GB RAM | eBay

Pre Built Supermicro 4 Node 6026TT TF X8DTT F 2U Server 8x Intel X5650 HC 192GB | eBay

2U Supermicro 6026TT HTRF Server 4 Hotswap Node 8x Intel X5650 Six Core 96GB RAM | eBay

These seem to offer good ram, and CPU power for the $ IMHO. Comparably to what I've got a cheap 4 node E5 v1/v2 setup for these 4 Node blow them out of water on CPU power, and even slightly more RAM. However, they eat power a bit more than the x58 from my experience. And if I had more room I'd snag some of the 4 nodes :)
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Just updating the thread. Had some calls with refurbishers... seems the X5675s are the ideal spot for my users right now. Time to do some rack planning and shuffling to see how many to get and redo the IB cabling. Guess that's what my staff are good for ;)
 
  • Like
Reactions: TangoWhiskey9

TangoWhiskey9

Active Member
Jun 28, 2013
402
59
28
We really hit the jackpot with the first c6100 wave. I hadn't heard of it before this site. Was such good value.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,045
1,583
113
CA
Awesome score!

I have a H2216WPJR in a box waiting for me to get my CPUs and RAM in!! Looking forward to your build.
 

Dajinn

Active Member
Jun 2, 2015
512
78
28
30
Not to necro this thread but the H2216 with the integrated infiniband, wouldn't that one be the one to get? Would free up an expansion slot.