Stacker cheap server cluster with 3 Asus Z9PH-D16's

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

legopc

Active Member
Nov 2, 2014
227
38
28
28
The Netherlands
Been some time ago since I posted here but as of lately I have been browsing here more often and getting back into enterprise toys. I have been out for some time because work kind of sucked the joy out of things, but after some school projects I caught myself looking on eBay and the likes more often.

While browsing I found a listing in Belgium for 2 Asus Z9PH-D16/QDR boards for a price of 50 euro. I made the guy an offer of 35 euro thinking he would never accept until he did haha. He then sent me an email saying he also still had a 3rd board still in the node, and if I wanted it for 35 euro. I also got him to send some cheap memory with it. That totaled to 140 euro-ish.

I purchased two 2609 v1's because I still had one laying around. Also, 2 cheap Chenbro heatsinks and an HP DPS-1200FB 1200w PSU, which totaled about 50 euro. I'm planning to use the HP PSU to power all the systems. I plan on doing this with one of them miner adapter boards for this PSU, which cost me 23 euro.

The thought behind that specific PSU is the adapter boards are stupid cheap, so is the PSU itself and it should be able to pull all the systems. The boards have 20pin ATX connectors which just consist of 8x ground, 8x 12v+, PSU_ON, and 5vstb. The DPS-1200FB doesn't do 5v standby but it does have a 12v standby. So the idea is to solder a buck-converter to that pin and power the systems that way. I will have to make some custom connectors but that should not be an issue.



Now you might ask: What do you need all this for? To which I would say: No freaking clue, just like messing around. Normally I do whatever I need to do on my Intel P4000 WS but that's starting to get a little big for my student room and loud. I also do prefer having multiple ESXi nodes with a proper vCenter instead of virtualizing everything in on my WS from a single disk.

I will be using an old ThinkPad T430 with I7 and 12gb memory with ESXi as a vCenter server and for small management tasks because its low power and quiet.

So here are the pictures :D :


Boards and the rest of the toys arrived.




Test fitting all the boards the way I want to stack them. Would still have to make some custom stuff for the PSU and possibly the drives


Testing all the boards and updating the BIOS.

During testing, even though I will only use one of the CPU's this is what I noticed on socket 2 of one of the boards. The board still in the node also somewhat died of mysterious reasons. It went from working fine while I was struggling with a defective DIMM to dead no matter what I did. CMOS reset, different BIOS chip, different CPU other memory, nothing worked. So I tried one last thing, I filled the second socket and installed one DIMM for CPU2, and low and behold it booted just fine. Moved the memory back to CPU1 and again no POST. I messaged the seller about this, the defective DIMM, and the socket and he was kind enough to offer me a refund of 70 euro. While doing some more testing with that board I did get it to boot, but with only one DIMM in D1 on CPU1. Whenever I would add a second one it would just stop POSTing.



Macgyver an ATX PSU to test the boards with the 8pin CPU connector :D.

So that's where I'm at right now. Still waiting on the PSU adapter boards from China, from there I can boot all the systems all at once and make a plan from there. I also still have some decisions to make regarding switches and case materials.
 

PigLover

Moderator
Jan 26, 2011
3,186
1,545
113
Nice!

You can probably get 1U active HSFs for that would work with that to keep the CPUs a bit better managed. But I'm afraid you might end up spending more on heatsinks than you did on the whole project so far!
 

legopc

Active Member
Nov 2, 2014
227
38
28
28
The Netherlands
Nice!

You can probably get 1U active HSFs for that would work with that to keep the CPUs a bit better managed. But I'm afraid you might end up spending more on heatsinks than you did on the whole project so far!
Thanks :D

Might be a good idea but I dont think I'll need them. I plan on covering the sides, and I have not yet decided on size but I think I'll be running 2 nice 80mm Delta's in the front. I hope its enough and temps so far seem to be stable. And I dont really have anything high workload.

Im afraid so haha, the Chenbro heatsinks are 14 euro a piece before shipping from the UK. 2 Heatsinks are 1 motherboard and the CPU's they are cooling are like 3-4 euro.
 

greenlogles

New Member
Jun 10, 2020
4
0
1
California
Wow. Awesome.
I've ordered same board and made custom power connector. However, board works only 1 second and turned off :(
I guess 450W should be enough, isn't it?

Would you please send me link where did you buy heat sinks?
I bout a few on Amazon, but they didn't fit (looks like they are compatible with desktop motherboards only)
 

legopc

Active Member
Nov 2, 2014
227
38
28
28
The Netherlands
Wow. Awesome.
I've ordered same board and made custom power connector. However, board works only 1 second and turned off :(
I guess 450W should be enough, isn't it?

Would you please send me link where did you buy heat sinks?
I bout a few on Amazon, but they didn't fit (looks like they are compatible with desktop motherboards only)
That was the heatsink I bought. But any narrow ILM 2011 heatsink should fit no issues.

450w should be just fine. Im currently powering the board off of an 8p cpu connector and boots just fine. Are you sure you got your pinout right? I guess this is some simple diagnostics of changing parts around. One thing you could try is removing all memory and booting it. In my case it just gave me a beepcode signaling to me that the boards is alive in some ways. Do you have the iKVM module installed? That could help with diagnostics.
 

greenlogles

New Member
Jun 10, 2020
4
0
1
California
That was the heatsink I bought. But any narrow ILM 2011 heatsink should fit no issues.

450w should be just fine. Im currently powering the board off of an 8p cpu connector and boots just fine. Are you sure you got your pinout right? I guess this is some simple diagnostics of changing parts around. One thing you could try is removing all memory and booting it. In my case it just gave me a beepcode signaling to me that the boards is alive in some ways. Do you have the iKVM module installed? That could help with diagnostics.
Thanks for the link.
I've double checked with board manual. Almost all pins are ground/ +12v (except other 4).
Ikvm boots fine and accessible via network, but ... I don't have credentials. All manuals show how to reset it via bios.
Removing battery and resetting bios via jumpers didn't help ((
I use only one cpu, but on another socket I just found a few pins which are slightly twisted. Can it be the problem?
 

Peanuthead

Active Member
Jun 12, 2015
839
177
43
44
I have some 1U and 2U heatsinks (2U are active, I think) for a few dollars (i.e. - getting a sandwich) and shipping.
 

legopc

Active Member
Nov 2, 2014
227
38
28
28
The Netherlands
Thanks for the link.
I've double checked with board manual. Almost all pins are ground/ +12v (except other 4).
Ikvm boots fine and accessible via network, but ... I don't have credentials. All manuals show how to reset it via bios.
Removing battery and resetting bios via jumpers didn't help ((
I use only one cpu, but on another socket I just found a few pins which are slightly twisted. Can it be the problem?
Hmm that should be fine and it should boot with just that. From the very very rare cases, I've seen it was a dead CPU but I don't know.
I think you could go a long way with flashing the IPMI just like some people did with the SM boards because this one is not that different.
I'd say create a thread and we'll try and diagnose the issue there. Do you have any spare components lying around? And where are you from btw?
 

greenlogles

New Member
Jun 10, 2020
4
0
1
California
Thanks, @Peanuthead for offers.
I've already ordered one from amazon.
Hopefully it will fit.

@legopc I have lots of memory, but have to buy one more cpu to check if current one doesn't work.
I've contacted MB seller and he proposed
to send me another board. So, I'll have second chance to boot it.
I'm from California, but originally from Eastern Europe :)
 

EasyRhino

Well-Known Member
Aug 6, 2019
509
381
63
What a strange board. Dual 20pin ATX connector, and a molex? But no 12v EPS that I could see? strange.
 

legopc

Active Member
Nov 2, 2014
227
38
28
28
The Netherlands
@legopc May I ask you to measure voltage on 12v line once you've powered on the board?
In my case I have only 11.35v with connected two 6pins connectors.
Thanks
Is that what you are looking for? 12v is pretty tolerant iirc. 11-13v should be just fine and I doubt its a power issue.
What a strange board. Dual 20pin ATX connector, and a molex? But no 12v EPS that I could see? strange.
Not that uncommon to be honest. Its just 2 8pin EPS connectors, PWR_OK, PS_ON and +5v STB.

I got both nodes working with the HP PSU and adapter board with a custom cable and step-down to +5v for the standby. Look pretty neat actually and works quite well. I'll take some pictures and post them here.

Also got a spare Thonkpad T430 with I7 running as 3rd ESXi server with a single vCenter just for management and because its silent and low power. Now the only thing to do is to think of a purpose for it all.
 
  • Like
Reactions: Agremlin