So I bought a winterfell node

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fake-name

Active Member
Feb 28, 2017
180
144
43
73
So, continuing:

One of the best ways to do fairly precise layout for drilling holes is to just have a printout done, and then centre-punch right through the printout.

Since this is fairly large, I wound up having it printed out at the local staples. $3.29 per page for 24x36" black-and-white prints.

Your every-day printer is *surprisingly * dimensionally accurate.
64859035_347888075868817_5721536289614856192_n.jpg 65828102_909195606106817_5944357977455591424_n.jpg
I laid out, punched, and drilled all the holes in the baseplate, plus laid-out, drilled + tapped holes in the 1/4" square bars.

65677885_1112025648981769_107934035682000896_n.jpg

Initial mockup went well, though I needed to order more screws.

s_DSC01068.jpg

Here's the power supply and one of the bus-bar mating assemblies installed as well. The other bus-bar connector isn't finished yet (the cables for that one were in use until just a little bit before the picture was taken).

s_DSC01070.jpg
The servers fit nice. They do hang out the end a fair bit, mostly because my rack is 28" deep, while the servers are ~34". I'm OK with this.

s_DSC01075.jpg

Sled carriers are installed in the rack here. The first set of support hoops don't have a top because I need to be able to angle the servers into the carrier a bit, because my server closet is literally a closet, and I don't have a full 34" from the front of the rack to the wall. It's super right in there.

s_DSC01078.jpg
Servers both installed and happy!
 

nutsnax

Active Member
Nov 6, 2014
247
92
28
113
didn't see but do you plan to sell these? Also do you make a mount for them on their side so you can fit more of them in a single row?
 

nutsnax

Active Member
Nov 6, 2014
247
92
28
113
This is actually the, more or less, complete idea I had - please praise me for my awesome MS Paint skills. These nodes have quick disconnect terminals that can hook into, what looks like, 1/16" or 1/8" copper bar, so if you're only hooking four of them together you could have a really neat setup that fits in about ~4u and uses the same power supply so added efficiency.

I guess you could do similar horizontally but I'd think there would be more complexity to it... this seems relatively straight-forward..

upload_2019-8-28_14-27-28.png
 
  • Like
Reactions: Tha_14

Syndroma

Member
Nov 5, 2017
30
13
8
42
Hi... I am looking for a similar setup. Could you share a pic of your setup with PSU connection?
Bought a PSU from miners, it has a lot of GPU connectors, the connectors have a different keying, but I forced them into the midboard, just making sure that yellow wires are on 12V side and black wires on GND side. M/B draws 300W, so I made sure I used enough independent cables. Works for half a year now.

DSC00503.JPG
 
  • Like
Reactions: Alex M. and Tha_14

Alex M.

New Member
Feb 29, 2020
3
0
1
Bought a PSU from miners, it has a lot of GPU connectors, the connectors have a different keying, but I forced them into the midboard, just making sure that yellow wires are on 12V side and black wires on GND side. M/B draws 300W, so I made sure I used enough independent cables. Works for half a year now.

View attachment 12368
Any chance you have a pic of the specific PSU and a clearer pic of the orientation of the atx cables on the mid-board. I’m a bit of noob at this, so it’d really be a great help.

This looks like the Mid-board from the Quanta F06 Leopard if I’m not mistaken. I have the same breakout board in mine and I’ve been trying to use a HP DPS-800GB A PSU with a mining Breakout board, but I haven’t had any luck getting power to the motherboard.

the breakout board came with 6 to 8 (6+2) pin ATX connectors, and I’ve connected only the 8-pin header to the breakout board, but it looks like you have the two 4-pin connecters wired up as well. Are those required? Or is it sufficient to just wire the 8-pin connector? Any help is greatly appreciated!
 

Syndroma

Member
Nov 5, 2017
30
13
8
42
This looks like the Mid-board from the Quanta F06 Leopard if I’m not mistaken.
This is no surprise, because these ones are not Winterfells, they self-identify as Leopard-DDR3.

the breakout board came with 6 to 8 (6+2) pin ATX connectors, and I’ve connected only the 8-pin header to the breakout board, but it looks like you have the two 4-pin connecters wired up as well. Are those required? Or is it sufficient to just wire the 8-pin connector?
No, they're not required for boot. I plugged them all so that 300W is spread among wires.
The orientation is simple: yellow to the one side (inner), black to the other side (outer). It is easy to see on the midboard areas were 12V and ground are connected. You only need 12V and GND to power up.

The motherboard is slow to boot. Also, if the CPU is not supporting DDR3 (most v3's don't), than no chance to boot up.
 

Alex M.

New Member
Feb 29, 2020
3
0
1
The motherboard is slow to boot. Also, if the CPU is not supporting DDR3 (most v3's don't), than no chance to boot up.

Ah, I see. I’ve been using two E5-2620 v3. For memory, I have two Kingston 4GB DDR3-1600/PC3-12800R ECC DIMMs.


On Intel’s ARK site, the E5-2620 v3 shows compatibility with “DDR4 1600/1866” but I haven’t seen any DDR4 DIMMs with speeds that low so I figure that DDR3-1600 would work. Am I incorrect in assuming that? I’ve seen on other forums that a E5-2678 v3 (which are similar to the 2680 v3) is what came from Quanta when delivered to Facebook. If you don’t mind me asking, what processor worked for you?

I apologize if this is a noob question, I am a beginner to all of this and would appreciate any help or suggestions!
 

Syndroma

Member
Nov 5, 2017
30
13
8
42
I’ve been using two E5-2620 v3.
Won't work.

so I figure that DDR3-1600 would work. Am I incorrect in assuming that?
DDR4 won't fit in DDR3 slots. You can't install DDR4 memory in this board. And if you use DDR3, the CPU should specifically support it. The list of such CPUs is kept secret by Intel, you can either try yourself or find reports on forums. Most of them are OEM versions not mentioned in Intel's datasheets. I got this list: E5-2678v3, E5-2629v3, E5-2649v3, E5-2669v3, E5-2696v3, E5-2686v3.

I use E5-2678v3 and am impressed by its performance (and price).
 

Alex M.

New Member
Feb 29, 2020
3
0
1
The list of such CPUs is kept secret by Intel, you can either try yourself or find reports on forums. Most of them are OEM versions not mentioned in Intel's datasheets. I got this list: E5-2678v3, E5-2629v3, E5-2649v3, E5-2669v3, E5-2696v3, E5-2686v3.

I use E5-2678v3 and am impressed by its performance (and price).
Okay, that makes sense now. I couldn’t find an official data sheet for the E5-2678 v3 or any cpu that supported DDR3 on Intel ARK for that matter. So I was a bit confused on how this DDR3 motherboard was compatible with V3 CPUs.

I’ll try and grab a E5-2678 v3 to test with.

Thanks for all your help! I really appreciate it.
 

Nhan Nguyen

New Member
Mar 11, 2020
6
0
1
So, continuing:

One of the best ways to do fairly precise layout for drilling holes is to just have a printout done, and then centre-punch right through the printout.

Since this is fairly large, I wound up having it printed out at the local staples. $3.29 per page for 24x36" black-and-white prints.

Your every-day printer is *surprisingly * dimensionally accurate.
View attachment 11529 View attachment 11531
I laid out, punched, and drilled all the holes in the baseplate, plus laid-out, drilled + tapped holes in the 1/4" square bars.

View attachment 11530

Initial mockup went well, though I needed to order more screws.

View attachment 11533

Here's the power supply and one of the bus-bar mating assemblies installed as well. The other bus-bar connector isn't finished yet (the cables for that one were in use until just a little bit before the picture was taken).

View attachment 11534
The servers fit nice. They do hang out the end a fair bit, mostly because my rack is 28" deep, while the servers are ~34". I'm OK with this.

View attachment 11535

Sled carriers are installed in the rack here. The first set of support hoops don't have a top because I need to be able to angle the servers into the carrier a bit, because my server closet is literally a closet, and I don't have a full 34" from the front of the rack to the wall. It's super right in there.

View attachment 11536
Servers both installed and happy!
It's amazing how you documented your findings. Great to see how BIOS in Serial Console works, so great!

I myself bought dual-node winterfell few week ago, everything seems working fine and the only issue I got is that vga output is intermittent. It outputs video for first boot but not after restart or shutdown and I need to manually press Power Off/Restart button to get vga working again. It happens both for Linux and Windows, do you have similar issue? Thanks in avanced!

My settings:
BIOS F03_3B11
UEFI boot HDD (disabled the rest)

p/s: so excited to know that winterfel server is still being used nowadays :D
 

komary

New Member
May 12, 2021
3
0
1
One of the servers I use to run my home projects started failing recently, and being a complete broke-ass, I spent a while trying to find a super-cheap replacement that might also be a slight upgrade.

Overall goal are to be competitive with my current setup, which is a dual E5-2660 V1 system. I'm going to reuse the RAM from that machine once I get this running. The machine will eventually run proxmox.

Anyways, I have no real requirement for rack compatability, so I wound up deciding to buy a Quanta "Winterfell" Open-Compute node (there's a long thread on them here).

Anyways, one of the issues I had when considering buying this thing was the lack of decent pictures of these things, so here we are (plus bringup).

So, if you haven't heard of them, "Winterfell" is the name of the second-generation intel-based Open compute project server nodes. It's a highly unusual chassis design done by Facefuck for their internal use that they wound up releasing, presumably in hopes that they'd get cheaper servers by virtue of more people buying them. The open compute project has substantial online documentation.

I went with winterfell, because I want E5 V2 CPUs. In this case, I'm using 2 E5-2650 V2 CPUs (mostly because they're also super cheap).

Here's the server itself:

View attachment 10971
It's a 4" x 7" x 35" (!!!) box. Yes, they're enormously long.

They're designed to slot into a custom rack, and the end of the server shoves onto a set of bus-bars which provide power. The server itself therefore requires 12V at lots of amps, and nothing else.

Considering the mobos are specced to support 2x either 95W or 135W TDP (depending on the manual), the overall dissipation is therefore in the range of 250-400W, which translates to 20-33 amps at 12V. The power cabling is unsurprisingly ridiculous.

The overall chassis:
View attachment 10973 View attachment 10974
View attachment 10985

Power input. The whole server is cooled by the two 60mm fans powered by the power input board. They run surprisingly quiet. The loudest part of the whole thing is actually the power supply I'm using for the 12V
View attachment 10975

These servers have a mezzanine 10GB SFP+ module.
View attachment 10976
View attachment 10977

They also have 2 PCI-e slots that can be set up as either 1 8X and 1 1X, or 2 4X slots. Physically, it's 1 16X and 1 8X that's open ended.
There's also a location for toollessly mounting one 3.5" HDD. I'm going to stick 2 SSDs in there.

View attachment 10978
View attachment 10979

The mobo itself has 2 SATA ports and 2 headers for a custom power cable to run storage devices. Fortunately, the cable is pretty simple (it's basically a floppy power connector -> SATA power connector), so it should be trivial to make one.
View attachment 10980
View attachment 10981

The bus-bar connector for the chassis is frankly kind of ridiculous and massively overkill here (they're rated to 105 amps!!!!!). One nice thing about the whole "Open" bit of the open-compute project is you can actually find documentation (here is the connector documentation).
It's designed for blind-mating.
View attachment 10983
View attachment 10982
View attachment 10984

The server should have a airflow duct, but in my case I had to specifically ask the seller to include it. I have no idea what they think you'd do without it.
View attachment 10986
View attachment 10987
View attachment 10988


My solution to getting 12V at all the amps is pretty simple. Just use bitcoin miner crap! It turns out you can buy breakout boards for common power supplies for ~$10. In this case, I'm using a Supermicro PWS-1K21P-1R, mostly because I have it and it's 80+ gold rated.

I powered the thing up in several steps. First, I checked the polarity of the cables, as I'm using leads from broken power supplies (whenever I have a power supply fail, I open the chassis and cut the leads off and save them). I then disconnected the power supply board from the main mobo, and powered that alone (see the next picture). Finally, I just booted the whole thing. Fortunately, no smoke was emitted.
View attachment 10989

The next challenge was the fact that it booted, but didn't *do* anything. There's a 1GB nic on the mobo, and when connected it appeared to come up (the link lights illuminate, and my switch saw some traffic from it), but plugging it directly to a test machine and trying to tshark the interface or arp-scan it yielded nothing. Either the onboard NIC isn't fully configured, or it's configured to not talk to anything.

I stuck a video card in the thing, but that didn't do anything either, so I suspect that the PCI-e connector is configured wrong (there's a *lot* of jumpers everywhere for it), or the crappy old 1-wide graphics card I have is dead (not unlikely, it's been floating about for a while on my workbench)

Now, again, there's a nice thing about the whole "open" bit, as the spec dictates a diagnostic header for the mobo. It has a 8-bit POST status code output, and a TTL serial tx/rx pair.

Reading the POST code manually yielded 0xAA, which the manual claims means the boot sequence has jumped to the BMC, but I'd expect the BMC to be the thing that talks on the 1GBE nic, and it's not talking.

Anyways, lets see if the serial port does anything. Some really horrible dangly wiring (it's a 2MM header, and I only have 0.1" header sockets on hand):
View attachment 10990
View attachment 10991
View attachment 10992
Hmm, it looks like it's talking 57600 baud. Let's hook up a serial interface:
View attachment 10972

It boots! And is generating terminal control codes that confuse putty.

I only have RX connected because of the nightmare headers at the moment. I'll grab some 2MM female headers tomorrow at work (fortunately, we use them extensively for hardware there. Convenient!).

My long-term goal here is to assemble a rack chassis that fits two of this + power-supply in a traditional 3U rack-mount chassis. I'm actually working up 3d-printable bus-bar support so I can use the node completely stock, without having to do any adapting of the wiring.

More to come!
Hello,

I am really proud of your job. I also just buy the Winterfell server and do not know how to get control BIOS and what is the cable remote to configure?

Please help to share your experience.
 

fake-name

Active Member
Feb 28, 2017
180
144
43
73
I am really proud of your job. I also just buy the Winterfell server and do not know how to get control BIOS and what is the cable remote to configure?
You have to make up your own, the OCP spec quite clearly documents the pinout for the debug header.

Or stick a graphics card in the thing. That's probably the better option if you already bought one and didn't do the research to figure out how to talk to it beforehand (really?).