Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[hmartin]

Well, it turns out I should not have bothered to solder the other two SAS headers. It seems only the first SAS header works.

This probably would have been obvious had I engaged my brain and looked at BIOS/lspci earlier:

00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 05)
09:00.0 Serial Attached SCSI controller: Intel Corporation C604/X79 series chipset 4-Port SATA/SAS Storage Control Unit (rev 05)

So the chipset only offers 6 SATA ports: 2 already on the motherboard + additional 4 via the first SAS header.

There is nothing in BIOS about the second SAS header ("HDD 4-7") and plugging drives into the SAS port does nothing. Drives aren't detected in BIOS or in the OS. Same with the third SAS header ("HDD 8-12").

There was something in the chipset datasheet about an external option ROM required for the SAS controller. It seems this is missing. So the most benefit you can get is 4 additional SATA 2 (3Gbps) drives from the chipset. If you're full on the PCIe slots and you desperately want to add more drives, it might be worth the effort, but overall I'd say: "nah"

 

[voxadam]

IIRC the Windmill boards were originally specced to include dual PCIe ×4 interfaces routed to a pair of SAS connectors. The idea was to connect nodes using PCIe directly (non-transparent bridging).

EDIT: It was section 10.2 of the original specification is where I saw mention of the external PCIe interfaces.

Though, a bit of digging through the manual confirms that those are indeed SAS interfaces, it's just that they're hidden behind "Intel® C602 upgrade ROM #2". You may be able to mod the firmware if you can find the appropriate option ROM.

It's not clear to me if the dual ×4 PCIe interfaces were ever realized in the final boards. I seem to remember reading that there was a significant bug in the Xeon v1 and v2 silicon that made efficient PCIe NTB interlinks problematic.

EDIT #2: Actually if you take a look at the diagram at the beginning of Section 4.1 in the manual it shows that the upgrade ROM mentioned is actually a physical ROM that by the looks of your photo would connect to jumper J44 located next to the HD power connectors.

 

[hmartin]

IIRC the Windmill boards were originally specced to include dual PCIe ×4 interfaces routed to a pair of SAS connectors. The idea was to connect nodes using PCIe directly (non-transparent bridging).

From the manual:
"The connector is a miniSAS-4i right-angle connector."

The miniSAS connectors on the side of the board are not right-angle connectors derp, yes they are. I think the PCIe signals are routed to the SAS headers on the rear of the board:

However I'm 95% certain the Windmill is missing the PCIe re-drivers required to make these ports work. You can see on the bottom of the board there are two placements for chips which are suspiciously absent. I suspect these are the PCIe re-drivers.

Okay, I've revised my thinking. In light of the fact these are silk screened MINISAS on the board and the traces going to the higher two SAS ports on the side are suspiciously length matched, I think you're right.

However if this is the case, we are especially screwed. You cannot hand solder SFF-8087 straight SAS headers as half the pins are under the socket.

It's not clear to me if the dual ×4 PCIe interfaces were ever realized in the final boards.

You might be able to find out if the links are active using an oscilloscope on the pads to the missing redrivers, assuming you can find out which redrivers they thought to use. Since I don't believe any OCP motherboard ever implemented the external PCIe links, I think this would be very difficult. Also, there are a TON of missing passives around these SAS headers.

SATA/SAS I don't mind screwing around by soldering some capacitors and headers. PCIe on the other hand... that's a very high speed bus which would not be so forgiving to making mistakes in component selection.

Actually if you take a look at the diagram at the beginning of Section 4.1 in the manual it shows that the upgrade ROM mentioned is actually a physical ROM that by the looks of your photo would connect to jumper J44 located next to the HD power connectors.

I would be surprised if an upgrade ROM connects to a 2 pin jumper interface. Besides, according to Section 4.1 there's supposed to be 4 SAS ports present without the upgrade ROM, and I'm not seeing any drives when I attach a SAS cable to the second port, so I'm not convinced the header is routed to the chipset. Either that, or for some strange reason the SAS controller is not backward compatible to SATA. I don't have any SAS drives to test.

Unless SFF-8087->SATA cables are incompatible with a SAS controller?

 

[hmartin]

It's not clear to me if the dual ×4 PCIe interfaces were ever realized in the final boards. I seem to remember reading that there was a significant bug in the Xeon v1 and v2 silicon that made efficient PCIe NTB interlinks problematic.

@voxadam: I think you're correct, the two SAS headers at the rear of the board are for the SAS controller. The other two headers that I've soldered are for the PCIe x4 links.

If I bought a SAS-SAS cable, I could connect the board to itself to test if the PCIe links are working. However I know little about PCIe NTB, so this raises some questions:

1. Will it cause damage to connect a PCIe link from the processor to another PCIe link from the same processor? According to the Windmill specification, the PCIe x4 links are from CPU0 ports 3a/3b.
   
   
2. Assuming (1) is false. Would connecting the PCIe links to each other be functional? Can I buy a SAS cable and connect the links to each other to confirm whether they are PCIe and if I've soldered correctly?
3. Is it possible to place a normal PCIe device on the end of a PCIe NTB link? If I designed a PCB which had a SAS receptacle and PCIe x4 slot, would an inserted card function? Or is the CPU expecting a PCIe bridge/switch on the other end of the link?

 

[voxadam]

If you're interested in NTB you might try #NTB on the OFTC IRC network; it's not a large busy channel so you'll probably have to wait around quite awhile but I have talked with some folks there in the past; actually, it was about this exact topic (someone was even familiar with this board).

 

[voxadam]

As far as #3 is concerned I'm reasonably sure there's nothing special about those ×4 links other than they support NTB. Baring any firmware/BIOS shenanigans I'm reasonably sure they should work just like any other ×4 PCIe interface, though, don't quote me on that.

 

[voxadam]

Wouldn't it make the most sense for the externally exposed connectors on the back of board to be the PCIe interfaces? First of all, they're exposed to allow for use as a machine-to-machine or machine-to-bridge interconnect. Second, the footprint looks like it would match that of a 90° SFF-8643 connector which IIRC is one of the connectors I've seen used for PCIe ×4 over cable connections. Then there's the fact that if J44 is indeed for a option ROM the location near the optional SAS interfaces would make sense.

EDIT: There's a good chance that my thought about the rear pads being for SFF-8643 is wrong; you're very likely correct that they're for SFF-8087 connections. Though, that doesn't change the rest of my thinking.

EDIT 2: Molex's site is giving me a headache and it's not the first time. I keep flip-flopping on what connector those rear footprints were intended for. Though, again, I'm still thinking the rear connectors were/are for the PCIe ×4 interfaces but the fact that I don't personally have a Windmill board and a PCIe protocol analyzer makes verifying any of this incredibly difficult.

 

[voxadam]

You cannot hand solder SFF-8087 straight SAS headers as half the pins are under the socket.

You might be able to using hot air. Another option would be to use 90° headers instead; you'd probably lose a lot of mechanical stability but as long as the footprints were reasonably similar you'd at least be able to access both rows.

 

[hmartin]

Wouldn't it make the most sense for the externally exposed connectors on the back of board to be the PCIe interfaces? First of all, they're exposed to allow for use as a machine-to-machine or machine-to-bridge interconnect.

That was my thinking as well. But I guess it also makes sense to have SAS connections on the back of the board to connect to a SAS shelf.

You might be able to using hot air. Another option would be to use 90° headers instead; you'd probably lose a lot of mechanical stability but as long as the footprints were reasonably similar you'd at least be able to access both rows.

As far as I can tell, the pads are the same layout as the right angle connectors but with less spacing between the pads. So if you chopped off the plastic supports on the bottom of the right angle connector it could be soldered with some difficulty. I don't have a hot air station and I'm not going to drop $20 on SAS headers I can't solder.

But you would have no mechanical strength apart from the traces, which sounds like an awesome way to tear some/all of the traces off the board.

Not worth it to gain 4 SAS ports IMHO.

Edit: actually, it should be pretty easy to confirm whether the two ports by the chipset are PCIe or not. I ordered an SFF-8087 cable and I'll use a remaining SAS header to solder to one of these GPU riser boards:

If it's really a PCIe port, I should have an x1 link to whatever is in the slot. Seems like the easiest method.

 

[GVP_Supp]

hmartin, Can you make a photo of the soldered connector?

 

[hmartin]

hmartin, Can you make a photo of the soldered connector?

You mean on the Windmill board, or on the PCIe riser?

For the Windmill board I already posted photos.

For the PCIe riser, I'm just proposing how I would approach it. I don't have the parts yet. On second thought I'm thinking it might be easier to use a PCIe riser with a flex cable, which you could solder directly to the SAS header on the motherboard, so you don't have to also buy and solder the SAS connectors.

However, one thing that seems like a major problem is there is no REFCLK on the SAS header, only SMBUS CLK. I have no idea how the PCIe link is supposed to function if the host doesn't supply a PCIe reference clock.

 

[Barnett8]

Thanks, now i'm tempted to flash my spare bios chip, and run it in one of my Wiwynn's just to see.

Has anyone tried this? I'd be interested in doing this, I kinda regret having a Wiwynn instead of a Quanta due to the BIOS.

 

[GVP_Supp]

You mean on the Windmill board, or on the PCIe riser?

I'm sorry. I was on the previous page when I wrote the question.

 

[Syndroma]

Thinking about one of these Quanta Windmill V2 systems. With the riser, do you think one of the new, low-profile Geforce GT 1030 cards would work - like this one?

I tried GV-N1030SL-2GL and it works fine. Its passive heat sink gets in the airflow and it stays cool.

 

[hmartin]

Has anyone tried suspend (S3) on the Windmill? Is it working for you?

I can get it to go into S3 from Linux (systemctl suspend), but it doesn't come back. The HDD & Power LED stay on but dim somewhat in S3 mode.

Exiting S3 using WoL results in the board drawing 50W, Power LED on, but no serial console output and no response via the network.

 

[kfriis]

Most people here are calling the Quanta servers for "Windmill". Is this just an easy way to refer to all the Quanta systems?

Here is some output from HWiNFO64 v5.54-3200 on one of the Quanta servers:

Motherboard
[Computer]
Computer Brand Name: Quanta Freedom

[Motherboard]
Motherboard Model: Quanta Winterfell
Motherboard Chipset: Intel C604/C602J (Patsburg-B/XB/J) [Upgrade ROM3/1 (B0(+XOR))]
Motherboard Slots: 1xPCI Express x1, 8xPCI Express x4, 3xPCI Express x8
PCI Express Version Supported: v3.0
USB Version Supported: v2.0

[PCH Features]
USB Port Count: 14 ports
RAID Capability: RAID0/1/5/10
SATA Ports 2 and 3: Supported
SATA Port 1 6 Gb/s: Supported
SATA Port 0 6 Gb/s: Supported
PCI Interface: Supported
PCI Express Ports 5 and 6: Supported
USB Redirect (USBr): Supported
Intel Anti-Theft Technology: Not Supported
PCI Express Ports 7 and 8: Supported
PCH Integrated Graphics Support: Not Supported
Data Center Manageability Interface (DCMI): Supported
Node Manager: Supported

[BIOS]
BIOS Manufacturer: American Megatrends
BIOS Date: 03/04/2014
BIOS Version: F03_3B08
UEFI BIOS: Capable
Super-IO/LPC Chip: Winbond/Nuvoton NCT6681D/NCT6682D

Notice it is labeled as "Winterfell" (CPU-Z shows similar information). This particular server runs E5-26xxV2 CPUs.

I wonder if somebody else has this same server and also the same BIOS?

I have previously asked here on the forums if the Quanta BIOS can be upgraded and was pointed to the Quanta web-site. But I think the BIOS files that I was referred to might be for the "real" Windmill servers and not necessarily the Winterfell. Or maybe, they are all the same?

So - I am looking for anyone else with a similar server to exchange information with about BIOS upgrades, HW compatibility etc.

Thanks!

 

[Syndroma]

The real Quanta Winterfells (F03C) look like this:
QUANTA RACK WITH 45 WINTERFELL NODE/ 21 PSU/ 2 PDU | eBay

Motherboards are the same, so the same BIOS can be used for Windmill.

 

[kfriis]

Thanks.

So does that mean that only the external enclosure is what differentiates a Windmill from a Winterfell? (if the boards are all the same).

I thought some of the older Windmills did not support E5-26xxV2 CPUs in which case there would be at least two different boards (regardless of enclosure). Maybe I am wrong.

If there is only one board, I guess it makes it much easier to talk about these servers. Windmill, Winterfell, physical enclosure, etc. does not matter then.

So everybody's output from CPU-Z and HWINFO64 says "Winterfell" under Motherboard?

 

[Syndroma]

I guess it depends on exact BIOS version. Updating from native Windmill one (which does not support V2) to newer F03C one may change the name. But it's my guess.

 

[Nz0]

Hello Guys

I am a n00b and just received 2 dual nodes. They look like the Wiwynn Sv7210 (Edit they are Quanta).

Before I power up the beasts, I am checking if everything is in place, but I cannot remove the plastic bezels over the heatsinks.

When I move them back to front they are unclipped, but I still cannot remove them.

Do you have any recommendation as I don't want to break my birthday gift ?

Thank you
Nz0