Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[svt3391]

Answers:

1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.

Is the project director in Taiwan? If so I probably can ask some friend to send some inquiry.

I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.

[edit]OK, I just realized you answered in a separate thread. I wanted to delete the following but think they may have some values to someone so I leave it here[/edit]

It's not from within an OS. When booting, you enter the BIOS setup. Based on the maker of the BIOS, many options should be available. Since most of the BIOS were made by AMI, who took over Phoenix I think, the options should be very similar.

For example, see if the top menu has something called "Advanced". When you get to that menu, there may be an entry called "VT-d Support". The default is probable 'Disabled". See if you can change it to "Enabled", save. Then reboot to BIOS again and see if the value is indeed saved.

The reason I was asking is because I doubt anyone will use 2 E5 CPUs with 16 cores and 32 threads total, with 128 GB of ram, to run Windows 7. Most would want to use a bare metal hypervisor such as VMware ESXi. With that much of threads and memory, there's a chance to allocate (pass-through) some system devices to a certain VM (such as a SAS card or a HBA card) for direct access. Without VT-d the direct device pass-through is not possible. Of course, if your E5 2670 is SR0H8 instead of SR0KX then your CPU may not support VT-d.

Thanks!

 

[NorCalm]

Thanks, the Wiwynn project director's name is G.C. Chang. I will PM you the contact details so you can forward it to your friend.

Yes, the $65 E5-2670s I picked up were SR0KX so should support VT-d.. next time I reboot I will take a look at the BIOS options.

As for Windows 7.. well you get two nodes on one chassis, so I thought it would be fun to have a bare metal $460 128GB dual E5-2670 Windows 7 x64 even if it's a little gratuitous. (Someone always asks!)

Is the project director in Taiwan? If so I probably can ask some friend to send some inquiry.

[edit]OK, I just realized you answered in a separate thread. I wanted to delete the following but think they may have some values to someone so I leave it here[/edit]

It's not from within an OS. When booting, you enter the BIOS setup. Based on the maker of the BIOS, many options should be available. Since most of the BIOS were made by AMI, who took over Phoenix I think, the options should be very similar.

For example, see if the top menu has something called "Advanced". When you get to that menu, there may be an entry called "VT-d Support". The default is probable 'Disabled". See if you can change it to "Enabled", save. Then reboot to BIOS again and see if the value is indeed saved.

The reason I was asking is because I doubt anyone will use 2 E5 CPUs with 16 cores and 32 threads total, with 128 GB of ram, to run Windows 7. Most would want to use a bare metal hypervisor such as VMware ESXi. With that much of threads and memory, there's a chance to allocate (pass-through) some system devices to a certain VM (such as a SAS card or a HBA card) for direct access. Without VT-d the direct device pass-through is not possible. Of course, if your E5 2670 is SR0H8 instead of SR0KX then your CPU may not support VT-d.

Thanks!

 

[svt3391]

Thanks, the Wiwynn project director's name is G.C. Chang. I will PM you the contact details so you can forward it to your friend.

Thanks! PM replied.

Just curious, all your threads and memory seemed saturated in the screen dump. What app you were running? I know you can run Prime or something similar to test the CPU but how your 128 GB memory were almost full?

 

[NorCalm]

Oh, that was me just trying to fill up the screen (usually all CPUs are flatlining at zero and memory doesn't even register). Here is the benchmark program I used to puff it all up:

y-cruncher - A Multi-Threaded Pi Program

Thanks! PM replied.

Just curious, all your threads and memory seemed saturated in the screen dump. What app you were running? I know you can run Prime or something similar to test the CPU but how your 128 GB memory were almost full?

 

[jap]

hi,

just for fun i measured power consumption of one node with one and dual cpu:

1x e5-2670 8*2gb dimm
idle 50w
1x core used 74w
2x core used 88w
3x core used 98w
4x core used 111w
5x core used 121w
6x core used 137w
7x core used 147w
8x core used 149w

2x e5-2670 8*2gb dimm + 4*4gb dimm
idle 64w
1x core used 92w
2x core used 117w
3x core used 137w
4x core used 152w
5x core used 162w
6x core used 173w
7x core used 189w
8x core used 207w
9x core used 233w
10x core used 246w
11x core used 260w
12x core used 303w
13x core used 316w
14x core used 323w
15x core used 326w
16x core used 328w

Bye

Jan

 

[jap]

hi,

anyone was able to use ipmi to remote control the server? i tried ipmitool, but without success.
maybe i don't know the proper username/password. I see, that the server bmc get a ip from dhcp,
but i was not able to establish communication via ipmi to that ip address.

tnx

Jan

 

[tuatara]

@jap note this part of the motherboard documentation:

"The OCP Intel motherboard is the first BMC-less platform for cloud computing that provides DCMI 1.5-based out-of-band management using the Intel PCH ME."

Take a look here for info on getting this working:
https://wiki.ubuntu.com/OpenCompute#Out-of-Band_Management

Which PSU were you using for those measurements? If the included PSU, were you running 208V AC or 48 V DC?

 

[jap]

@jap note this part of the motherboard documentation:

"The OCP Intel motherboard is the first BMC-less platform for cloud computing that provides DCMI 1.5-based out-of-band management using the Intel PCH ME."

Take a look here for info on getting this working:
OpenCompute - Ubuntu Wiki

I allready read this, but still not finished to complete working dcmi on ubuntu - i have modules, but dcmitool get some errors..

but there should be ipmitool functional - not by me :-(

from ubuntu wiki: "Out-of-band management for Open Compute v2 (Windmill) platform
It's not a problem to use ipmitool to do remote out-of-band management. Note: Add -C3 to the ipmitool command when v1.8.12 is used. Bug 1176202."

Which PSU were you using for those measurements? If the included PSU, were you running 208V AC or 48 V DC?

I used the included PSU and 240 volt power - i'm from europe, here is 240 volt the power standard

Jan

 

[capn_pineapple]

As for Windows 7.. well you get two nodes on one chassis, so I thought it would be fun to have a bare metal $460 128GB dual E5-2670 Windows 7 x64 even if it's a little gratuitous. (Someone always asks!)

[generic "BUT WILL IT RUN CRYSIS???" post]

 

[NorCalm]

[generic "BUT WILL IT RUN CRYSIS???" post]

I wonder how I could squeeze a decent GPU into the chassis to play it!

 

[NorCalm]

Thanks, Jan - this is great data!

hi,

just for fun i measured power consumption of one node with one and dual cpu:

1x e5-2670 8*2gb dimm
idle 50w
1x core used 74w
2x core used 88w
3x core used 98w
4x core used 111w
5x core used 121w
6x core used 137w
7x core used 147w
8x core used 149w

2x e5-2670 8*2gb dimm + 4*4gb dimm
idle 64w
1x core used 92w
2x core used 117w
3x core used 137w
4x core used 152w
5x core used 162w
6x core used 173w
7x core used 189w
8x core used 207w
9x core used 233w
10x core used 246w
11x core used 260w
12x core used 303w
13x core used 316w
14x core used 323w
15x core used 326w
16x core used 328w

Bye

Jan

 

[tuatara]

I'll add my data. I'm powering it with a 110V - 220V transformer so there's an added inefficiency that will increase the wall power figures.

Configuration:
single node with 2x E5-2670 v1 with 16 * 4GB
(other node present but not powered on)

Transformer on with no load: ~ 14W

Node off (PSU on with fans spinning): 26W
CPUs idle in C2 state: 70W (75W with ConnectX-3 10 GbE mezzanine)
All cores maxed with "stress-ng -c 32": 249W

CPU temperatures stabilize at 80C after the fans ramp up.

 

[smithse79]

CPU temperatures stabilize at 80C after the fans ramp up.

That seems excessively high.

 

[tuatara]

That seems excessively high.

Yeah, it's definitely higher than I expected but I wonder if that's the intended behavior? 80C is within spec for the CPU and the motherboard specification states:

"The sensors should make sure that no CPU throttling is triggered due to thermal issues, under the following environmental conditions:
• Inlet temperature lower than 30C (including 30C), and 0 inch H2O pressure
• Inlet temperature higher than 30C but lower than 35C (including 35C), and 0.01 inch H2O pressure"

"The motherboard supports auto fan speed control for the system fans connected to it. "

"Fan speed control should set system fans running at lowest speed and provide enough damping to avoid speed vibration."

@jap / others: what CPU temperatures do you observe when under full load for a long time?

 

[smithse79]

What are you using to monitor temps?

 

[tuatara]

The CPU temperature as reported by the dev.cpu.XX.temperature sysctls under FreeBSD. I believe these are the on-die core temperatures.

coretemp

 

[NorCalm]

In treating these as cost-efficient workstations, I'm trying to expand I/O as much as possible.

This may be a big difference between these Wiwynn SV7210 and the Quanta, but the three MiniSAS headers appear unpopulated:

Imgur: The most awesome images on the Internet

I don't know much about SAS but I can see the controller from the operating system, so it appears that it's just the headers that are missing. You can also see in the picture the BIOS version loaded.

SAS controllers without any way to get it out doesn't seem useful, but is there a way to extend the SAS bus using a PCIe card so I can utilize the controller without doing some crazy SMT soldering?

 

[tuatara]

I can confirm that the miniSAS headers are missing on the Quanta boards too. The controllers are present.

The other differences that I've identified are that the WiWyn's are new, sealed in the box while my Quanta appears used (though very clean) and that the WiWyn includes a riser to split the 16x PCIe while the Quantas have a ConnectX-3 10 GbE in the mezzanine slot.

 

[NorCalm]

Thanks @tuatara!

I wonder how closely to spec all these Windmill OCP boards are... would the BIOS be interchangeable?! What is the version/date of the Quanta BIOS?

I can confirm that the miniSAS headers are missing on the Quanta boards too. The controllers are present.

The other differences that I've identified are that the WiWyn's are new, sealed in the box while my Quanta appears used (though very clean) and that the WiWyn includes a riser to split the 16x PCIe while the Quantas have a ConnectX-3 10 GbE in the mezzanine slot.

 

[tuatara]

I don't think they're actually for SAS/SATA, just that they use a miniSAS connector:

"The Efficient Performance motherboard has two PCIe* x4 external connectors on the motherboard. These connectors can be used to build a PCIe* connection between two systems.
The PCIe* x4 connector can be hot inserted and removed. A PCIe* 3.0 re-driver is used for PCIe* external links and supports a miniSAS cable up to 2 meters long."

I'll check the exact BIOS version and date tonight but from memory it was in 2012 for my Quanta, whereas I think someone earlier reported a version dated in 2013.