Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[smithse79]

I too thought it would be much more elegant taking a single $99 node and powering it from ATX. Once I saw what you get "for free" ($199 includes the two $99 nodes, plus the chassis, plus the power supply, plus the power bus board) I got lazy and just paid $48.

I'm glad you confirmed that you can use a video card for this. Will make it much easier to get configured. Can it boot from USB?

 

[NorCalm]

For the VT-d question, I don't know if this answers your question, but on a lark I installed the Xen Hypervisor on my Ubuntu 16.04 LTS beta.

This is the output of the initial dom0 dmesg upon boot:

(XEN) Processor #45 6:13 APIC version 21
(XEN) Processor #47 6:13 APIC version 21
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71
(XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2600.028 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables enabled.​

Awesome write up! Especially about the voltage converter, which seems to be a very easy to follow approach.

Questions:

1. are you able to find any BIOS update? At Wiwynn's web site I could not even find this SV7210 listed.
2. in the BIOS, are you able to enable the VT-d and it keeps the settings even after reboot? one of my friend said his company used some of those OpenCompute unit and he was not able to keep the VT-d, even using the E5-2670 SR0KX which has the VT-d support from the chip
3. is there a power button on each of the unit? From the photo it's hard to see how to power up individual unit. Or did you just use the transformer's power switch to either power up or down both units?

Thanks for the great work!

 

[NorCalm]

I'm glad you confirmed that you can use a video card for this. Will make it much easier to get configured. Can it boot from USB?

Yes. There are two USB 2.0 ports, one in the front, and one on the motherboard (they call it "internal").

They don't seem to be identical; the internal one sometimes works for boot.. and sometimes only works for a keyboard.

It's foolproof to use the front USB port, if your USB drive fits in the port. When you boot hit <DEL> to go to the BIOS and enable the USB key for boot (seems to support UEFI but I haven't tested it).

Also, note that there are some quirky things about more modern video cards - the GeForce GT 710 2GB didn't work at all and I had to return that. The $10 special I mentioned in the original article works great. The Windmill BIOS is dated Apr 2013.

 

[svt3391]

Answers:

1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.

Is the project director in Taiwan? If so I probably can ask some friend to send some inquiry.

I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.

[edit]OK, I just realized you answered in a separate thread. I wanted to delete the following but think they may have some values to someone so I leave it here[/edit]

It's not from within an OS. When booting, you enter the BIOS setup. Based on the maker of the BIOS, many options should be available. Since most of the BIOS were made by AMI, who took over Phoenix I think, the options should be very similar.

For example, see if the top menu has something called "Advanced". When you get to that menu, there may be an entry called "VT-d Support". The default is probable 'Disabled". See if you can change it to "Enabled", save. Then reboot to BIOS again and see if the value is indeed saved.

The reason I was asking is because I doubt anyone will use 2 E5 CPUs with 16 cores and 32 threads total, with 128 GB of ram, to run Windows 7. Most would want to use a bare metal hypervisor such as VMware ESXi. With that much of threads and memory, there's a chance to allocate (pass-through) some system devices to a certain VM (such as a SAS card or a HBA card) for direct access. Without VT-d the direct device pass-through is not possible. Of course, if your E5 2670 is SR0H8 instead of SR0KX then your CPU may not support VT-d.

Thanks!

 

[NorCalm]

Thanks, the Wiwynn project director's name is G.C. Chang. I will PM you the contact details so you can forward it to your friend.

Yes, the $65 E5-2670s I picked up were SR0KX so should support VT-d.. next time I reboot I will take a look at the BIOS options.

As for Windows 7.. well you get two nodes on one chassis, so I thought it would be fun to have a bare metal $460 128GB dual E5-2670 Windows 7 x64 even if it's a little gratuitous. (Someone always asks!)

Is the project director in Taiwan? If so I probably can ask some friend to send some inquiry.

[edit]OK, I just realized you answered in a separate thread. I wanted to delete the following but think they may have some values to someone so I leave it here[/edit]

It's not from within an OS. When booting, you enter the BIOS setup. Based on the maker of the BIOS, many options should be available. Since most of the BIOS were made by AMI, who took over Phoenix I think, the options should be very similar.

For example, see if the top menu has something called "Advanced". When you get to that menu, there may be an entry called "VT-d Support". The default is probable 'Disabled". See if you can change it to "Enabled", save. Then reboot to BIOS again and see if the value is indeed saved.

The reason I was asking is because I doubt anyone will use 2 E5 CPUs with 16 cores and 32 threads total, with 128 GB of ram, to run Windows 7. Most would want to use a bare metal hypervisor such as VMware ESXi. With that much of threads and memory, there's a chance to allocate (pass-through) some system devices to a certain VM (such as a SAS card or a HBA card) for direct access. Without VT-d the direct device pass-through is not possible. Of course, if your E5 2670 is SR0H8 instead of SR0KX then your CPU may not support VT-d.

Thanks!

 

[svt3391]

Thanks, the Wiwynn project director's name is G.C. Chang. I will PM you the contact details so you can forward it to your friend.

Thanks! PM replied.

Just curious, all your threads and memory seemed saturated in the screen dump. What app you were running? I know you can run Prime or something similar to test the CPU but how your 128 GB memory were almost full?

 

[NorCalm]

Oh, that was me just trying to fill up the screen (usually all CPUs are flatlining at zero and memory doesn't even register). Here is the benchmark program I used to puff it all up:

y-cruncher - A Multi-Threaded Pi Program

Thanks! PM replied.

Just curious, all your threads and memory seemed saturated in the screen dump. What app you were running? I know you can run Prime or something similar to test the CPU but how your 128 GB memory were almost full?

 

[jap]

hi,

just for fun i measured power consumption of one node with one and dual cpu:

1x e5-2670 8*2gb dimm
idle 50w
1x core used 74w
2x core used 88w
3x core used 98w
4x core used 111w
5x core used 121w
6x core used 137w
7x core used 147w
8x core used 149w

2x e5-2670 8*2gb dimm + 4*4gb dimm
idle 64w
1x core used 92w
2x core used 117w
3x core used 137w
4x core used 152w
5x core used 162w
6x core used 173w
7x core used 189w
8x core used 207w
9x core used 233w
10x core used 246w
11x core used 260w
12x core used 303w
13x core used 316w
14x core used 323w
15x core used 326w
16x core used 328w

Bye

Jan

 

[jap]

hi,

anyone was able to use ipmi to remote control the server? i tried ipmitool, but without success.
maybe i don't know the proper username/password. I see, that the server bmc get a ip from dhcp,
but i was not able to establish communication via ipmi to that ip address.

tnx

Jan

 

[tuatara]

@jap note this part of the motherboard documentation:

"The OCP Intel motherboard is the first BMC-less platform for cloud computing that provides DCMI 1.5-based out-of-band management using the Intel PCH ME."

Take a look here for info on getting this working:
https://wiki.ubuntu.com/OpenCompute#Out-of-Band_Management

Which PSU were you using for those measurements? If the included PSU, were you running 208V AC or 48 V DC?

 

[jap]

@jap note this part of the motherboard documentation:

"The OCP Intel motherboard is the first BMC-less platform for cloud computing that provides DCMI 1.5-based out-of-band management using the Intel PCH ME."

Take a look here for info on getting this working:
OpenCompute - Ubuntu Wiki

I allready read this, but still not finished to complete working dcmi on ubuntu - i have modules, but dcmitool get some errors..

but there should be ipmitool functional - not by me :-(

from ubuntu wiki: "Out-of-band management for Open Compute v2 (Windmill) platform
It's not a problem to use ipmitool to do remote out-of-band management. Note: Add -C3 to the ipmitool command when v1.8.12 is used. Bug 1176202."

Which PSU were you using for those measurements? If the included PSU, were you running 208V AC or 48 V DC?

I used the included PSU and 240 volt power - i'm from europe, here is 240 volt the power standard

Jan

 

[capn_pineapple]

As for Windows 7.. well you get two nodes on one chassis, so I thought it would be fun to have a bare metal $460 128GB dual E5-2670 Windows 7 x64 even if it's a little gratuitous. (Someone always asks!)

[generic "BUT WILL IT RUN CRYSIS???" post]

 

[NorCalm]

[generic "BUT WILL IT RUN CRYSIS???" post]

I wonder how I could squeeze a decent GPU into the chassis to play it!

 

[NorCalm]

Thanks, Jan - this is great data!

hi,

just for fun i measured power consumption of one node with one and dual cpu:

1x e5-2670 8*2gb dimm
idle 50w
1x core used 74w
2x core used 88w
3x core used 98w
4x core used 111w
5x core used 121w
6x core used 137w
7x core used 147w
8x core used 149w

2x e5-2670 8*2gb dimm + 4*4gb dimm
idle 64w
1x core used 92w
2x core used 117w
3x core used 137w
4x core used 152w
5x core used 162w
6x core used 173w
7x core used 189w
8x core used 207w
9x core used 233w
10x core used 246w
11x core used 260w
12x core used 303w
13x core used 316w
14x core used 323w
15x core used 326w
16x core used 328w

Bye

Jan

 

[tuatara]

I'll add my data. I'm powering it with a 110V - 220V transformer so there's an added inefficiency that will increase the wall power figures.

Configuration:
single node with 2x E5-2670 v1 with 16 * 4GB
(other node present but not powered on)

Transformer on with no load: ~ 14W

Node off (PSU on with fans spinning): 26W
CPUs idle in C2 state: 70W (75W with ConnectX-3 10 GbE mezzanine)
All cores maxed with "stress-ng -c 32": 249W

CPU temperatures stabilize at 80C after the fans ramp up.

 

[smithse79]

CPU temperatures stabilize at 80C after the fans ramp up.

That seems excessively high.

 

[tuatara]

That seems excessively high.

Yeah, it's definitely higher than I expected but I wonder if that's the intended behavior? 80C is within spec for the CPU and the motherboard specification states:

"The sensors should make sure that no CPU throttling is triggered due to thermal issues, under the following environmental conditions:
• Inlet temperature lower than 30C (including 30C), and 0 inch H2O pressure
• Inlet temperature higher than 30C but lower than 35C (including 35C), and 0.01 inch H2O pressure"

"The motherboard supports auto fan speed control for the system fans connected to it. "

"Fan speed control should set system fans running at lowest speed and provide enough damping to avoid speed vibration."

@jap / others: what CPU temperatures do you observe when under full load for a long time?

 

[smithse79]

What are you using to monitor temps?

 

[tuatara]

The CPU temperature as reported by the dev.cpu.XX.temperature sysctls under FreeBSD. I believe these are the on-die core temperatures.

coretemp

 

[NorCalm]

In treating these as cost-efficient workstations, I'm trying to expand I/O as much as possible.

This may be a big difference between these Wiwynn SV7210 and the Quanta, but the three MiniSAS headers appear unpopulated:

Imgur: The most awesome images on the Internet

I don't know much about SAS but I can see the controller from the operating system, so it appears that it's just the headers that are missing. You can also see in the picture the BIOS version loaded.

SAS controllers without any way to get it out doesn't seem useful, but is there a way to extend the SAS bus using a PCIe card so I can utilize the controller without doing some crazy SMT soldering?