XEON-D `LIKE` on the cheap :D

[T_Minus]

This is my 'always on' backup systme... I was going to go with an 8C Xeon_D but couldn't drop $800 for such a system, and this is what I ended up with, some parts I already had to make it even cheaper.

The goal was ultra-cheap, as low power as possible and ability to run some VMs and MAYBE act as backup/additional security camera system.

Chassis: 12x Hot Swap Rosewill (already owned, may swap to small tower)
Motherboard: SuperMicro X10SRAF 1x 2011-v3/4 capable motherboard (Already owned)
CPU: 2011-v4 12 Core / 24 Thread ES 1.5GHZ (Purchased for <$130)
RAM: 4x 8GB DDR4 ECC RDIMM (Traded some SSD a couple months ago for these)
3x Mirror Backup Copy Pool:
- WD RE 5TB Refurb/White Label (Ebay, already owned)
- Seagate 5TB USB External (NIB, already owned)
- WD RED 5TB (Traded CPU for Amazon $ + Amazon Gift Card Balance = $82 out of pocket)
Ideally I would have 3x 5TB WD RED for lowest power, but I already owned the other 2 and now I guess I have 3x different drives to spread failure possibility around, LOL!
HBA: Perc h310
SSD: 4x 300GB Intel 320 (may swap out to something else as deals come in / trade these)

Power consumption in ESXI @ idle: 46w
Power consumption w/HBA in ESXI @ idle: 54w

I had a Intel E5-1650 v3 in there and it idled around 70w so a rather huge difference at idle between the higher clocked v3 and low low clocked high-core v4.

I'll be adding the SSD and HDD and reporting new power usage soon!

 

[T_Minus]

NIC Added: X520-DA1 ($28 -- NATEX)

Power increase 4-5w w/NIC not in use yet.

58w @ Idle in ESXI

WD RED 5TB Installed
62w @ Idle in ESXI
Bounces 62/63 which is between near idle rating and near max... maybe due to cheap $30 meter.

 

[T_Minus]

The WD RE and Seagate 5TB drives are not powering up for some reason... only the RED drive blipps the power meter +3w and spikes +10w when plugged in... the RE and Seagate show no power meter movement.... powering down to finish pass-through we'll see if she comes back with all drives, maybe something is up with hot swapping them.

 

[T_Minus]

Testing my 1st 10G Intel NIC from Natex I get this:

2016-06-02T19:06:18.692Z| vcpu-2| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-2)
2016-06-02T19:06:18.692Z| vcpu-2| I120+ PCIPassthruChangeIntrSettings: 0000:02:00.0 failed to register interrupt (error code 195887111)

And the Napp-IT VM shuts down!

Removed the NIC from pass-through and VM boots and stays up.
Pass through LSI/Perc310 and VM boots and stays up.

Going to have to test NIC on bare metal next

Still something weird going on with backplane/power/disks as no disks are showing up... could be weird CPU issue with PCIE lanes since it is ES too, well see.

 

[T_Minus]

Well I have 3x of these: Rosewill RSV-SATA-Cage-34 - Hard Disk Drives - Black, 3 x 5.25" to 4 x 3.5" Hot-Swap - SATA III / SAS - Cage-Newegg.com in a rackmount Roseiwll Chassis.

my 1st cage: 1/4 slots work
my 2nd cage: 2/4 slots work

I have not tried the 3rd, but I mixed/matched disks in positions and direct-sata onboard and direct-hba no backplane all disks work fine, yet those are only spots that even POWER UP the disk let alone 'detect' the drive.

 

[T_Minus]

Idling around 68w with:
- 5TB WD RE
- 5TB WD RED
- 5TB SeaGate

Intel NIC strill installed but not inuse or passed through. Going to test it on bare metal tomorrow.

Not exactly super low power :/ the 3x disks + hba @ idle add 22w of power. When CPU is more utilized, disks are all writing power usage ~100w.

I sure wish they made 5TB SSD for $200 and that onboard SATA could be passed through then could likely keep it at 50w or less, and write much faster. The 10G is rather wasted for sending data to be backed up with only 1 mirror set and ~150MB/s write. That should be plenty for a 'online' backup/replication server for my home, and in the future hope to replace the RE with RED and add 3x more when I need the capacity which should help writes a bit

 

[Patriot]

@T_Minus what is the Qspec on that 1.5ghz V4 xeon?

 

[T_Minus]

@Patriot it's an early one, I don't have it in front of me right now though.

 

[T_Minus]

Tried to pass-through the Intel 10G NIC 1 more time before powering down.

Going to try another PCIE slot to see if that does anything. Then off to another system to test it.

 

[T_Minus]

Failed to register interrupt -- Same error.

Going to try another card.

 

[T_Minus]

2nd Intel NIC -- Same Error!

 

[T_Minus]

Looks like this is a common problem with NICs and ESXI... I see the error all over the web for FreeNAS, pfSense, etc... when attempting to pass-through.

 

[T_Minus]

Removed Intel 10G NIC -- idle power is identical to when it's installed, must idle at little power.

System so far is working fine minus the 10G pass-through issue.

 

[kidchunks]

Not sure if you tried ONLY passing through the 10gb NIC? Perhaps you're running out of interrupt vectors on the host.

 

[T_Minus]

@kidchunks I didn't. I'm not sure why it would be running out, but I believe that's the error/cause too, strange as I found numerous others having this issue with various VMs (pfsense, freenas, etc...)

Since I'm not running 10G at hte moment I just removed it and will dig into it later

 

[T_Minus]

So far the super-cheap E5 2011-v4 CPU is working great.

A lot more affordable than a XeonD since I had most parts but idle power is a bit higher due to 12C vs 8C and it's not a xeon-d, and requires pcie devices vs on-board latest gen on the xeonD.

Bummed that the Rosewill hot-swap cages are really crappy though... likely will migrate to a mid-tower since I don't need so many bays for my on-line 'backup' system.

You can find RETAIL E5-2011 v3 4 C CPU on ebay for really cheap too, so if you don't want/need high-cores, don't want ES you can still do essentially this and get a system with a lot more expansion (pcie) than Xeon-D still idle low (not very low) with same features (SAS, 10G) for much much cheaper than XeonD.