50+TB NAS Build, vSphere Cluster and Network Overhaul

[nry]

ESXi Node 3 / Media PC 2

As with Node 2 / Media PC 1, this will be dual booted using iPXE from iSCSI LUNs on the file server.

Full spec:
Case: Cheap generic ATX
Mobo: Asus P5Q-E
PSU: Tagan TG480-U01
CPU: Core 2 Duo E6750
RAM: 8GB as 4x 2GB DDR2
GFX: HD5770
Boot Disk: iSCSI on NAS
OS: ESXi 5.1 / Windows 7
Power consumption idle: TBC

Pre cable management

Post cable management

 

[nry]

NAS Build

NAS build is now completed :clap: currently waiting on the 2nd RAID6 array to init but besides that it's done.

Full spec:
Case: Xcase 424S
Mobo: Asus P8B-WS
PSU: Corsair HX650
CPU: Xeon E3-1245
RAM: 4x Corsair 4GB DDR3
NIC1: Intel X540-T2
Boot Disk: OCZ Agility 60GB
OS: Ubuntu Server x64 12.04.2 LTS
RAID Controller: Areca 1882i-8
SAS Expander: Chenbro CK23601
Array 1: 7x Hitachi 7K3000 3TB in RAID6
Array 2: 8x Hitachi 5K3000 3TB in RAID6
Array 3: 8x Mixed 1TB - RAID10
Hot Spare: Hitachi 5K3000
Power consumption idle: TBC

Few pictures of the server online, top two rows are array 1, middle two rows are array 2 (4th row, end drive is the hot spare), bottom two rows are the 1TB drives.

 

[nry]

Been having a lazy day today sitting in the sun, but needed a few minutes of shade so came to play with the access point and investigate idle power usage

WiFi

For the WiFi on my network I use the Ubiquiti UniFi access points, they are about £65 each and by far the best wireless equipment I have ever used. Would love the pro version, but can't justify the £150+ they cost.
They have wireless N but only has a 100mbit ethernet port. That said I have easily hit 10MB/s transfers and always seemed to have decent range.

The best bits about these in my openion is that they only need 1 cable connected! Idle they consume a whopping 3.4w!

Another reason why I like these so much is that they have a pretty cool software interface for managing them with.

Remote Node

I have decided to have a backup ESXi node at my mum's house with a bunch of unpurchased disks, as this can't run 24/7 I need to investigate a way of remotely powering this up.

Solution? Raspberry Pi!

Idle it consumes 3w (although after taking this photo it has settled down at 2.4w) using the RS power supply.

That's it for now, back outside! BBQ and Beer time!

 

[s3ct0r]

One thing you might want to consider, depending on what you are using all this storage for, is drive reliability. Consumer grade SATA drives are a good cheap solution, but reliability goes out the door...especially at heavy usage loads. Worst yet, unless you randomize the purchase order, failures for same batch drives occur around the same time. We wont go into the math of it all, but that RAID6 you think is protecting you fairly well could be your worst nightmare.

What are you employing to monitor all the disks for errors?

 

[nry]

Storage is for:
- iSCSI - both important data and temporary data (eg running VMs from non-persistent storage)
- VPS Backups
- Workstation backups
- Media storage, most of this is backed up else where in one way or another

I have had a few failures and the controller manages to pick these up no problem. One was a false positive and the other was a complete failure and the drive was replaced under warranty.
The Hitachi disks were picked based on a report by Backblaze who use the 5K3000 disks in their storage pods. Details here Backblaze Blog » Petabytes on a Budget v2.0:Revealing More Secrets

Any recommendations on monitoring the disks would be great

 

[s3ct0r]

Storage is for:
- iSCSI - both important data and temporary data (eg running VMs from non-persistent storage)
- VPS Backups
- Workstation backups
- Media storage, most of this is backed up else where in one way or another

I have had a few failures and the controller manages to pick these up no problem. One was a false positive and the other was a complete failure and the drive was replaced under warranty.
The Hitachi disks were picked based on a report by Backblaze who use the 5K3000 disks in their storage pods. Details here Backblaze Blog » Petabytes on a Budget v2.0:Revealing More Secrets

Any recommendations on monitoring the disks would be great

A decent S.M.A.R.T. monitoring package should be more than sufficient in your case. As long as you can keep an eye on errors, relocations and etc you can anticipate failures. This is to an extent. Keep in mind the mechanical parts in consumer grade disks are of "less" quality as well, hence the 1/2 or sometimes 1/3 the lifespan.

What OS are you running on the storage array?

 

[nry]

OS is Ubuntu server 12.04.2 LTS x64

As far as I am aware the Areca controller monitors SMART.

 

[nry]

No real updates today, had to hit the real world and do some work for a change

Thought I would investigate how much power the Virgin Media SuperHub consumes.

12w! I have the wireless turned off too, not too impressed by this and wondering if a different power adapter may improve this. Not too sure it's worth the effort for saving a watt or two.

Spent a little time investigating into layer 3 switches so I can setup the network properly with VLAN routing etc. Unfortunately these all seem like they consume a fair bit of electricity so leaving them on 24/7 is probably a no go.
My plan is to use the 8 port HP ProCurve as the 24/7 switch connecting the following devices:
- UPS
- PDU
- Node0
- WiFi
- Possibly uplink to a layer 3 switch

My requirements for a layer 3 switch:
- As quiet as possible!
- Low power (802.3az would be nice)
- 10GbE uplinks for connection directly to file server
- Static routing (standard feature of layer 3 I believe)
- 24 ports minimum
- Cheap? :laugh:
- VLAN support (again standard feature)

I figured if I ever needed more than 24 ports I could use one of the HP ProCurve's connected to the layer 3 switch and only connect lower bandwidth devices such as my TV, amp, Xbox etc

So far my research has come up with a few models:

Dell PowerConnect 6224/6248
- Cheap, 2nd hand I have found the 24 port for under £200 and the 48 port for around £400
- Can get a twin port 10GbE SFP+ module for£135
- Awful reviews of endless problems with layer 3 routing

HP 2910AL
- Found on eBay for £600!
- 10GbE module is around £400!

HP A5120-48G
- Found on eBay for under £200
- Guessing it's very noisy as has atleast 4 fans!
- 10GbE module is £700+

Cisco SG500X-24
- Around £970 new
- Has 4x 10GbE SFP+ ports built in
- Cisco gets recommended the most
- Possibly low power - 802.3az
- Have read that the SG500-24 version is fanless, not too sure the 10GbE model is though

Now do I have a spare £1k for a switch? No
Not too sure what to do now haha

 

[s3ct0r]

OS is Ubuntu server 12.04.2 LTS x64

As far as I am aware the Areca controller monitors SMART.

That is correct, in part. Areca will pass some, not all (yes they are greedy) errors up to the OS, after which point they get buried until you see a red light. Controllers think they "know" best, which they do then a drive actually fails, but are not so good with the leadup.

You need to have a continuous eye on what is happening. Run smartmontools under Ubuntu and wrap a notification around them. If errors start to creep up shoot an e-mail or a text message to yourself.

Once a failure happens you start running in degraded state putting more load on the remaining drives. I assume when you had a fail there was a spare around. If two of them are about to die, wouldn't it be nice if you can anticipate it?

Also, don't take RAID6 rebuilds for granted, if two drives die, given your array size and drive types you are very likely to suffer a failed rebuild. Very likely as in almost 100% on a 21TB array

 

[s3ct0r]

*21TB array...

 

[s3ct0r]

Btw, Areca has a nice CLI that can be take advantage of as well.

 

[nry]

Haven't had chance to use the cli for that yet. Will need to investigate some options for monitoring
Cheers

 

[nry]

Node 0 Build

Almost finished this build...

Started out using the Intel CPU cooler which is around 2cm too tall for a 1U case, but for testing it's fine.

I bought a Gelid Solutions Silence iPlus for £20, which is the perfect height for a 1U case. Not tested it with the case lid on yet but seems to keep the CPU happily at around 27oC idle.

Power consumption during boot is around 63w, idle 31w :thumbsup:
With a X540-T2 NIC in the maximum power consumption I have seen hits around 75w and idle around 43w. Not too sure I will have this connected to 10GbE yet. Was only using it to transfer VMs off the current 24/7 setup.

Reason this build hasn't been completed yet... PSOD! :god:
Just my luck this, have installed ESXi on countless non-server hardware without a single glitch, finally use some proper server kit and I get issues!

 

[nry]

Remote backup server

I had decided on building a remote backup server as cheaply as possible using existing parts.

I found a case on eBay for just under £25 delivered! with 9x 5.25" bays, looked terrible in the pictures but I didn't care as no one will ever see it. I hope :blush:

Even the box looks horrific (apologies to anyone with this case who loves it :/)

Then once removed from the box, it didn't get much better! Build quality I give it 3/10

Anyway components in... pre cable management

Post cable managment, you can see the 20GB HDD mounted to the top with cable ties. This will be the boot drive.

One good thing about this case is it has holes to route cables on the rear of the motherboard tray which helped alot to keep the cables neat.

And the final product, minus drives...

Case: Aerocool Strike-X Advance Mid Tower
Mobo: Asus M3N78-VM
PSU: OCZ 550w
CPU: AM2 x240 (I think)
RAM: 4GB DDR2 as 4x 1GB
GFX: Onboard
Boot Disk: 20GB SATA (ex Xbox 360 HDD)
RAID: 3ware 9690sa-8i
HDD Bays: 2x IcyDock 5in3 5.25"
OS: Undecided
Power consumption idle: TBC

Not too sure if I should put ESXi on this, or just Ubuntu. Not even sure what I am doing for disks and RAID if any yet!

 

[nry]

Have spent all evening trying to setup Node 0 and not having much luck with the WAN interface for pfSense.

As the SuperMicro X9SCM board has two onboard NICs my plan was to use one for LAN and the other for WAN, but passing the NIC through to the VM directly (assuming this is slightly more secure).

The two onboard NICs are:
82579L
82579LM

The L model works with both pfSense and ESXi out of the box, the LM model on the other hand does not work with pfSense 2.0.3 and for it to work in ESXi you need to install an additional driver found here: Enabling Intel NIC (82579LM) on Intel S1200BT under ESXi 5.1 | Virtual Drive

So I passed though the 82579L to pfSense and was picked up no problem. But for what ever reason I can't get any traffic to flow over it, nothing obvious in the logs either. After 2 hours I have given up, removed the pass-though and created a WAN vSwitch in ESXi. This was setup in under 10 minutes! which is what I had hoped for with the pass-through!
So for now I have given up on passing through the NIC.

Here is how the cluster stands at the moment in vSphere control panel.

 

[nry]

Node 0

Finally finished the 24/7 server. Got a flexible PCIe x8 riser off eBay for the 10GbE card.

Full spec:
Case: Xcase 100S
Mobo: SuperMicro X9SCM-F
PSU: picoPSU 120w / 150w 12v brick
CPU: Xeon E3 1220 v2
RAM: 16GB as 4x 4GB DDR3 ECC
GFX: Onboard VGA
Boot Disk: Kingston 16GB USB2
HDD1: OCZ Agility 3 240GB
HDD2: Hitachi 5K3000 3TB
NIC1: Intel/Dell X540-T2 Dual port 10GbE
OS: ESXi 5.1
Power consumption idle: 42w idle/76w load (maximum recorded) - As a note without the 10GbE card it idles at 31w

 

[nry]

Rack and wiring

Here are all the rack mount servers, top down...

Node 0
Node 5
Node 2
Node 1
NAS
UPS

Rear, the switch will go here, slightly concerned about airflow out of Node 5 but it never really gets very warm.

And the cabling!

Each coloured cable has a purpose:
- Green - 10GbE
- Red - WAN
- Blue - Trunked/multi VLAN
- Yellow - Management VLAN
- Black - Normal traffic for TV/Xbox etc

All the kit in the rack is either connected directly to the UPS or to the switched PDU 'hanging' from the bottom of Node 1

In the bundle of cables going to the rack, there are 16 cables consisting of power for the Virgin Media modem and 8 port switch, HDMI/USB from Media PC1 and a whole bunch of network cables!

 

[nry]

Going back on how noisy this setup is... think I need some higher powered fans in the NAS enclosure.

Currently I follow the rule of cold air in through the front and hot air out the back. Which after almost 3 hours idling is not achieving the required cooling!

I have 3x 120mm Sharkoon fans pulling air over the 24 drives, these have a advertised 36CFM airflow
On the rear I have 2x 80mm Sharkoon fans pushing air out of the back of the case these have a advertised 11CFM

Idle CPU temperatures are in the mid 50s!!!!

Areca RAID controller CPU temperature idle 69/70!

HDD temperatures (rough averages) working from top of case to bottom
Row 1 & 2 - Hitachi 5K3000 - 45/46 ish
Row 3 & 4 - Hitachi 7K3000 - 60!
Row 5 & 6 - Mixed 1 TB 40-45ish

Now to pull some of the original very loud fans out, refit them and test temperatures again

 

[PigLover]

I don't think your fans are the problem. Bigger issue is that you've mostly blocked the exhaust fans with that APC PDU. It doesn't really matter what fans you have installed if there is no space for them to move any volume of air out the back.

Before you rip and replace the fans try taking the PDU out of the way and see what happens.

 

[nry]

Good point! Pretty sure it only covered about 5mm of the fan though, worth checking as changing the 120mm fans is a huge job pretty sure the motherboard has to come out!

Also need to check the BIOS isn't running them at lower RPM, don't think lm-sensors picked up RPM

Do have a feeling though that the fans simply don't have enough power to pull air over those drives though