50+TB NAS Build, vSphere Cluster and Network Overhaul

[nry]

Missed my build progress post off...

Node 0 Rebuild

After putting the 24/7 node in a 1U case and having nothing but problems with it overheating I decided to scrap the node running in my 2U case and move node 0 to the 2U.
Now runs alot quieter and cooler

Also upgraded from a 120w picoPSU to a 160w, and a 150w power brick to a 200w.

Final spec:
Case: Xcase RM 204 Short
Mobo: SuperMicro X9SCM-F
PSU: picoPSU 160w XT / Dell DA-2 200w 12v brick
CPU: Xeon E3 1220 v2
RAM: 16GB as 4x 4GB DDR3 ECC
GFX: Onboard VGA
Boot Disk: Kingston 16GB USB2
Backplane: Startech 4x 2.5" in 5.25" Bay
HDD1: OCZ Agility 3 240GB
HDD2: Hitachi 5K3000 3TB
NIC1: Intel/Dell X540-T2 Dual port 10GbE
OS: ESXi 5.1

Some photos

Comparison of the new dell brick with the old 150w brick on top!

 

[nry]

Node 4 / Media PC

This is probably my longest serving computer I still own. Originally planned to sell it but as it's so small I decided to keep it as its useful if I am working away from home.

Full spec:
Case: Antec ISK-150
Mobo: Zotac 9300
PSU: picoPSU 120w / 150w Power brick
CPU: Core 2 Duo E6600
RAM: 4GB as 2x2GB DDR2
GFX: Asus HD4350
Boot Disk: Network PXE
OS: ESXi 5.1 / Windows 7

Ever since owning this unit I wanted to put the external brick inside, and today managed to do this

Very squashed in there!!

With top internal mount in place for 2x 2.5" HDDs and 1 slim DVD drive

From the front (camera has focused on the wrong area )

Rear ports

 

[Mike]

I've had better luck with the 3 fans turned 'round in a similar case.

 

[nry]

Had it that way for years and never had an issue.
Shouldn't really make much difference I don't think.

Cluster

Did the final setup on the cluster today, total of 96.85GB memory across the hosts with 60Ghz CPU and 4.87TB of storage space.
1.91TB is RAID10 on the NAS device

Power consumption

Don't have load numbers for all devices as it's a pain to get them. Anyway here is a list of all the devices power consumption

APC UPS - 41w idle / 150w+ load
APC 16a Switched PDU - 5w idle / 15w max
HP ProCurve 1800-24G - 17w
HP ProCurve 1810-8G - 3w
NAS - 173w idle / 340w boot up
Node 0 - 42w idle / 70w load
Node 1 - 65w idle
Node 2 - 73w idle
Node 3 - 130w idle
Node 4 - 57w idle
Node 5 - 82w idle
Raspberry Pi - 3w

 

[nry]

Node 0

Due to the recent heatwave in the UK, Node 0 struggled.
The zalman fan mate died, fans stopped running, the whole box over heated and now the newish Hitachi 5K3000 3TB is getting a ton of errors. The OCZ SSD has temperature warnings but seems to have survived.

I also scrapped the idea of a picoPSU, bought another short 2U xcase (need one for a project in november), which came with an Antec EathWatts 380w which Im pretty happy with, not tested power usage yet though.

Currently have the original 3x 60mm (1 of them has died completely) running full speed all the time as the X9SCM motherboard only supports PWM control and these fans are 3pin. My next issue is I can't find any 60mm PWM fans to replace these with, while I am not going for silent, I would like quiet when idle! Any suggestions welcome here

Currently thinking 3x Sharkoon 60mm fans
With
Sunbeamtech Rheosmart 3 Fan Controller - Kustom PCs Sunbeamtech Rheosmart 3 Fan Controller

Any suggestions here would be great


10GbE and Networking Improvements

Currently I do all VLAN routing via pfsense which is giving me pretty poor performance as expected. I really want a layer 3 switch with 10GbE but struggling to find a solution within my budget.

I am seriously considering the Cisco SG500X-24 which has 4x 10GbE SFP+ ports, but my issue here all my current kit is 10GbaseT so would be looking at the Netgear XS708E and connecting the two switches together with a DAC SFP+ cable. My main concern here is the noise, the Cisco is 40dB which is going to be screaming little fans and the netgear will have the same no doubt!
As I mentioned before I am not going for silent, but quietish when idle is a must. This is living in a home and ideally with the door shut on the room its all living in I don't want to be able to hear it on the other side!

Another option I have considered is selling all the 10GbaseT kit and switching to SFP+ as I currently only have 4 devices which will require the 10GbE network and I could get away with the SG500X switch.
Problem then is waiting for some PCIe cards to pop up.

Another issue I need to consider is in the future I would like 10GbE on my workstation at some point, which is a Mac, most solutions seem to be fiber based but with some availability for 10GbaseT.
As I will no doubt end up with the new Mac Pro when released, the only option for 10GbE looks to be an external thunderbolt setup.
Currently been investigating this option with a Sonnet Echo Express SE with
Myricom 10G-PCIE

Just voicing some idea's there, need to do some more reading before buying anything. But very tempted by a SG500X as they are only £818.05

Any suggestions on other switches and second hand kit I could get are welcome!

 

[PigLover]

I have the SG-500x-24g. It is a very nice switch. Reliable and easy to manager.

However, since all of your nodes currently have 10base-T you'd definitely be better off with the netgear.

 

[nry]

Think the net gear is only layer 2 though so would need both.

Seriously considering the sg500x at the moment.
Does anyone know if the brocade 1020 can be used for TCP/IP and with HP DAC cables
Don't want to end up with a FC only card

 

[PigLover]

What layer-3 capabilities do you need in the switch.

 

[nry]

What layer-3 capabilities do you need in the switch.

If im honest I don't NEED anything haha, all toys/learning.
I would like interVLAN routing and ACL.

Basically want to be able to specify

VLAN 20 can talk to VLAN 30 and 40 but not 50
etc


Which I believe the SG500 supports?

 

[mrkrad]

brocade cards require brocade brand sfp or dac cables. this sucks.

hp does the same crap.

if you get some good cheap SFP+ dual port 10gbe nic's like the emulex BE2 (i picked up a bunch @ 75 each and used them all) then you can hack the switch to run quiet since there are only two real heat points (CPU/NPU) which water cooling kits can cover, then you can use a very slow fan to make it silent.

The 10gBASE-T takes up too much power. 24 ports of 10gbase-T can eat up 500 watts (on each side switch/servers).

the netgear $1200-1300 deal on ebay is a perfect candidate to watercool and uses very little power at balls out.

also 10gbase-T is 8 times more latency per hop.

 

[nry]

After much reading I am thinking for 10GbE that SFP is the way to go. Wish I had done this in the first place!

Thanks for this. The card I have been looking at specifically is the Intel X520-DA2, I think I have found two Dell branded versions of this (2094N and 942V6) from a company in the UK, going to give them a call on monday and ask.
Under £100 each! Otherwise I have found Intel branded for around £185 which adds up very quickly!

Does Dell/Intel require own branded SFPs?
The cable I had my eye on is the "HP X240 10GB SFP+ SFP 1.2m Cable (JD096B)" which I found on eBay for £20

I think that with a SG500X switch I can manage with the 4 ports. If I need a 5th I guess I could make a bridge on a dual port card and do it that way, not brilliant but if it's a low priority machine it will be fine.

Im thinking in the future if I need more I could invest in a 24 port SFP switch, seen a few on eBay (with a broken port!) for around £1200-1400, specifically the Dell 8024F

With regards to hacking up a switch, would rather avoid doing this to be honest, given the cost of them! If I got a cheaper 2nd hand version I probably wouldn't mind so much

 

[PigLover]

If im honest I don't NEED anything haha, all toys/learning.
I would like interVLAN routing and ACL.

Basically want to be able to specify

VLAN 20 can talk to VLAN 30 and 40 but not 50
etc


Which I believe the SG500 supports?

SG500X-24G only has the most minimalistic Layer-3 support.

Very. very simple ACL (address/address-list allow/deny).
DHCP sniffing and relay.

As far as I can tell from the UI there is no inter-VLAN routing.

 

[nry]

SG500X-24G only has the most minimalistic Layer-3 support.

Very. very simple ACL (address/address-list allow/deny).
DHCP sniffing and relay.

As far as I can tell from the UI there is no inter-VLAN routing.

From a few posts I have read it seemed to suggest the console was a little more powerful that the UI
The following statement makes me think intervlan is possible...
Cisco 500 Series Stackable Managed Switches Data Sheet  [Cisco Small Business 500 Series Stackable Managed Switches] - Cisco Systems

"For example, the switches provide static LAN Layer 3 routing, allowing you to segment your network into workgroups and communicate across VLANs without degrading application performance"

Also in the ACL section...

"Support for up to 2K (2048) rules on 500 Series and 3K (3072) on 500X series.

Drop or rate limit based on source and destination MAC, VLAN ID or IP address, protocol, port, DSCP/IP precedence, TCP/ User Datagram Protocol (UDP) source and destination ports, 802.1p priority, Ethernet type, Internet Control Message Protocol (ICMP) packets, Internet Group Management Protocol (IGMP) packets, TCP flag.

Time-based ACLs supported."

This is all a learning curve for me so might be missing something!

 

[mrkrad]

I think only qlogic and brocade care about media type as far as gbic and SFP+.

Seriously though I'd get a $1200USD 24port SFP+ all day long but I can understand you wanting to save a few bucks and get a 48port gigabit with 2 or 4 10gb uplink ports for $200-400 - networking is expensive!!

I scored the deal of the century $220 for 24 DAC cables (brand new).

What I do is target cards that are oem and come with optics . I've scored 5 so far. 3 of them were intel chipset (so no special drivers) and $120~ each for what is a X520-SR2 (X520-DA2 plus 2 optics).

If you haven't noticed yet, the cabling can cost more than your whole server !!

The powerconnect 8024F is very nice! it is the same NPU as the Netgear xs m 7224S but different cpu. It has evolved (DCBx).

But honestly when you start adding QOS/VLAN/routing that is where switches start to lag out and crash. Bugs. If you are doing simple L2 switching you will have a pillar of solid.

DAC cables are $20USD from 10gtek in china if you can order enough of them - otherwise just find some local guy and buy used.

Hooking up non-branded optics is a sure-fire way to get no support and no warranty!

Avoid switches with broken ports, the PHY commands many ports at once so it's likely there is a bank (of 8) ports jacked up. Basically the broadcom switches (8024F,xsm 7224s) you have 4 phy for 24 SFP+ and two PHY for the 10gbase-T (2 ports each).

Static electricity can cause all sorts of other damage!!


Gigabit mostly with 10gb is cheap:

http://www.ebay.com/itm/Fortinet-10...49?pt=US_Network_Switches&hash=item417546b6e1

http://www.ebay.com/itm/MARVELL-swi...30?pt=US_Network_Switches&hash=item27d57f6406

http://www.ebay.com/itm/WOVEN-BROCA...66?pt=US_Network_Switches&hash=item2a2ef8795e

http://www.ebay.com/itm/HP-BROCADE-...35?pt=US_Network_Switches&hash=item2a2e567dab

pure 10gb:
http://www.ebay.com/itm/J8474A-HP-P...38?pt=US_Network_Switches&hash=item5af3872a62

http://www.ebay.com/itm/NEW-FUJITSU...08?pt=US_Network_Switches&hash=item3ccef58040

http://www.ebay.com/itm/IBM-Force10...41?pt=US_Network_Switches&hash=item3f26c373e1

 

[nry]

From what I have read the X520-DA2 seems like a good choice with ESXi 5.1 and will probably try to get a few of them.

I have not found any network switches with 2-4 10gb uplinks for under £500 here!! Maybe I am missing something but here in the UK there is very limited choice for second hand kit. Import tax is silly here too so would like to avoid buying out of the country as much as possible.

Cabling costs looks to be £15-30 per lead, which isn't too bad I guess

You mention issues with bugs in switches with regards to layer 3 features. Are there any switches which come recommended? Assume they are the ones which cost silly money.

Think I am trying to achieve the impossible here
- Low power
- Low cost
- Layer 3 with intervlan routing and acl
- Quiet

My budget here is about £2000, £2500 if I can get my Mac workstation on the 10GbE network too.

 

[mrkrad]

Well you could always ship the device in pieces and mark it damaged for repair (reassembly is repair, so its not a lie). The value of components in non-functioning status of a $300 switch is what?

you only need static routing (layer-3 lite).

I don't quite understand if you have only a few devices, why not just go nic to nic and/or use software switching (openvswitch)??

 

[nry]

Not too sure if that would fly, not imported much in the past but it's always a huge hassle.

Do like the look of the Cisco 3750x with a 10GbE module, but out of 4 SFP ports it only has 2 SFP+ ports! Madness
From what I understand this is a pretty high end switch which has redundant PSUs and fan control, almost perfect but not too sure how the licenses work on this switches

While to start with the 10GbE network will only be Node 0 server to NAS (this is currently how it functions), it will expand quickly to Node 1, Node 2 and my workstation
I am also looking at replacing Node 1 with a Dell C6100 or similar so that's 4 additional ports required at some point in the future.
Soon to be added to the kit is my mobile node which it would be nice to connect up to all of this.

In total that will be 8 10GbE ports!
All other kit runs at either 1Gbit or 100mbps

Currently pfsense does all my routing over a single 1gb pipe, which results in around 55MB/s transfers, a little too slow! If I could use some sort of software switch with good results I would be happy. If it fails, it fails not a production environment at the end of the day.

Starting to think this was all a bad idea haha and I need to sort my spending problem out!

 

[mrkrad]

Sounds like a bad use of money honestly after looking at the used ebay market over there.

 

[nry]

Sounds like a bad use of money honestly after looking at the used ebay market over there.

Think you have hit the nail on the head there.

Things do pop up every now and again so I will get some saved ebay searches set up and keep an eye out for things. In no rush to get 10GbE fully setup until start of next year anyway

 

[nry]

Looks like I can get a Dell PowerConnect 8024F dirt cheap, little scratched up but should do the job.

And found a Dell PowerConnect 5524 on eBay for £415 with has 2x 10GbE SFP ports.

Thinking using the 8024F as the core switch with L3 routing as that seems to do everything?
Then have the 5524 as an 'edge switch'.

I am struggling to find how good the 8024F is at L3 routing though! Main issue with alot of this enterprise kit is there is next to no reviews etc online. Guessing it all goes by word of mouth?

The two come in at a little over £1000 which I think is excellent for two switches with 10GbE!! But as the saying goes 'you get what you pay for' makes me wonder if I'm going to be stuck with basically what is two L2 switches, due to them being crap at L3

Also while I see the 8024F has fan temperature control, I can't find any mention of that for the 5524 which concerns me a little, need to keep the noise levels fairly undercontrol!

Suggestions/comments on my mad idea?