50+TB NAS Build, vSphere Cluster and Network Overhaul

[mrkrad]

The 8024F uses the same strata broadcom as the xsm7224s but just uses an arm cpu.

You have to realize that broadcom sells their "FASTPATH" suite of software to all these guys. It runs on vxworks(#1), or linux or freebsd(#2).

So in the case of the 8024f and xsm7224s you are talking the same npu, same 128gb SLC CF card, same 512ECC sodimm (2gb max), different cpu and slightly different set of bugs/fixes.

you will find there is a crappy heatsink on the cpu, a massive heatsink on the broadcom, and small heatsinks on the nic ports (4 x 6 sfp and 1 x 10gbase-T).

Two generic fan sets, two generic delta power supplies.

Easy as heck to modify to be water cooled using some heat pump and water cooling kits if you want.

If you spend enough time looking at 10gbe switches you'll start to see that they all pretty much look the same.

it's mostly the software. And when I say software, i'm talking about advanced features.

Every problem you will find with high end nic's and switches are related to advanced features.

What exactly are you going to be routing at 10gbps? You have two 10gbps internet connections?

 

[nry]

The xsm7224s is L2 out of the factory isn't it then can be upgraded to L3 with an additional license?

My setup consists of a fair few VLANs, such as iSCSI, VoIP, Wireless, Guess, development and so on

My NAS can saturate a 10GbE link too, as with my SSDs I have just added it hits around 1.6GB/s read

I have some budget for doing this upgrade and the 8024F and 5524 fall into it with spare! So seriously considering this option

 

[mrkrad]

vlan's are used to separate traffic, they are not routing.

Use a router to get to the internet and do vpn and such?

 

[nry]

But from what I understand that is what I want interVLAN routing for.

To talk between the VLANs, eg on my phone (wireless VLAN) I'd like to be able to communicate with my media PC on which ever VLAN that sits on to control the mouse.

I will be using pfSense as my internet router, not too sure how that would get integrated with a L3 switch as currently that does everything (albeit a little slow)

 

[alex1002]

How are your hp switches for the iscsi traffic?

 

[nry]

Seem fine performance wise the whole network is poor though hence me looking at 10GbE. Im sure if they were just setup all on one VLAN it would perform slightly better.
Remember before I could easily max the 1GbE pipe out

 

[mrkrad]

all those features just add overhead and can slow things down.

why not just use openvswitch and setup a bunch of nic's on an ml350 and go to town? or direct link 10gb like the P2000/MSA units? every hop is latency and with 10gbase-T that's a lot of latency.

Honestly a pair of xsm7224s with nothing (just storage on one switch, lan on the other) - nothing special, fast, reliable, simple.

vlan's are a security nightmare and many switches leak when flooded - aka easy way to break in.

since most nic's are dual port just K.I.S.S.


$2600 - two 10gbe switches
$240 - 24 DAC cables (brand new !!)
$75 * 12 - emulex nic's (easy , solid, windows client drivers, vcenter plugin, esxi flashing, one unified driver for all generations)

*.servers on fiber latency 10gbe = win.

solid - no surprises - no dickin' around plug n play. That's what I'm talking bout folks.

Going to have to upgrade the firewall to two $109 L5639's to keep up

ebay bargain hunting FTW

I picked up half dozen nic's (intel,broadcom,qlogic) with SFP+ for $75 to $129 (dual SFP+ SR) - cabling is cheap. $1.25 a meter for laser optimized OM3 that can do 300 meter all day long (mono price fiber!)

I'm going to have to sell the other brands (broadcom,qlogic,netxen) they are too much work especially without windows 2008/2012 to deal with (flashing, management,firmware, monitoring). Just not worth the headache.

With 12 servers rocking full fiber - 100% uptime, set it up and fuggedaboutit.

I would like to find some 40gbe ethernet - should be not that hard since mellanox FDR 56gbps = 40gbps of ethernet right?

 

[nry]

I have purchased some new kit!
As mentioned previously I could get a Dell 8024F and Dell 5524 for just a little over £1k, well at £1100 and they should be with me on Monday.

I was thinking about having 2x 5524 switches, one to live with all the servers to drive the normal 100/1000 ethernet kit and one to be next to my desk, which would be used for my laptop, and building/testing of kit. Probably slightly overkill.
One of the 10GbE ports to the 8024F and one to my Mac

But the problem with a second one at my desk would be getting a fiber link to it, might be almost impossible while I already have a few cat6 runs. If I could find a switch with a 10Gbase-T uplink to connect to the 8024F I might be onto a winner. But then I would need a SFP+ port for the Mac as all solutions within a remotely acceptable price bracket are SFP.
Did look at the Dell 6224 as you can add two modules into it, one SFP+ and one RJ45. Could be ideal but this will be later in the future.
Have seen a second hand 6224 go for £110 on eBay and a 10GbE SFP module for £56! Going to just keep an eye out and see what I can find. As this isn't a top priority just yet. Will probably wait until the Mac Pro is out and spend a few grand getting that all connected up etc

I know VLANs can be a nightmare if setup incorrectly (which they probably will) but I really feel I need them to separate out the traffic for my development. Also when I have external developers working with me I would like the ability to limit what they have access to giving them a selection of resources basically, while it may not be perfect or 100% hacker proof I feel fairly confident that they wouldn't bother trying to hack into anything seeing as I push so much work their way!

There is no way I need 2x 10GbE switches! At most I would probably make use of the dual ports on the NICs and use one for LAN and the other for iSCSI. With that approach that's still 12 servers which I can't see me hitting any time soon!

On the NIC front I managed to get a Dell 2094N and Dell 942V6 which are basically X520-DA2 cards from what my googling shows. Paid £89 and £92 for them, one is low profile the other is full profile and unfortunately the company only had 1 of each in stock
Found another full profile one on eBay for £150 and a bunch of low profile ones for £185!! but if I wait and keep an eye out I am hoping for a few more around the £100 mark.
Had a look at importing some and don't think that it's going to be worthwhile.
No idea where you found them that cheap, all ones I found were $150+

I was also under the impression that fiber uses more power than DAC cables?

I had a look at the infiniband stuff, that's even rarer over here not seen any 20Gb versions never mind 40!

Anyway some photos...

To get PWM control of my 60mm fans in node 0 I got this controller, not tested yet but should do the job

Before deciding to make the switch to SFP+ I bought a Dell RK375 off eBay for well under £100 doubt I will have any need for this so will probably just sell it on now.

And some NICs and DAC cables!

Bought 2x HP X240 1.2M SFP+ cables for £37, and 2x Dell 337MK 5M SFP+ cables for £19 each. Really could do with some 2-3m ones to make cable management possible but they are all well over £40

Also how do you remove one of the SFPs I put it in the card to test it and now I can't remove it. Seems to me you pull the little cord and they should just slide out but no luck

Really tempted by one of these 2U Supermicro 6026TT-HDTRF servers
Supermicro SuperServer 6026TT-HDTRF 2 x System Nodes - Barebones Server | eBay
Then I would want to max each one out haha, would be a little too expensive I think

Got alot of plans at the moment, just trying to control my spending problem haha

Thanks for all your input by the way, much appreciated!

 

[mrkrad]

SFP optics - you have to remove the fiber then you can pull this little latch down and tug out

DAC cables have a strap to do the similar action. the little ringlets on those H3C cables are what you pull on.

Some SFP+ optics are a real bitch to remove! But I suppose you will never need to remove it since it is just solid man.

DAC cables can perform differently than fiber but at short hauls (1,2,3meter) they rock.

I think you'll be really happy with that setup. the 8024F is solid. The cables are excellent quality. The cards are top shelf goods

If you have any problems let me know.

TIPS: serial console, disable XON/XOFF - boot loader secret menu option 30 (google the password) lets you change anything (serial#, asset tag, mac address, passwords, etc). All FASTPATH/vxworks switches have this menu some are password protected, most are not lol.

Leave flow control off with 10gbe !! the hosts and switch and guest vm's.

 

[nry]

Cheers

I kept trying to pull the ringlets but no go. Got a little screwdriver and carefully put some pressure on the clip (essentially what pulling that cord does...) and it just removed no hassle

Really tempted by this 2U server now, but finding suitable memory for the Supermicro boards cheaply is a pain and don't really need it. Also would like to upgrade my NAS with a Supermicro X9SCM board or similar for IPMI but again would need suitable memory. Guess I am going a little OTT here, wish it was all done properly in the first place!

 

[TallGraham]

Wow! This is looking even better now with the 10Gb networking.

I used to be a Cisco Partner in a former life and the college I now work at has Cisco networking everywhere. Trust me when I say that the high spec switches you are looking at will easily runs 100's of VLANs quite easily. The processors in them are designed for this. Easy thing to remember with networking is Layer 2 is switching and Layer 3 is routing. A Layer 3 switch is basically a switch that can do routing as well.

Have you thought about stacking switches? If they are decent like the 3750X then you can create a etherchannels that split across both physical switches as when stacked it acts as one big logical switch. Means you have redundancy then if a switch goes down. 3750X switches also stack power which is super cool. So if a power supply goes the other switch will keep it powered.

I've been saving up for some more bits for my build. Hopefully going to get the NAS/SAN part built soon.

 

[nry]

I have a feeling they should be ok, they probably won't even see close to the traffic that they are designed for.
I have looked at the 3750X but it only has 2x SFP+ ports with the very expensive add in module so decided against it. Even though I did like the reviews etc it got.

Seen your update in your post, look forward to some photos!

I have just purchased a Supermicro X8SIL-F for £49, now to find a cheap 1156 xeon processor. This is for my mobile rack which doesn't need to be high powered, Got my eye on one or two on ebay at the moment.
Thinking about upgrading Node 0 to 32GB RAM and using the RAM currently in that, in this new node.

Also considering losing Node 1 (Xeon 1245 w/ 32GB DDR non-ECC ram) and replacing with a Supermicro dual node 2U system, but not too sure on this just yet.

Also would like to replace the NAS board with something with IPMI too but I think that will wait

 

[nry]

Networking

Well the switches arrived today in the worlds largest box.

Even though the photo of the 8024F and description said it was scratched I seem to have one which looks basically new?!

This thing is LOUD, much worse than I thought it would be

And the Dell 5524, which is almost silent once booted up was fairly noisy during boot.

Now how do I access the console haha

 

[nry]

After alot of messing around, ordering a DB9 to RJ45 lead for £2
I got the soldering iron out and made a lead. As I wanted to log into the console.

Finally got into the web interface to poke around and it seems pretty decent, very responsive!

 

[nry]

Networking

Opened up the 8024F switch today, curious to see whats going on inside and any possible cooling modifications which could be made.

Looks like each PSU has 2x 40mm fans too!

Been thinking that putting a few 120/140mm fans on the top of the case might do the trick, but I am unsure at the moment. Don't want to damage the unit!

 

[Jeggs101]

That is a nice switch! I had to look up what the 8024F was. Not bad for around $3k

 

[nry]

Node 5

Some more toys arrived today.

Supermicro X8SIL and Xeon X3470 picked up fairly cheaply off eBay for my mobile rack.

Now to find 32GB DDR ECC for Node 0!

 

[mrkrad]

$11 for 4gb RDIMM ECC is not bad! The 16GB modules are about $155 (new) to $130 (used).

 

[mrkrad]

wow! that's identical even the power supply model is the same as the XSM7224!! where is the CF card? Mine has 128meg APACER SLC and then there is a 512meg ECC SODIMM (can do 2GB !!)


The big heatsink is the strata broadcom NPU - notice the weak heatsink (mine didn't even use goop!) and the CPU likewise.

I was thinking you could do a water cooler on the NPU and CPU and perhaps one big 120mm for the PHY . If you do not use the 10GBASE-T ports that is probably 15 watt each of heat (LOTS!).

google the secret menu password -> boot with putty com port with XON/XOFF flow disabled , boot to rom monitor, and use secret menu 30

All the VXWORKS function the same. You can change anything using the 30 menu.

What I did was remove the CF card, and DD dump and verify the backup of the rom in case it ever got roached, then I did some playing around and it's funny you can change mac addres, serial, asset tag with menu 30

That truly is a beauty of a switch - I don't think folks understand the power of 24 10GB ports "that just works".

Mine can do 1 or 10GB on fiber and the gigabit can drop down to 100 or 1000

 

[nry]

Not too sure where the CF card is, can't see it looking at the high res photo. This has 1GB ram by the looks of it

I have thought about watercooling it, but as this will be sitting at the top of my 42U rack with thousands of pounds worth of kit in, I would feel much better if it was all air cooled incase it leaked!

Think my plan is to remove the heatsinks and check there is something there, maybe even replace with arctic silver 5. Then knock up a prototype top lid in MDF with space for 4x 120mm fans and see how it gets on.
If it works I will look at getting a sheet of 3-5mm plastic to form a new case top (do not want to cut the original incase I sell it on in the future!) and fit 4x 140mm fans which I intend to buy.

Will have a look into this secret menu, still not even got into the console on this thing yet!!

Once again thanks for all your input Do you have any photos of your setup?