50+TB NAS Build, vSphere Cluster and Network Overhaul

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
N

nry

Active Member
Feb 22, 2013
312
61
28
Some more updates!

The good bits!

Bought a bunch of new 3M Dell 3M DAC cables for a very reasonable price :)



I have wanted an external SSD for some time now as my 500GB external drive is extremely slow, but as my iMac or MBP does not have USB3 I had to look to other options. I found the LaCie 256GB rugged thunderbolt/usb3 enclosure for £225 delivered. It's now £215 typical.



Quick benchmark :)



The bad bits!

1) Transferring files over 10GbE to it (with the drive daisy chained to the Sonnet enclosure) I see a maximum of 140MB/s which is too slow for my liking! But I want to fix the issue with the slow 10GbE first...



2) On the 10GbE testing front I am still struggling to get full performance out of the NICs, when I first tested I was getting 9.09 Gbits/sec now I get under 5 using the same cables. I must have changed a setting somewhere to cause this

3) On the Mac I seem to hit 4.37Gbit/sec maximum all the time now either direct to another machine or through the 8024F switch

Code: 
➜  ~  iperf -c 10.0.21.125
------------------------------------------------------------
Client connecting to 10.0.21.125, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 10.0.21.124 port 51762 connected with 10.0.21.125 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.08 GBytes  4.37 Gbits/sec
➜  ~  iperf -c 10.0.22.101
------------------------------------------------------------
Client connecting to 10.0.22.101, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 10.0.22.100 port 51809 connected with 10.0.22.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.09 GBytes  4.37 Gbits/sec
4) Also having issues where the Intel SFP+ Modules I picked up use to work with the Intel X520-DA2 NICs without any issue, now they simply will not detect them yet they work fine between the Dell 8024F and Dell 5524 switch!

5) I bought 3x 60mm fans to replace the poor ones in N0, turns out I needed 50mm :(

:confused:
 
Last edited:
N

nry

Active Member
Feb 22, 2013
312
61
28
On issue 2 above, where I wasn't seeing full performance of the 10GbE network I have done some more tests... and this is looking like a non issue

Some tests, testing different DAC cables too as I wasn't sure if the 3M ones were performing correctly...

NAS direct to N5 (5m DAC)

Code: 
root@nas:~# iperf -c 10.0.22.102
------------------------------------------------------------
Client connecting to 10.0.22.102, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.0.22.101 port 36949 connected with 10.0.22.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.38 Gbits/sec
NAS > 8024F > N5 (5M DACs)

Code: 
root@nas:~# iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.0.21.125 port 49943 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec
NAS > 8024F > N5 (5M DAC / 1M DAC)

Code: 
root@nas:~# iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.0.21.125 port 49944 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec
NAS > 8024F > N5 (5M DAC / 3M DAC)

Code: 
root@nas:~# iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.0.21.125 port 49945 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec
VM (N0) > 8024F > N5 (3M DAC / 5 M DAC)

Code: 
x1nick@ubuntu:~$ iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.0.21.123 port 50466 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec
Next was to connect the iMac up and do similar tests...

Code: 
➜  ~  iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 10.0.21.124 port 53614 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.09 GBytes  4.37 Gbits/sec
Still no luck!

Read somewhere changing the tcp window scale option can improve things, not that I have any clue what this is, bug figure I'd give it a go as by default it is on

Code: 
➜  ~  sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323: 1
➜  ~  sudo sysctl -w net.inet.tcp.rfc1323=0
net.inet.tcp.rfc1323: 1 -> 0
And the results, well that didn't do much good :rolleyes:

Code: 
➜  ~  iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 10.0.21.124 port 53661 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.32 GBytes  1.99 Gbits/sec
I have followed every tip inside the Myricom readme here https://www.myricom.com/scs/READMES/README.myri10ge-macosx
And a whole range of tips/changes here https://www.myricom.com/software/my...hoot-slow-myri10ge-or-mx-10g-performance.html

Also tried the suggestions for the iperf options...

Code: 
  ~  iperf -c 10.0.21.102 -w 768k -l 256k
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size:  769 KByte (WARNING: requested  768 KByte)
------------------------------------------------------------
[  4] local 10.0.21.124 port 54258 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.04 GBytes  4.33 Gbits/sec
But still around the 4.37Gbit/s limit, starting to think maybe the thunderbolt link is limiting it somehow or the Sonnet enclosure isn't good enough. I think the card only runs at PCIe x4 mode in this enclosure but that should give it enough bandwidth to max out 10GbE

On the Myricom readme they provide a way to show the PCIe bandwidth...
Code: 
➜  ~  sysctl net.myri10ge | grep dma
net.myri10ge.en3.dma_read_bw_MBs: 806
net.myri10ge.en3.dma_write_bw_MBs: 825
net.myri10ge.en3.dma_read_write_bw_MBs: 978
net.myri10ge.en4.dma_read_bw_MBs: 805
net.myri10ge.en4.dma_write_bw_MBs: 825
net.myri10ge.en4.dma_read_write_bw_MBs: 978
I seem to be limited to 540MBs which I'm sure I could live with :rolleyes: but would like to max this device out if possible!

Any suggestions would be more than welcome :)
 
N

nry

Active Member
Feb 22, 2013
312
61
28
Networking

Got a few hours free today so figured I'd try silent up the 8024F

With my makeshift fan shroud made of wood (for testing)

One thing to note was that the fan headers are not the standard layout and had to move the pins around to get them to work.



The noise was noticeably less high pitched but the issue is the PSU still has 2x 40mm fans which are also VERY noisy and I am guessing these don't spin down.

So opened up the PSU to see if this could be cooled from above



My issue with what I am doing here is that the air is being blown down onto components with no real thought into where it's going to go after. Just hoping for the best basically which isn't the best approach.

Another issue is the switch boots with these fans connected but once fully booted the fault light is flashing probably due to RPM speeds. Guessing there is no option for altering the minimum RPM speed.
 
Last edited:
N

nry

Active Member
Feb 22, 2013
312
61
28
Some more photos....

Node 5

All finished now



Node 0
Got 4x 50mm fans to replace the ones in Node 0. Not too impressed with the airflow of these





As the 2U case only supports 3x 50mm fans I had to improvise a little with sticky pads



Temp rack

Top down:
Dell 8024F
Node 5
Node 0
Node 2
NAS
Node 1
APC SmartUPS 2200



Rear wiring



House wiring combined with some other bits and Node 4

 
3

33_viper_33

Member
Aug 3, 2013
204
3
18
What chassis/server is your nas node? I'm trying to decide if I want a SM, chenbro or Norco 24 bay chassis. My main concern is noise which is why I was leaning toward the NORCO with 120mm fans. How is the noise level of yours? What hardware and os are you running?
 
R

Rain

Active Member
May 13, 2013
276
124
43
33_viper_33 said:
What chassis/server is your nas node? I'm trying to decide if I want a SM, chenbro or Norco 24 bay chassis. My main concern is noise which is why I was leaning toward the NORCO with 120mm fans. How is the noise level of yours? What hardware and os are you running?
Click to expand...
While it doesn't specifically answer your question, I'll throw in a vote for the Norco. I have a RPC-4224 with the 3x120mm fan wall and three 120mm Noctua fans. The drives stay nice and cool and its very quiet! I'm also a fan of Norco's drive trays.
 
N

nry

Active Member
Feb 22, 2013
312
61
28
Its X-case which I am fairly sure is the same as Norco but with a X branded front. Basically for the UK market.

The stock fans in this case were VERY noisy! I first changed them for Sharkoon 120mm fans, but the CFM and RPM of these was far to low to cool the 5K Hitachi drives never mind the 7K Hitachi drives and my drives started hitting 50-70oC with an ambient temperature below 20oC. So have dug some fans out which I would are inbetween the Sharkoon and the standard fans (roughly at a guess)

These are then controlled by the motherboard fan headers and I think range from 8v to 12v. While not silent I would say that it's quiet enough to not be heard else where in the house. If it was next to my TV then that's a whole new issue!

What you need to remember with these cases there is very little room around the disks for air to travel, so a fan with some ability to shift air is a must.

Some may have had luck with running these sorts of cases quietly but with all my testing I can honestly say I have had no luck. Even with the near enough freezing temperatures all year round here in the UK!

To be honest I have long given up on trying to make this server silent. If I remember rightly there are 9 fans and 16 HDDs spinning away, basically fighting a losing battle there lol

Hope this is of some help.
 
N

nry

Active Member
Feb 22, 2013
312
61
28
Slight update

Having no end of issues with my 4x 4GB DDR3 ECC Ram modules, think 2 have failed on me just haven't had the time to test this properly.

Networking

Managed to order all the parts to finish off the cooling on my 8024F 10GbE switch :)
Not completely silent, nor would I class it as quiet but not having the high pitched noise of the 40mm fans is so much nicer. Runs much cooler too but the firmware reports fan failures due to the lower RPM, may look at having a little Arduino chip to read the fans RPM and output a higher RPM but this will have to wait till the new year I think



Generic

Not really relevant to this post but figured I'd post it anyway :p

Treated myself to a new tool box and a bunch of new tools. I managed to pick up basically the whole range of Stanley VDE screwdrivers and pliers etc for under £50 off eBay off someone who was having a clearout, all look new too. I guess I don't have much need for VDE range but they were cheaper than buying the standard ones separate so figured why not.





Would like to find myself a better network cable tester which doesn't cost the earth, but again not a high priority just yet.
 
Last edited:
N

nry

Active Member
Feb 22, 2013
312
61
28
Haven't really set the 10GbE switch up yet, more on that later when I get time.

They are Samsung modules, they use to work fine but for some reason have given up.
Tried them in my X9SCM board too and only shows up as 12GB. Maybe this is a good excuse to buy another 32GB haha
 
N

nry

Active Member
Feb 22, 2013
312
61
28
Mobile network

During November I have a number of events where I am required to setup a wireless network with fail over support. This will be done using Node 0 (primary) and Node 5 (secondary).
The requirements are:
- I must be able to accommodate a single device failure eg wireless access point, server, switch or cabling issues.
- The events power supply may have power glitches causing the servers to reboot so need to be protected by a UPS. As I need to allow for complete UPS failure I require 2 UPSs. Managed to source a used one (photoed) and a new one off eBay fairly cheaply. Complete power failure isn't a major issue as the event will not continue in this case.
- Aim for glitch free fail over (wishful thinking!)
- Live backup (mySQL replication between the two servers will do the job)

Photoed below is Node 5 with the used APC 450 UPS inside a short 4U ABS case. This weighs a fair bit! Will try get it on the scales at some point.





 
Last edited:
N

nry

Active Member
Feb 22, 2013
312
61
28
Mobile Network

Got a UniFi Pro access point for this project I'm working on (well technically two)



Here is the full setup in testing, with a handful of the iPads...

 
Last edited:
N

nry

Active Member
Feb 22, 2013
312
61
28
Workstation Upgrade

When I first purchased my iMac it only had an i5, 4GB ram and a 1TB hard disk. After a few upgrades in the past including installing a OCZ SSD Vertex 4 I finally decided to upgrade the processor.

Managed to snap a i7 2600 off eBay for £109 :D

Removing the old processor...



All working :)



Overall was fairly confident in doing all of this, for me the most difficult part was putting the glass back on without any dust inside. Unless you have a clean room I think this may be impossible!

Also had an eye out on eBay for a matching Samsung 2443 monitor for the last year and one finally popped up under £100 so snapped that up too.



No full photos of my whole workstation, but a full spec list so far
  • iMac 2011 27"
  • - i7 2600 CPU
  • - 2x 4GB DDR3, 2x 8GB DDR3 (total 24GB)
  • - OCZ Vertex 4 128GB
  • Samsung 2443BW (left monitor)
  • Samsung 2443 (right monitor)
  • Razer Mamba 2012 Mouse
  • Sonnet Echo Express SE (Thunderbolt PCIe enclosure)
  • - Myricom 10G-PCIE2-8B2-2S (10GbE PCIe card)
  • - Myricom 10G-SFP-SR (Fiber SFP)
  • Denon AVR 1910 Amp connected to iMac via optical
  • Kef HTF7003 LCR soundbar
  • Kef KUBE-1 Subwoofer
  • Samsung ML2240 Laser printer
  • APC Switched PDU AP7920
Do have my eyes on the new CalDigit Thunderbolt station too as I have no run out of thunderbolt ports on my iMac :eek:
This would also give me USB3
CalDigit Thunderboltâ„¢ station
 
Last edited:
N

nry

Active Member
Feb 22, 2013
312
61
28
Arduino Investigation

Decided to buy a £5 Arduino clone with the intention of investigating using this as
1. A projector screen relay control, possibly ethernet based
2. A fan RPM converter for the Dell 8024F switch
So far I made an LED flash.... exciting stuff!

 
M

mackle

Active Member
Nov 13, 2013
221
40
28
I love the idea of using an Arduino or a rasp-pi as a low powered system/network monitor.
 
N

nry

Active Member
Feb 22, 2013
312
61
28
PoE Network

Managed to snap up a brand new HP 2520G-8 PoE switch off eBay for under £100. Slightly overkill for what I will be doing with it, but considering the price difference between this and the 4 port Netgear and similar I consider it worth while :)



Not too sure if I posted these before, but they are PoE splitters by TPLink which simply split the voltage supply out from a PoE switch. Picked this up for £7 each :)




Arduino Networking

Bought an ethernet module for my Arduino chip, the ENC28J60, while it has some bad reviews I figure I would give it a go for £5.

Here it is compared to my Arduino board and the PoE splitter!

 
Last edited:
You must log in or register to reply here.
Top Bottom