50+TB NAS Build, vSphere Cluster and Network Overhaul

[nry]

Workstation Ports

As mentioned previously I had my eye on one of these CalDigit Thunderbolt Stations as
1) I have run out of USB ports
2) I don't have USB3 on my 2011 iMac
3) I have run out of thunderbolt ports on my iMac due to the additional displays.

Today it arrived

 

[MiniKnight]

Great stuff. How are the TP-Link splitters working? You got a great deal on them.

Also that switch. Wow man! Best we see in the US is around $400.

 

[nry]

Those TP-Link splitters can be picked up new for £11 on eBay here. If they prove to be successfully I may pick up a few more for other project ideas.

But currently they don't seem to work with the PoE switch, the green power LED on the splitter simply pulses on and off slowly. Which I believe is a power issue looking at the manual.

Now I have no idea what the issue could be or how to resolve it, need to plug one of the UniFi AP's into the switch and make sure the switch isn't actually faulty! Might explain why the switch was so cheap haha

 

[MiniKnight]

Those TP-Link splitters can be picked up new for £11 on eBay here. If they prove to be successfully I may pick up a few more for other project ideas.

But currently they don't seem to work with the PoE switch, the green power LED on the splitter simply pulses on and off slowly. Which I believe is a power issue looking at the manual.

Now I have no idea what the issue could be or how to resolve it, need to plug one of the UniFi AP's into the switch and make sure the switch isn't actually faulty! Might explain why the switch was so cheap haha

Wow - found similar pricing here $16 new. TP Link TL POE10R Poe Splitter | eBay

 

[nry]

if they work well could be a really useful device.

One of my plans is to use a Raspberry Pi with a camera module and PoE splitter.
Find a nice case and I will have a full HD IP camera for under £50

In theory anyway!

 

[Mike]

Doesn't the raspberry do only 100mbit? You could use the 2 remaining twisted pairs ghetto style and put 5v or so on it, or regulate on the other end. .

 

[nry]

Doesn't the raspberry do only 100mbit? You could use the 2 remaining twisted pairs ghetto style and put 5v or so on it, or regulate on the other end. .

I have tried this and while it works perfectly fine, I didn't feel too confident mixing a bunch of hacked network leads into my network potentially causing damage to equipment if I connect something up incorrectly.

 

[seang86s]

Those TP-Link splitters can be picked up new for £11 on eBay here. If they prove to be successfully I may pick up a few more for other project ideas.

But currently they don't seem to work with the PoE switch, the green power LED on the splitter simply pulses on and off slowly. Which I believe is a power issue looking at the manual.

Now I have no idea what the issue could be or how to resolve it, need to plug one of the UniFi AP's into the switch and make sure the switch isn't actually faulty! Might explain why the switch was so cheap haha

FWIW, I have used the D-Link version of this hardware successfully to PTZ cameras, Linksys Media Center extenders and possibly the first MP3 car stereo on the market - the empeg.

Keep in mind that POE can deliver 15 watts max, so either 1.5 amps @ 12VDC or 3 amps @ 5VDC.

 

[nry]

I think my POE splitters are limited to either 10 or 11w. Either way for what I have planned I should be well below this.

 

[nry]

Thunderbolt Toys

Spent some time messing around with the thunderbolt toys I have. Unfortunately this haven't been as plug an play as I had hoped for.

The three thunderbolt devices I have are:
- CalDigit Station
- Sonnet Echo Express SE (with 10GbE card)
- LaCie 256GB SSD

Here it all is wired up on my MBP, pretty neat to have all of this running from a single port on the Mac

Using a device on it's own with no daisy chaining there are no issues. Every device works as expected.
Start introducing daisy chaining and the fun begins...

MBP > CalDigit > LaCie
- No issues recorded here

MBP > Sonnet > LaCie
- No issues recorded here

MBP > CalDigit > Sonnet > LaCie
- USB ports on CalDigit work for 10 seconds before disconnecting and reconnecting
- Connecting a Razer Mamba mouse to the CalDigit causes the MBP to restart
- LaCie shows up inside the system information thunderbolt tree but not inside disk utility

MBP > Sonnet > CalDigit > LaCie
- LaCie shows up inside the system information thunderbolt tree but not inside disk utility

Here is the thunderbolt tree, the 'Rugged mini' is the LaCie drive.

Anyway thought I should benchmark the USB3 support of the CalDigit Station, these are connected in the following thunderbolt configuration
MBP > Sonnet > CalDigit

SanDisk 32GB Extreme USB 3.0 Flash Drive

LaCie 256GB SSD connected via USB3

For comparison this is the LaCie drive connected via thunderbolt (almost identical to my previous post regarding this)

Now the 10GbE I am struggling to get anything over 2.6Gbits, previous testing on the iMac resulted in a maximum 4.7Gbits. Not too sure if this is due to the performance of a MBP.

 

[nry]

Thunderbolt Networking

According to this link OS X 10.9 brings fast but choppy Thunderbolt networking | Ars Technica

It is possible to link two Mac's together with a thunderbolt lead and create a 10Gb link, neat idea but when I attempted it couldn't get either machine to recognize the fact I had plugged them in.

If anyone has any ideas on this I would be interested to hear them
If not I will come back to this at a later date.

 

[nry]

Thunderbolt Networking

Rebooted both machines and connected up the thunderbolt lead, worked first time this time round!

Unfortunately performance isn't quite as high as I expected it to be under initial testing...

➜  ~  iperf -c 10.0.22.20
------------------------------------------------------------
Client connecting to 10.0.22.20, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 10.0.22.10 port 53967 connected with 10.0.22.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.21 GBytes  1.90 Gbits/sec


➜  ~  iperf -c 10.0.22.20 -t 30 -P 2
------------------------------------------------------------
Client connecting to 10.0.22.20, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  5] local 10.0.22.10 port 53980 connected with 10.0.22.20 port 5001
[  4] local 10.0.22.10 port 53981 connected with 10.0.22.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-30.0 sec  3.19 GBytes   914 Mbits/sec
[  4]  0.0-30.0 sec  3.17 GBytes   908 Mbits/sec
[SUM]  0.0-30.0 sec  6.36 GBytes  1.82 Gbits/sec


➜  ~  iperf -c 10.0.22.20 -t 15 -P 2 -l 256k
------------------------------------------------------------
Client connecting to 10.0.22.20, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 10.0.22.10 port 53994 connected with 10.0.22.20 port 5001
[  5] local 10.0.22.10 port 53993 connected with 10.0.22.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-15.0 sec  1.50 GBytes   861 Mbits/sec
[  5]  0.0-15.0 sec  1.50 GBytes   861 Mbits/sec
[SUM]  0.0-15.0 sec  3.01 GBytes  1.72 Gbits/sec


➜  ~  iperf -c 10.0.22.20 -w 512k -l 256k -P 2 -f m -t 60
------------------------------------------------------------
Client connecting to 10.0.22.20, TCP port 5001
TCP window size: 0.50 MByte (WARNING: requested 0.50 MByte)
------------------------------------------------------------
[  5] local 10.0.22.10 port 54576 connected with 10.0.22.20 port 5001
[  4] local 10.0.22.10 port 54577 connected with 10.0.22.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-60.0 sec  6392 MBytes   894 Mbits/sec
[  4]  0.0-60.0 sec  6392 MBytes   894 Mbits/sec
[SUM]  0.0-60.0 sec  12784 MBytes  1787 Mbits/sec

 

[nry]

PoE Network and WiFi upgrade

Happy new year to who ever is still reading this! Hope everyone had a good Christmas and new year.

While feeling slightly worse for wear today I decided to upgrade my wireless access point to the Pro model which I had posted previously, powered by the 8 port HP ProCurve PoE switch.

First up was testing power consumption, 4w idle. Only slightly more than the standard model.

The switch uses almost 9w idle which is less than I expected for a PoE switch, pretty happy with this. With the access point connected it hits around 14w maximum and idles around 12w.

New access point in the place of the old one We have a much brighter night light for the hall way now haha

For now the network wiring is complete until I have time to work on the Arduino PoE modules or maybe run some cat6 to the back of the house to get better WiFi in the garden, but that's very low priority right now.
All 24 ports on the Dell 5524 switch are full and the 2 10GbE ports are in use too! Probably should have bought the 48 port PoE version.

WiFi Benchmark

Hopefully will get chance to test the multi client performance in the future, but for now here is the before and after iperf results.

Standard UniFi AP

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.0.5.10 port 5001 connected with 10.0.30.102 port 49487
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.1 sec  44.9 MBytes  37.4 Mbits/sec

UniFi Pro AP

------------------------------------------------------------
Client connecting to int.lsnet.co, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  5] local 10.0.30.102 port 57835 connected with 10.0.5.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   109 MBytes  91.6 Mbits/sec

 

[TallGraham]

Yes, I am still reading everything

Great idea for using the wireless access point as a night light in the hallway too. I am getting there slowly but surely with my build. Pesky things like Christmas and New Year seem to make holes in the hardware budget.

 

[nry]

NAS Upgrade

One thing which has been bugging me for some time now was not having IPMI on my NAS motherboard. The Asus P8B-WS is a nice board, but a more workstation orientated board after all. Have a few saved searches on eBay to try help my spending problem. Last week a refurbished Supermicro X9SCM popped up, managed to snap that for £87 delivered

For now I have taken 16GB memory out of Node 0 and placed this in the NAS as the original memory was non-ECC.

Exciting photo of the board in place!

May put the Asus board to use in the remote backup server with the 2500S chip from my iMac as that supports VT-d which means I could potentially have ESXi installed on it with HDD controllers passed through to a ZFS server.
Really need to finish that backup server as currently I only have about 80% of my data backed up off site!

 

[nry]

Workstation

Been incredibly busy with work the last few weeks so haven't had chance to work on any of this. Today I spent some time while feeling a little sorry for myself, investigating the power usage of my workstation. The results...

iMac
- Off - 0-2.1w (switches directly between the two)
- Start-up - 166w
- Desktop load - 220w (max recorded during monitoring)
- Idle (monitor on full brightness) - 146w
- Display off - 49w
- Sleep - 2.2w

Monitor 2443BW (older)
- Off - 0w
- On - 43w
- Standby (flashing led) - 3w

Monitor 2443 (newer)
- Off - 0.5w
- On - 44w
- Standby (flashing led) - 0.6w

Caldigit Thunderbolt Dock
- Off - 0.7w
- On (standard devices loaded) - 8w
- On (LaCie SSD and CF adapter in use) - 13w max

Sonnet (with 10GbE adapter)
- Off - 1.7w
- On - 21w (10GbE offline)

Samsung Printer
- Idle - 5w
- Printing/startup - 840w max

Denon 1910 Amp
- Off - 28w
- On (no music) - 49w
- On (normal music level) - 49w
- On (louder music volume) - 51w

Kef Sub
- Does not turn off (even after 30 mins)
- On (no music - green led) - 9.6w
- On (music playing normal volume) 10w

APC 8 port (currently not in use)
- All ports on - 8-9w
- All ports off - 3.9w

Netgear 5 port (currently not in use)
- No devices/Idle - 3w

Quite surprised by some of these figures! I thought the monitors would have been higher, but at the same time didn't expect the off usage of things like the amp and sub to be so high! I work this out to be roughly for the whole lot

Off: 42w
Standby: 51w
On: 326w
Maximum: 1246w

While I always try and turn everything off at the wall, I am pretty lazy when it comes to doing this and usually just put the iMac to sleep leaving me with a fairly substantial usage of 46w idle! (printer is never left on when not in use).

At the current British Gas kWh price of 11.96p (Not 100% sure this is the correct one!) to leave my kit idle is £3.96 per month.
Obviously it's not idle all the time, if I am working a standard week (50 hr week) it works out at about £2.86 per month.

While to some this may be nothing, our current electricity bill is around £180 per month (we use gas heating as well which is not included in this). So am currently looking at ways of cutting this down.

Toying with the idea of putting the APC 8 port PDU at the workstation to automatically turn the kit off when not in use, should be fairly easy to put a script together to detect if the iMac is awake or not. This would bring the idle usage down to about 6w

Or I could stop being so lazy and actually turn the kit off properly...

 

[TallGraham]

Or I could stop being so lazy and actually turn the kit off properly...

And where is the fun in doing that?

 

[nry]

My thoughts exactly

 

[gigatexal]

was the expander getting too hot, warranting the need for a heatsink/fan?

 

[nry]

was the expander getting too hot, warranting the need for a heatsink/fan?

Originally it had a small fan on its heatsink which was incredibly noisy and the heatsink was too hot to touch. So guess it does get pretty hot!
With the scythe fan on it's near enough silent and can feel no heat what so ever (idle drives).