Alright, Finally got my fibers and transceivers from the Fiber Store.
I may have slightly kinked the fiber when I was running it through my hole to the basement, but it doesn't seem to be impacting it (at least not much). I have never dealt with these fibers before, so I have no idea how sensitive they are to kinking.
Anyway, I ran it between my Linux Mint 17.1 workstation on one end and my ESXi 5.5U2 server on the other. Set up as below. Vswitch1 is a dedicated network for storage traffic, so I keep it away from everything else. This is where I connected the brocade. All of the guests connected to this vswitch are using vmxnet3.
So I just popped in the transceivers, connected the fiber and everything just worked. No trouble shooting or anything. Off to the races. I wish my other current project (MythTV Backend with XBMC/Kodi Frontend) were behaving this well.
First test, ssh into my Ubuntu server guest to start up an iperf in server mode and then test from my workstation:
Code:
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.xxx.xxx port 54545 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.76 GBytes 8.02 Gbits/sec
Not quite where I had hoped, but not too shabby for my first try, using default settings (no -P 30)
Tried again with -P30 and 120 seconds instead.
Code:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-119.0 sec 4.46 GBytes 322 Mbits/sec
[ 5] 0.0-119.0 sec 2.13 GBytes 153 Mbits/sec
[ 6] 0.0-119.0 sec 4.96 GBytes 358 Mbits/sec
[ 9] 0.0-119.0 sec 4.96 GBytes 358 Mbits/sec
[ 10] 0.0-119.0 sec 4.62 GBytes 333 Mbits/sec
[ 11] 0.0-119.0 sec 1.98 GBytes 143 Mbits/sec
[ 19] 0.0-119.0 sec 1.96 GBytes 142 Mbits/sec
[ 20] 0.0-119.0 sec 3.95 GBytes 285 Mbits/sec
[ 23] 0.0-119.0 sec 4.19 GBytes 303 Mbits/sec
[ 27] 0.0-119.0 sec 5.51 GBytes 398 Mbits/sec
[ 8] 0.0-119.0 sec 2.07 GBytes 149 Mbits/sec
[ 26] 0.0-119.0 sec 1.64 GBytes 118 Mbits/sec
[ 21] 0.0-119.0 sec 2.02 GBytes 146 Mbits/sec
[ 22] 0.0-119.1 sec 1.97 GBytes 142 Mbits/sec
[ 24] 0.0-119.1 sec 2.12 GBytes 153 Mbits/sec
[ 12] 0.0-119.1 sec 4.31 GBytes 311 Mbits/sec
[ 25] 0.0-119.1 sec 1.79 GBytes 129 Mbits/sec
[ 7] 0.0-119.1 sec 1.83 GBytes 132 Mbits/sec
[ 4] 0.0-119.1 sec 1.81 GBytes 131 Mbits/sec
[ 14] 0.0-120.0 sec 4.88 GBytes 350 Mbits/sec
[ 13] 0.0-120.0 sec 4.80 GBytes 343 Mbits/sec
[ 16] 0.0-120.0 sec 4.99 GBytes 357 Mbits/sec
[ 18] 0.0-120.0 sec 4.66 GBytes 333 Mbits/sec
[ 17] 0.0-120.0 sec 2.55 GBytes 182 Mbits/sec
[ 30] 0.0-120.0 sec 5.04 GBytes 361 Mbits/sec
[ 28] 0.0-120.0 sec 5.49 GBytes 393 Mbits/sec
[ 31] 0.0-120.0 sec 1.91 GBytes 137 Mbits/sec
[ 29] 0.0-120.0 sec 5.37 GBytes 384 Mbits/sec
[ 32] 0.0-120.0 sec 4.82 GBytes 345 Mbits/sec
[ 15] 0.0-120.1 sec 2.15 GBytes 154 Mbits/sec
[SUM] 0.0-120.1 sec 105 GBytes 7.51 Gbits/sec
Hmm, a little bit puzzling and a little bit disappointing, but I have other (albeit light) traffic going over the network so that could be throwing it off, or I could just be reaching the capacity of my hardware, or maybe that kind damaged the fiber after all (I do have another I can try with.)
Then I did iperf against my FreeNAS guest, on the same network on the same ESXi box and this is where it gets interesting.
Code:
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.xxx.xxx port 36620 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.46 GBytes 2.97 Gbits/sec
OK so this is a little alarming, especially since NFS transfers to FreeNAS was one of the main reasons I got the adapters, transceivers and fiber.
Tried again with -P 30 and -t 120
Code:
[ ID] Interval Transfer Bandwidth
[ 14] 0.0-120.0 sec 1.78 GBytes 127 Mbits/sec
[ 3] 0.0-120.0 sec 1.38 GBytes 98.6 Mbits/sec
[ 5] 0.0-120.0 sec 1.27 GBytes 90.7 Mbits/sec
[ 4] 0.0-120.0 sec 1.37 GBytes 98.1 Mbits/sec
[ 7] 0.0-120.0 sec 1.23 GBytes 88.2 Mbits/sec
[ 6] 0.0-120.0 sec 1.82 GBytes 130 Mbits/sec
[ 10] 0.0-120.0 sec 1.34 GBytes 95.8 Mbits/sec
[ 15] 0.0-120.0 sec 1.77 GBytes 127 Mbits/sec
[ 17] 0.0-120.0 sec 1.25 GBytes 89.4 Mbits/sec
[ 19] 0.0-120.0 sec 1.32 GBytes 94.2 Mbits/sec
[ 20] 0.0-120.0 sec 1.33 GBytes 95.1 Mbits/sec
[ 22] 0.0-120.0 sec 1.30 GBytes 92.9 Mbits/sec
[ 27] 0.0-120.0 sec 1.22 GBytes 87.5 Mbits/sec
[ 31] 0.0-120.0 sec 1.13 GBytes 81.0 Mbits/sec
[ 9] 0.0-120.0 sec 1.67 GBytes 120 Mbits/sec
[ 11] 0.0-120.0 sec 1.29 GBytes 92.5 Mbits/sec
[ 13] 0.0-120.0 sec 1.29 GBytes 92.7 Mbits/sec
[ 16] 0.0-120.0 sec 1.29 GBytes 92.5 Mbits/sec
[ 23] 0.0-120.0 sec 1.77 GBytes 127 Mbits/sec
[ 25] 0.0-120.0 sec 1.19 GBytes 84.9 Mbits/sec
[ 26] 0.0-120.0 sec 1.28 GBytes 91.5 Mbits/sec
[ 28] 0.0-120.0 sec 1.36 GBytes 97.4 Mbits/sec
[ 30] 0.0-120.0 sec 1.50 GBytes 107 Mbits/sec
[ 32] 0.0-120.0 sec 1.33 GBytes 94.9 Mbits/sec
[ 8] 0.0-120.0 sec 1.33 GBytes 95.3 Mbits/sec
[ 18] 0.0-120.0 sec 1.32 GBytes 94.1 Mbits/sec
[ 24] 0.0-120.0 sec 1.29 GBytes 92.3 Mbits/sec
[ 29] 0.0-120.0 sec 1.32 GBytes 94.2 Mbits/sec
[ 12] 0.0-120.0 sec 1.28 GBytes 91.8 Mbits/sec
[ 21] 0.0-120.0 sec 1.29 GBytes 92.6 Mbits/sec
[SUM] 0.0-120.0 sec 41.3 GBytes 2.96 Gbits/sec
Hmm. Rather disappointing and very surprising. considering last week (before I got the fiber) I ran an iperf between the same Ubuntu Server and FreeNAS boxes (vmxnet3 -> vswitch -> vmxnet3) and got 18Gbit/s with a single connection...
Just to rule out any brocade/transceiver/fiber problems I did the same again, Ubuntu Server to FreeNAS.
Code:
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.xxx.xxx port 45525 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.21 GBytes 2.75 Gbits/sec
Yep, definitely something up with FreeNAS, not the brocade.
So strange, since the results just last week were phenomenal, and I don't think I've changed anything.
As a final test, I DD:ed a virtual box drive image stored on FreeNAS via NFS to /dev/null on my workstation:
Code:
$ dd if=./Windows\ 7\ x64.vdi of=/dev/null
65120744+0 records in
65120744+0 records out
33341820928 bytes (33 GB) copied, 78.3183 s, 406 MB/s
Yeah, not meeting my expectations to FreeNAS at this point, though it is definitely faster than gigabit, so I guess that's not bad.
Off to troubleshoot FreeNAS and figure out what went wrong since last week, I guess....