Solaris 11.3 network slow...

kemic

New Member
Aug 15, 2015
14
4
3
Hey everyone, I've been racking my brain for hours now trying to figure this out. I have a fresh install of Solaris 11.3, setup a simple ZFS pool (4 disks raidz2), configured everything and started rsyncing some data over. That's when I noticed the absolute garbage speed of my 1Gb connection. I first thought it was my pool config but I was able to rule that out with some dd tests. I started monitoring the connection with dlstat and noticed the receive speed never goes over 2MB/s! For the sake of troubleshooting, I forced the port on the switch to do 100Mbps instead of 1Gbps and somehow my speeds went up to 9MB/s. :| I'm pretty stuck here... Anyone have any suggestions before I install another OS to determine if it's a hardware problem? I'm not on site but will check/replace the cable too tomorrow...
 

gea

Well-Known Member
Dec 31, 2010
2,489
838
113
DE
I would first check the pure network with iperf between server and client
 

kemic

New Member
Aug 15, 2015
14
4
3
Thanks for the quick reply!

Code:
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  4.50 MBytes  37.7 Mbits/sec
[  3]  1.0- 2.0 sec  4.38 MBytes  36.7 Mbits/sec
[  3]  2.0- 3.0 sec  6.62 MBytes  55.6 Mbits/sec
[  3]  3.0- 4.0 sec  2.62 MBytes  22.0 Mbits/sec
[  3]  4.0- 5.0 sec  8.12 MBytes  68.2 Mbits/sec
[  3]  5.0- 6.0 sec  4.12 MBytes  34.6 Mbits/sec
[  3]  6.0- 7.0 sec  3.88 MBytes  32.5 Mbits/sec
[  3]  7.0- 8.0 sec  2.75 MBytes  23.1 Mbits/sec
[  3]  8.0- 9.0 sec  3.50 MBytes  29.4 Mbits/sec
[  3]  9.0-10.0 sec  3.38 MBytes  28.3 Mbits/sec
[  3] 10.0-11.0 sec  2.38 MBytes  19.9 Mbits/sec
[  3] 11.0-12.0 sec  4.62 MBytes  38.8 Mbits/sec
[  3] 12.0-13.0 sec  5.25 MBytes  44.0 Mbits/sec
[  3] 13.0-14.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 14.0-15.0 sec  5.00 MBytes  41.9 Mbits/sec
[  3] 15.0-16.0 sec  3.88 MBytes  32.5 Mbits/sec
[  3] 16.0-17.0 sec  3.88 MBytes  32.5 Mbits/sec
[  3] 17.0-18.0 sec  2.75 MBytes  23.1 Mbits/sec
[  3] 18.0-19.0 sec  3.62 MBytes  30.4 Mbits/sec
[  3] 19.0-20.0 sec  4.12 MBytes  34.6 Mbits/sec
[  3] 20.0-21.0 sec  1.17 MBytes  9.79 Mbits/sec
[  3] 21.0-22.0 sec  2.86 MBytes  24.0 Mbits/sec
...
Here's some test from two physical servers (one CentOS and the other is the problematic Solaris server) in my network, both on the same subnet.

Code:
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   114 MBytes   957 Mbits/sec
[  3]  1.0- 2.0 sec   113 MBytes   945 Mbits/sec
[  3]  2.0- 3.0 sec   112 MBytes   938 Mbits/sec
[  3]  3.0- 4.0 sec   112 MBytes   937 Mbits/sec
[  3]  4.0- 5.0 sec   113 MBytes   947 Mbits/sec
[  3]  5.0- 6.0 sec   112 MBytes   937 Mbits/sec
[  3]  6.0- 7.0 sec   113 MBytes   945 Mbits/sec
[  3]  7.0- 8.0 sec   112 MBytes   936 Mbits/sec
[  3]  8.0- 9.0 sec   113 MBytes   945 Mbits/sec
[  3]  0.0- 9.1 sec  1.00 GBytes   943 Mbits/sec
Here's a test from the same CentOS physical to a CentOS virtual (also on the same subnet).

Something for sure is off...

Code:
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  9.62 MBytes  80.7 Mbits/sec
[  3]  1.0- 2.0 sec  9.12 MBytes  76.5 Mbits/sec
[  3]  2.0- 3.0 sec  9.00 MBytes  75.5 Mbits/sec
[  3]  3.0- 4.0 sec  9.38 MBytes  78.6 Mbits/sec
[  3]  4.0- 5.0 sec  9.38 MBytes  78.6 Mbits/sec
[  3]  5.0- 6.0 sec  9.12 MBytes  76.5 Mbits/sec
[  3]  6.0- 7.0 sec  9.12 MBytes  76.5 Mbits/sec
[  3]  7.0- 8.0 sec  9.38 MBytes  78.6 Mbits/sec
[  3]  8.0- 9.0 sec  9.12 MBytes  76.5 Mbits/sec
[  3]  9.0-10.0 sec  9.25 MBytes  77.6 Mbits/sec
[  3] 10.0-11.0 sec  9.25 MBytes  77.6 Mbits/sec
[  3] 11.0-12.0 sec  9.12 MBytes  76.5 Mbits/sec
[  3] 12.0-13.0 sec  9.25 MBytes  77.6 Mbits/sec
[  3] 13.0-14.0 sec  9.12 MBytes  76.5 Mbits/sec
Here's another test, same as the first problematic one, with the port forced to 100Mbit on the switch...

EDIT:

Here's the command I ran server (Solaris) side:
Code:
# iperf -s -i 1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1000 KByte (default)
....
Here's the command I ran client (CentOS) side:
Code:
# iperf -n 1024M -i 1 -c <IP>
------------------------------------------------------------
Client connecting to <IP>, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
...
 
Last edited:

kemic

New Member
Aug 15, 2015
14
4
3
Going to swap out the network cable before blowing away the OS. Will report back tomorrow.
 

kemic

New Member
Aug 15, 2015
14
4
3
I changed the network cable and i'm getting much more acceptable speeds:

Code:
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  86.1 MBytes   722 Mbits/sec
[  3]  1.0- 2.0 sec  84.5 MBytes   709 Mbits/sec
[  3]  2.0- 3.0 sec  86.0 MBytes   721 Mbits/sec
[  3]  3.0- 4.0 sec  86.4 MBytes   725 Mbits/sec
[  3]  4.0- 5.0 sec  85.9 MBytes   720 Mbits/sec
[  3]  5.0- 6.0 sec  85.4 MBytes   716 Mbits/sec
[  3]  6.0- 7.0 sec  87.6 MBytes   735 Mbits/sec
[  3]  7.0- 8.0 sec  86.0 MBytes   721 Mbits/sec
[  3]  8.0- 9.0 sec  89.9 MBytes   754 Mbits/sec
[  3]  9.0-10.0 sec   109 MBytes   916 Mbits/sec
[  3] 10.0-11.0 sec  86.1 MBytes   722 Mbits/sec
[  3]  0.0-11.6 sec  1.00 GBytes   741 Mbits/sec
See not quite as good as the other gigabit tests in my environment but I'll take it for now...