Had to resort to two phys Linux systems but getting somewhere...I could not see link lights on the cards/IS5022 switch so I direct connected them restarted opensm and BAM lights up. Cannot figure out why the IB switch wont play nice so far. Do I need a OpenSM running on each Linux instance for them to work while connected to the IS5022 Mellanox IB switch?
Crossover/direct connect iperf w/ 65520 MTU set/connected mode/default CentOS7 load. May try w/ OFED drivers loaded next.
EDIT: HA just noticed that w/ my L5630 system driving the server it's maxing out on CPU at 100% while my E5 2670 system in client mode pushes 25% CPU. Let's flip-flop and see what it does....yep a bit better w/ E5 2670 system driving iperf server (40% CPU) and L5630 system as client (still 100%) but better throughput.
[ 4] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 45410
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 17.8 GBytes 15.3 Gbits/sec
[ 5] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 45412
[ 5] 0.0-10.0 sec 17.9 GBytes 15.3 Gbits/sec
[ 4] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 45460
[ 4] 0.0-10.0 sec 17.8 GBytes 15.3 Gbits/sec
So now I am CPU constrained to really see these QDR HCA's fly right since it looks like iperf is single threaded?