I have 3 Threadripper servers with MCX354A-QCBT ConnectX-3 cards connected to a Voltaire 4036 via WLGORE QSFP+ cables. I would like to do an ethernet bridge with docker swarm/kubernetes, but am seeing aweful EIPOIB performance for some strange reason. Regular IPOIB gets around 12-16Gbps without any specific tuning, but EIPOIB is performing at 100Kbps on the direct or partition interface. It will randomly get 100-800Mbps but then performance immediately drops on subsequent runs.
I'm using the latest MLNX_OFED_LINUX-4.5-1.0.1.0. LOAD_EIPOIB=yes & SET_IPOIB_CM=no are the only changes I've made to /etc/inifiband/openib.conf. /sbin/mlnx_tune seems to make little difference.
Anybody know why EIPOIB is doing barely better than an ISDN line?
partitions.conf
------------------
Default=0xffff , ipoib , rate=7, mtu=5 : ALL=full;
Partition16=0x8010 , ipoib , rate=7, mtu=5, defmember=full: ALL=full;
[xxxxxxx@moner3 ~]$ iperf -c 192.168.7.1 ; EIPOIB MTU 4092, 2044, or 1500
------------------------------------------------------------
Client connecting to 192.168.7.1, TCP port 5001
TCP window size: 165 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.7.3 port 48052 connected with 192.168.7.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.6 sec 162 KBytes 125 Kbits/sec
[xxxxxxx@moner3 ~]$ iperf -c 192.168.5.1 ; IPOIB MTU 65520
------------------------------------------------------------
Client connecting to 192.168.5.1, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.5.3 port 33826 connected with 192.168.5.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 16.0 GBytes 13.7 Gbits/sec
[xxxxxxx@moner3 ~]$ iperf -c 192.168.6.1 ; 10Gbe AQC107 MTU 1500
------------------------------------------------------------
Client connecting to 192.168.6.1, TCP port 5001
TCP window size: 604 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.6.3 port 33226 connected with 192.168.6.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9.44 GBytes 8.11 Gbits/sec
I'm using the latest MLNX_OFED_LINUX-4.5-1.0.1.0. LOAD_EIPOIB=yes & SET_IPOIB_CM=no are the only changes I've made to /etc/inifiband/openib.conf. /sbin/mlnx_tune seems to make little difference.
Anybody know why EIPOIB is doing barely better than an ISDN line?
partitions.conf
------------------
Default=0xffff , ipoib , rate=7, mtu=5 : ALL=full;
Partition16=0x8010 , ipoib , rate=7, mtu=5, defmember=full: ALL=full;
[xxxxxxx@moner3 ~]$ iperf -c 192.168.7.1 ; EIPOIB MTU 4092, 2044, or 1500
------------------------------------------------------------
Client connecting to 192.168.7.1, TCP port 5001
TCP window size: 165 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.7.3 port 48052 connected with 192.168.7.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.6 sec 162 KBytes 125 Kbits/sec
[xxxxxxx@moner3 ~]$ iperf -c 192.168.5.1 ; IPOIB MTU 65520
------------------------------------------------------------
Client connecting to 192.168.5.1, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.5.3 port 33826 connected with 192.168.5.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 16.0 GBytes 13.7 Gbits/sec
[xxxxxxx@moner3 ~]$ iperf -c 192.168.6.1 ; 10Gbe AQC107 MTU 1500
------------------------------------------------------------
Client connecting to 192.168.6.1, TCP port 5001
TCP window size: 604 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.6.3 port 33226 connected with 192.168.6.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9.44 GBytes 8.11 Gbits/sec