Paltry EIPOIB Performance

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Ethan Waldo

New Member
Dec 9, 2018
10
2
3
I have 3 Threadripper servers with MCX354A-QCBT ConnectX-3 cards connected to a Voltaire 4036 via WLGORE QSFP+ cables. I would like to do an ethernet bridge with docker swarm/kubernetes, but am seeing aweful EIPOIB performance for some strange reason. Regular IPOIB gets around 12-16Gbps without any specific tuning, but EIPOIB is performing at 100Kbps on the direct or partition interface. It will randomly get 100-800Mbps but then performance immediately drops on subsequent runs.

I'm using the latest MLNX_OFED_LINUX-4.5-1.0.1.0. LOAD_EIPOIB=yes & SET_IPOIB_CM=no are the only changes I've made to /etc/inifiband/openib.conf. /sbin/mlnx_tune seems to make little difference.

Anybody know why EIPOIB is doing barely better than an ISDN line?

partitions.conf
------------------
Default=0xffff , ipoib , rate=7, mtu=5 : ALL=full;
Partition16=0x8010 , ipoib , rate=7, mtu=5, defmember=full: ALL=full;

[xxxxxxx@moner3 ~]$ iperf -c 192.168.7.1 ; EIPOIB MTU 4092, 2044, or 1500
------------------------------------------------------------
Client connecting to 192.168.7.1, TCP port 5001
TCP window size: 165 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.7.3 port 48052 connected with 192.168.7.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.6 sec 162 KBytes 125 Kbits/sec

[xxxxxxx@moner3 ~]$ iperf -c 192.168.5.1 ; IPOIB MTU 65520
------------------------------------------------------------
Client connecting to 192.168.5.1, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.5.3 port 33826 connected with 192.168.5.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 16.0 GBytes 13.7 Gbits/sec

[xxxxxxx@moner3 ~]$ iperf -c 192.168.6.1 ; 10Gbe AQC107 MTU 1500
------------------------------------------------------------
Client connecting to 192.168.6.1, TCP port 5001
TCP window size: 604 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.6.3 port 33226 connected with 192.168.6.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9.44 GBytes 8.11 Gbits/sec
 

Takrbark3

New Member
Dec 17, 2017
27
1
3
42
Hi!

Old topic, but for future reference.
EIPOIB (Ethernet Services over IPoIB) is supported only from Connectx4-card or higher.

EIPOIB function is required, if you want to use the IB interface to look like an "Ethernet-device", but over IB link:
> Adding IB to the ethernet bridge, interconnect VirtualMachines
> Using IB as a BOND interface and/or BOND add to ethernet bridge
> VLANs with Bond and Bridge
...
 
Last edited: