Hey folks, long time lurker, first time poster!
I have a test lab where I am testing the performance of Infiniband, specifically IPoIB, and think I am running into some performance problems, looking for any feedback anyone may have.
Setup:
10 x Dell R610 nodes, 48GB Ram, dual quad core 5600 cpu's, 6x600GB 10K SAS hdd's
10 x MHQH19B-XTR single port QDR Infiniband cards in each server (Firmware version: 2.9.1200)
1 x Mellanox 4036 switch with latest firmware that I can find (version: 3.9.1 / build Id:985), subnet manager running on the switch
Cards report up and running at 4xQDR:
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:0007:f2e7
base lid: 0x3
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand
Problem is the IP performance over the Infiniband fabric is not that great, here are some IPerf test results. The results are running the cards in connected mode, with 65520 MTU. Datagram mode was worse. Kernel tweaking done via the Mellanox docs didn't make things any faster or slower basically performance was the same either way.
(Latest Centos 6.4 64 bit on each host node)
iperf -fg -c 10.10.10.11
------------------------------------------------------------
Client connecting to 10.10.10.11, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.12 port 42552 connected with 10.10.10.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 12.7 GBytes 10.9 Gbits/sec
[root@vz3 ~]# iperf -fg -c 10.10.10.11
------------------------------------------------------------
Client connecting to 10.10.10.11, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.12 port 42553 connected with 10.10.10.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 12.7 GBytes 10.9 Gbits/sec
(Not very scientific but a test nonetheless)
SCP large file directory over IBoIP network:
file7 100% 4152MB 148.3MB/s 00:28
file10 100% 4152MB 148.3MB/s 00:28
file9 100% 4152MB 153.8MB/s 00:27
(Same test / same files over a 2xGE bonded interface, which is FASTER then over the IB network)
file7 100% 4152MB 138.4MB/s 00:30
file10 100% 4152MB 166.1MB/s 00:25
file9 100% 4152MB 159.7MB/s 00:26
file6 100% 4152MB 166.1MB/s 00:25
In summary: IPoIB on a QDR40Gb/sec fabric seems to be performing horrible. I know IPoIB isn't the most efficient way to do this (would be doing SRP if I could, but can't), but I would expect it to perform better then 30% of line rate.
Am I missing anything? I can run any test if someone has suggestions, or test another OS/method/etc.
Thanks in advance for any suggestions!
Tom
I have a test lab where I am testing the performance of Infiniband, specifically IPoIB, and think I am running into some performance problems, looking for any feedback anyone may have.
Setup:
10 x Dell R610 nodes, 48GB Ram, dual quad core 5600 cpu's, 6x600GB 10K SAS hdd's
10 x MHQH19B-XTR single port QDR Infiniband cards in each server (Firmware version: 2.9.1200)
1 x Mellanox 4036 switch with latest firmware that I can find (version: 3.9.1 / build Id:985), subnet manager running on the switch
Cards report up and running at 4xQDR:
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:0007:f2e7
base lid: 0x3
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand
Problem is the IP performance over the Infiniband fabric is not that great, here are some IPerf test results. The results are running the cards in connected mode, with 65520 MTU. Datagram mode was worse. Kernel tweaking done via the Mellanox docs didn't make things any faster or slower basically performance was the same either way.
(Latest Centos 6.4 64 bit on each host node)
iperf -fg -c 10.10.10.11
------------------------------------------------------------
Client connecting to 10.10.10.11, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.12 port 42552 connected with 10.10.10.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 12.7 GBytes 10.9 Gbits/sec
[root@vz3 ~]# iperf -fg -c 10.10.10.11
------------------------------------------------------------
Client connecting to 10.10.10.11, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.12 port 42553 connected with 10.10.10.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 12.7 GBytes 10.9 Gbits/sec
(Not very scientific but a test nonetheless)
SCP large file directory over IBoIP network:
file7 100% 4152MB 148.3MB/s 00:28
file10 100% 4152MB 148.3MB/s 00:28
file9 100% 4152MB 153.8MB/s 00:27
(Same test / same files over a 2xGE bonded interface, which is FASTER then over the IB network)
file7 100% 4152MB 138.4MB/s 00:30
file10 100% 4152MB 166.1MB/s 00:25
file9 100% 4152MB 159.7MB/s 00:26
file6 100% 4152MB 166.1MB/s 00:25
In summary: IPoIB on a QDR40Gb/sec fabric seems to be performing horrible. I know IPoIB isn't the most efficient way to do this (would be doing SRP if I could, but can't), but I would expect it to perform better then 30% of line rate.
Am I missing anything? I can run any test if someone has suggestions, or test another OS/method/etc.
Thanks in advance for any suggestions!
Tom