Infiniband Requests

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

cactus

Moderator
Jan 25, 2011
830
75
28
CA
I have a lot of Infiniband hardware and am finally getting around to testing it. Are there any topics the forum would like me to focus on or put at the head of the queue?
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Would love to see:
1. Setup with the IB switch.
2. Integration with IP networks.
 

joisey04

Member
Jul 12, 2012
30
0
6
I'm interested to connect my VMWare Server (ESXi 5.1) with my Storage (Napp-IT) with the cheapest and fastest possible device to device connection.
Infiniband or 10Gb Ethernet??
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
After the very important "here's how to get it all working" hints, I'd like to see benchmarks that show large and small block read and write performance at high queue depth for a storage server setup - one machine with disk serving up data to one or more client machines via various IB protocols to see which provides the best performance. Protocols: SMB, SDP, iSCSI, IPoIB, and maybe NFS. That's not asking too much is it? ;)
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Sad. Currently Linux bridge, or anything that requires an Ethernet header like a vSwitche, does not work with IPoIB. There is work on an eIPoIB driver to enslave the IPoIB interface to an ethX, from what I can tell, this is being used in the Mellanox driver for VMWare. For now L3 routing is needed to get an Ethernet/IP network to talk to an IPoIB network.
 

bp_968

New Member
Dec 23, 2012
45
0
0
I could use some direct help here real quick if anyone has the answer. I have 3 IB HCAs. 1 MHEA28-XTC and 2 MHGH28-XTC. The MHEA28-XTC installs fine with the OFED driver package and was up and running right away on my Windows 7 machine, even without a reboot. The 2 MHGH28-XTC cards both refuse to function (in windows7). I've tried the OFED package, I've tried the windows drivers windows finds and installs and I've tried the drivers I found on Mellanox's website. I've tried the cards in 2 different windows 7 machines that are highly different hardware (i5 ivy bridge and a i7 950) and they report the same problem on each card and each machine:

This device cannot start. (Code 10)

When both of the MHGH28-XTC cards arrived they were packed very poorly and the brackets on both of them were bent at a 45 degree angle. I wanted to toss this out there and see if just maybe this issue was a common one with an easy fix I somehow missed, otherwise I'll be contacting the seller/ebay/paypal for a RMA/refund based on the poor packaging and bent brackets.

Thanks!
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
I could use some direct help here real quick if anyone has the answer. I have 3 IB HCAs. 1 MHEA28-XTC and 2 MHGH28-XTC. The MHEA28-XTC installs fine with the OFED driver package and was up and running right away on my Windows 7 machine, even without a reboot. The 2 MHGH28-XTC cards both refuse to function (in windows7). I've tried the OFED package, I've tried the windows drivers windows finds and installs and I've tried the drivers I found on Mellanox's website. I've tried the cards in 2 different windows 7 machines that are highly different hardware (i5 ivy bridge and a i7 950) and they report the same problem on each card and each machine:

This device cannot start. (Code 10)

When both of the MHGH28-XTC cards arrived they were packed very poorly and the brackets on both of them were bent at a 45 degree angle. I wanted to toss this out there and see if just maybe this issue was a common one with an easy fix I somehow missed, otherwise I'll be contacting the seller/ebay/paypal for a RMA/refund based on the poor packaging and bent brackets.

Thanks!
Check their firmware and if possible, use another OS to test them. I found it simplest to use the boot image at the bottom of the page here to update the firmware.
 

bp_968

New Member
Dec 23, 2012
45
0
0
Firmware fixed it!! Thanks so much, it was really starting to annoy me. I still can't believe it was firmware (never seen a card gag out in windows on a firmware version as high as 2.XX like these did).

Merry Christmas and thanks again!

Check their firmware and if possible, use another OS to test them. I found it simplest to use the boot image at the bottom of the page here to update the firmware.
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
So, I have been working on IPoIB stuff lately and have a lot of iperf data to put together including data using Dell XR997 10GbE to compare with. I decided to move to some storage related testing starting with simple iSCSI over IPoIB. Using FIO and the following job file, ConnectX(20Gbps) to ConnectX got a sequential read of ~300MB/s and ~35k iops. A run with DD to /dev/null and bs=8k gives 327MB/s. The cards are in datagram mode and single thread iperf performance in that mode is 9.84Gbps(1.23GB/s). The iSCSI export is a image file in tmpfs of an i7-930 with DDR3-1600 9-9-9-24. DD to /dev/null and bs=8k of the image file I have exported gives 8GB/s and FIO gives ~4.4GB/s with ~550k iops to tmpfs mount.

Code:
[seq-read]
rw=read
ioengine=libaio
iodepth=1
size=8g
directory=/home/max/Away
fadvise_hint=0
blocksize=8k
direct=0
numjobs=1
nrfiles=1
runtime=10m
time_based
For future testing I am thinking of using the job files here. If anyone has suggestions on cross platform file benchmark tools to use or a specific job file to run using FIO, let me know.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,529
241
63
hope you make the writeups easy to read like your great Ubuntu Infiniband guide
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Has anyone used a ConnectX-2/3 VPI IB card as an Ethernet adapter with a QSFP to SFP+ cable?

Can this actually replace say, an Intel X520-DA2?
In theory, but no experience besides CX-2 to CX-2. CX-2 with EN can do mtu of 9600 vs 16K for my Intel AT cards. You have to use connectx_port_config to switch one or both ports. You can find supported port configs here. Also, check if the switch can deal with Ethernet.
 

wuffers

New Member
Dec 24, 2012
19
0
0
In theory, but no experience besides CX-2 to CX-2. CX-2 with EN can do mtu of 9600 vs 16K for my Intel AT cards. You have to use connectx_port_config to switch one or both ports. You can find supported port configs here. Also, check if the switch can deal with Ethernet.
Yeah, I saw the port configs on page 17 of the ConnectX-2 user manual. On page 38 they also show the Windows driver setting to choose the port protocol, but just prior to that the screenshot shows Device Manager with the card as an IPoIB adapter. Does this really function as Ethernet on a physical layer or does it do the encapsulation and is therefore working in IPoIB (and thus not supporting bridging, MAC spoofing, and whatever else)?

I need 10GbE networking and planning to connect via an uplink switch with SFP+ ports, so it won't be adapter to adapter.
 
Last edited:

wuffers

New Member
Dec 24, 2012
19
0
0
So I have an official answer from a Mellanox engineer that VPI cards do operate as pure Ethernet without any special firmware ("plug and play" with carrier detect or forcing them as previously mentioned), and there's no encapsulation going on. I've ordered some QSA (QSFP to SFP+ adapters) to test that out so I can use SFP+ cabling throughout, but in the end I ordered some Intel X520-DA2s for the project at hand for 10GbE. Too much uncertainty without real world tests.

What kind of latency should I expect from the ConnectX-2 cards when connected via an IB switch (my Voltaire 4036s came, man those things are LOUD)? I just started playing with them now that the cabling came in and upgrading firmware on each HBA and switch. ibping is giving me ~0.200ms average. Is that about right? Does latency improve with better hardware? I've got them now connected to some old hardware that had an extra PCIe slot or didn't need that slot for graphics.

I had OI installed but scrapped that for Ubuntu since it didn't have the OFED IB diagnostic tools (I'm just doing this to validate the gear). ibping, iblinkinfo, ibstat all work but I couldn't get any of the testing tools to work like ib_write_lat, ib_send_lat, etc. I haven't tried compiling OFED yet, that might make a difference. The other PC with Windows seems to work with all the ib* commands.
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Nice to hear an official "Yes, they work as standard Ethernet".

Ill boot some stuff up tonight to get ping tests. 0.2ms seems right for the first ping when using IPoIB, but it normally drops for all the rest. I have been doing P2P, so maybe the switch is adding to your latency.

I have a lot of performance data to get into an article, but I noticed CX-2 are slower in EN than my Intel AT 10GbE gear.
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
ConnectX-2 Point-to-Point with opensm running on host TT

Using ibping:
Code:
max@TT:~$ sudo ibping -S
Code:
max@TC2:~$ sudo ibping -G 0x0002c903004a9ba3
Pong from TT.(none) (Lid 10): time 0.304 ms
Pong from TT.(none) (Lid 10): time 0.253 ms
Pong from TT.(none) (Lid 10): time 0.253 ms
Pong from TT.(none) (Lid 10): time 0.251 ms
Pong from TT.(none) (Lid 10): time 0.254 ms
Pong from TT.(none) (Lid 10): time 0.251 ms
Pong from TT.(none) (Lid 10): time 0.256 ms
Pong from TT.(none) (Lid 10): time 0.253 ms
Pong from TT.(none) (Lid 10): time 0.250 ms
Pong from TT.(none) (Lid 10): time 0.255 ms
First run of ping after reboot using IPoIB:
Code:
max@TC2:~$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_req=1 ttl=64 time=0.880 ms
64 bytes from 10.0.0.1: icmp_req=2 ttl=64 time=0.114 ms
64 bytes from 10.0.0.1: icmp_req=3 ttl=64 time=0.112 ms
64 bytes from 10.0.0.1: icmp_req=4 ttl=64 time=0.109 ms
64 bytes from 10.0.0.1: icmp_req=5 ttl=64 time=0.110 ms
64 bytes from 10.0.0.1: icmp_req=6 ttl=64 time=0.109 ms
64 bytes from 10.0.0.1: icmp_req=7 ttl=64 time=0.106 ms
64 bytes from 10.0.0.1: icmp_req=8 ttl=64 time=0.106 ms
64 bytes from 10.0.0.1: icmp_req=9 ttl=64 time=0.110 ms
^C
--- 10.0.0.1 ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 7998ms
rtt min/avg/max/mdev = 0.106/0.195/0.880/0.242 ms
 
Last edited:

wuffers

New Member
Dec 24, 2012
19
0
0
Thanks for those results. What kind of hardware are you running on?

I have an Athlon 64 X2 4600 (2.4Ghz) on one side, and an Intel E8400 on another. I'm actually seeing different ibping results.

Intel E8400 as ibping server, I'm getting ~0.200ms. With the Athlon CPU as the ibping server, I'm getting around ~0.120ms. With IPoIB, normal ping is around ~0.500ms average (using hrping on Windows). I'm not getting much love with the default ib_send/write_bw throughput tests though, ~1.4Gbps with QDR. With iperf I was only getting 1.2Gbps, as it shows ~3.5% on the Networking tab for Task Manager (and this was using Connected Mode).

Seems like the switch is only adding ~.020-0.030 ms latency from what I can tell.

I guess I should take these results with a grain of salt, until the real server hardware shows up. All my HBAs and switches came out okay, only one failed PSU but no dead ports.