infiniband ethernet on vsphere and etc...

dswartz

Active Member
Jul 14, 2011
389
33
28
I had been hacking around on a vsphere 5.5 host connected to an ESOS (storage distro running off usb flash drive). Did lots of fiddling around with SRP, trying to figure out how to get iSER working (no joy at this point). The storage server (supermicro x9scl-f) didn't seem to like the used connnectx-2 card I got on ebay (I would reboot, and the IB stuff would have just disappeared, despite the card being visible via lspci.) So I shelled out a little extra for a new connectx-3 card for the storage server. I discovered that if I reverted to the 1.9 (ethernet only) vsphere IB driver, the connectx-3 would connect to the connectx-2 as a 10gbe enet link (direct connect with QSFP+/QSFP+ cable.) I decided to replace the single port connectx-2 in the vsphere host with a dual-port card - since the same guy I got the other connectx-3 card from had more in stock, I bought another and swapped it in. To my surprise, vsphere and linux now report a 40gbe ethernet link! Good times :) I'd like to get iSER working to reduce CPU load on both sides, but SCST doesn't support that yet (that is the iSCSI target SW ESOS uses.) They do have an SCST branch in development supporting iSER, so I think I'll just wait it out until then. The other vsphere host only has 1gbe to the storage server, but it's a backup box anyway. I did throw the replaced connectx-2 card in that box, so I can at least do vmotion at 10gbe :)
 

Entz

Active Member
Apr 25, 2013
269
62
28
Canada Eh?
Are there any advantages to using iSER over GBe? Wouldn't that would require going through RoCE as well?

It is really too bad that the ESXi drivers are not VPI aware. Would be nice to run one port as native 10/40GBe (not IPoIB) and the other as Infiniband (iSER/SRP). Right now it is either or (unless I am missing something)
 

dswartz

Active Member
Jul 14, 2011
389
33
28
I'm only speculating here, but what I have read is that not shoving all the data through TCP/IP reduces the CPU load. How much, probably depends. I seem to recall reading somewhere that you can in fact run the two ports in different modes, although I seem to recall there being an assymetry (e.g. if port 1 is IB, port 2 can be enet, but not the other way around. Or maybe I have that backwards lol)
 

Baddreams

New Member
Mar 15, 2011
21
7
3
I had been hacking around on a vsphere 5.5 host connected to an ESOS (storage distro running off usb flash drive). Did lots of fiddling around with SRP, trying to figure out how to get iSER working (no joy at this point). The storage server (supermicro x9scl-f) didn't seem to like the used connnectx-2 card I got on ebay (I would reboot, and the IB stuff would have just disappeared, despite the card being visible via lspci.) So I shelled out a little extra for a new connectx-3 card for the storage server. I discovered that if I reverted to the 1.9 (ethernet only) vsphere IB driver, the connectx-3 would connect to the connectx-2 as a 10gbe enet link (direct connect with QSFP+/QSFP+ cable.) I decided to replace the single port connectx-2 in the vsphere host with a dual-port card - since the same guy I got the other connectx-3 card from had more in stock, I bought another and swapped it in. To my surprise, vsphere and linux now report a 40gbe ethernet link! Good times :) I'd like to get iSER working to reduce CPU load on both sides, but SCST doesn't support that yet (that is the iSCSI target SW ESOS uses.) They do have an SCST branch in development supporting iSER, so I think I'll just wait it out until then. The other vsphere host only has 1gbe to the storage server, but it's a backup box anyway. I did throw the replaced connectx-2 card in that box, so I can at least do vmotion at 10gbe :)
Yeah, I tried like hell to get iSER working with ConnectX-2 cards in a variety of setups, in the end the only way it would work was between two Linux machines, one running the Linux-IO iSCSI target. But, man oh man was it fast.
 

ehorn

Active Member
Jun 21, 2012
342
52
28
Nice to see the connectx-3's coming down in price quickly. They offer much wider support.