infiniband ethernet on vsphere and etc...

Discussion in 'Networking' started by dswartz, May 15, 2015.

  1. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    376
    Likes Received:
    28
    I had been hacking around on a vsphere 5.5 host connected to an ESOS (storage distro running off usb flash drive). Did lots of fiddling around with SRP, trying to figure out how to get iSER working (no joy at this point). The storage server (supermicro x9scl-f) didn't seem to like the used connnectx-2 card I got on ebay (I would reboot, and the IB stuff would have just disappeared, despite the card being visible via lspci.) So I shelled out a little extra for a new connectx-3 card for the storage server. I discovered that if I reverted to the 1.9 (ethernet only) vsphere IB driver, the connectx-3 would connect to the connectx-2 as a 10gbe enet link (direct connect with QSFP+/QSFP+ cable.) I decided to replace the single port connectx-2 in the vsphere host with a dual-port card - since the same guy I got the other connectx-3 card from had more in stock, I bought another and swapped it in. To my surprise, vsphere and linux now report a 40gbe ethernet link! Good times :) I'd like to get iSER working to reduce CPU load on both sides, but SCST doesn't support that yet (that is the iSCSI target SW ESOS uses.) They do have an SCST branch in development supporting iSER, so I think I'll just wait it out until then. The other vsphere host only has 1gbe to the storage server, but it's a backup box anyway. I did throw the replaced connectx-2 card in that box, so I can at least do vmotion at 10gbe :)
     
    #1
  2. Entz

    Entz Active Member

    Joined:
    Apr 25, 2013
    Messages:
    269
    Likes Received:
    62
    Are there any advantages to using iSER over GBe? Wouldn't that would require going through RoCE as well?

    It is really too bad that the ESXi drivers are not VPI aware. Would be nice to run one port as native 10/40GBe (not IPoIB) and the other as Infiniband (iSER/SRP). Right now it is either or (unless I am missing something)
     
    #2
  3. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    376
    Likes Received:
    28
    I'm only speculating here, but what I have read is that not shoving all the data through TCP/IP reduces the CPU load. How much, probably depends. I seem to recall reading somewhere that you can in fact run the two ports in different modes, although I seem to recall there being an assymetry (e.g. if port 1 is IB, port 2 can be enet, but not the other way around. Or maybe I have that backwards lol)
     
    #3
  4. tjk

    tjk Active Member

    Joined:
    Mar 3, 2013
    Messages:
    236
    Likes Received:
    32
    What switch are you using to connect these to?
     
    #4
  5. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    376
    Likes Received:
    28
    No switch. Back to back with 1M cable.
     
    #5
  6. Baddreams

    Baddreams New Member

    Joined:
    Mar 15, 2011
    Messages:
    21
    Likes Received:
    7
    Yeah, I tried like hell to get iSER working with ConnectX-2 cards in a variety of setups, in the end the only way it would work was between two Linux machines, one running the Linux-IO iSCSI target. But, man oh man was it fast.
     
    #6
  7. ehorn

    ehorn Active Member

    Joined:
    Jun 21, 2012
    Messages:
    340
    Likes Received:
    51
    Nice to see the connectx-3's coming down in price quickly. They offer much wider support.
     
    #7
Similar Threads: infiniband ethernet
Forum Title Date
Networking QSFP Cables - same for Ethernet and Infiniband? Dec 22, 2017
Networking 100 Gigabit Ethernet and Infiniband from Mellanox - ConnectX4 arrives Nov 14, 2014
Networking infiniband 10G ethernet confusion (it just works???) Nov 3, 2013
Networking Infiniband to 10g Ethernet Cisco Apr 23, 2013
Networking Infiniband to Ethernet bridging question Feb 22, 2013

Share This Page