10Gb SFP+ single port = cheaper than dirt

Discussion in 'Great Deals' started by MiniKnight, Sep 17, 2015.

  1. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,330
    Likes Received:
    205
    Thanks I ended up getting this for 36. Hoping since they are dell branded i wont have an issue with my dell powerconnect 5524 switch and dell server. The cables (you get 2 btw) are cisco 3 meter cables. For 36 i figured i can play with 10GB. :)
     
    #161
  2. compuwizz

    compuwizz New Member

    Joined:
    Feb 25, 2017
    Messages:
    24
    Likes Received:
    5
    Are these multiqueue NICS? Just wondering how they present their interrupts to the processor.
     
    #162
  3. sgilb

    sgilb New Member

    Joined:
    Jun 5, 2017
    Messages:
    2
    Likes Received:
    0
    Looking for drivers for these for Windows 8.1. If anyone can point me to one would be most helpful. Have it them working right now with a driver from a 10Gtec version but trying to check for others. also anyone have any firmware update info?
    thanks
     
    #163
  4. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,657
    Likes Received:
    399
    Drivers are available on the mellanox site, download the winof package.
    The latest firmware is available in a package from HP, I can post the link later.
     
    #164
  5. sgilb

    sgilb New Member

    Joined:
    Jun 5, 2017
    Messages:
    2
    Likes Received:
    0
    "Drivers are available on the mellanox site, download the winof package.
    The latest firmware is available in a package from HP, I can post the link later."

    Thanks - if you could send a link it would be great.
     
    #165
  6. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,657
    Likes Received:
    399
    #166
  7. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,463
    Likes Received:
    500
    Slightly OT: Whats the current 10gbe/sfp+ recommendation for ESX 6.5? The Connectx-2 cards wont work unfortunately and -3 are still relatively expensive...
     
    #167
  8. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,657
    Likes Received:
    399
    According to the vmware hcl connect-x2 (and 1) are supported in esxi 5 - 6.5
     
    #168
  9. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,463
    Likes Received:
    500
    Really? I'll check that. I thought I had connected one once and it was not detected by the new driver model. And new and old driver can't coexist iirc. But I might be able to remove new model and use only old one ... hmm.
    Will have to try that, thanks :)

    Edit:
    Indeed it works :)

    Quick steps just for documentation shamelessly taken from @inbusiness :

    Run this (from Which ESXi driver to use for SRP/iSER over IB (... | Mellanox Interconnect Community)

    * disable native driver for vRDMA - this is very buggy

    esxcli system module set --enabled=false -m=nrdma

    esxcli system module set --enabled=false -m=nrdma_vmkapi_shim

    esxcli system module set --enabled=false -m=nmlx4_rdma

    esxcli system module set --enabled=false -m=vmkapi_v2_3_0_0_rdma_shim

    esxcli system module set --enabled=false -m=vrdma



    * uninstall inbox driver - also useless function that can't support ethernet iSER properly

    esxcli software vib remove -n net-mlx4-en

    esxcli software vib remove -n net-mlx4-core

    esxcli software vib remove -n nmlx4-rdma

    esxcli software vib remove -n nmlx4-en

    esxcli software vib remove -n nmlx4-core

    esxcli software vib remove -n nmlx5-core

    Reboot,
    Get package as described here:
    ConnectX-2 and ESXi 6.0
    http://www.mellanox.com/downloads/Software/MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585.zip

    * install Mellanox OFED 1.8.2.5 for ESXi 6.x.

    esxcli software vib install -d /var/log/vmware/MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585.zip

    reboot
    get this
    upload_2017-6-6_20-58-54.png


    O/c will be a pain to update since this will not work due to conflicts, but thats the price I gues.
     
    #169
    Last edited: Jun 6, 2017
    _alex likes this.
  10. Ellwood

    Ellwood Member

    Joined:
    Nov 20, 2016
    Messages:
    33
    Likes Received:
    11
    Yeah, if anyone knows an easy way to rename that.... let me know as I have the same issue. Supposedly host profiles, but I only have one esxi host, so I can't put it in maintenance mode and update with vcenter host profile (that I'm aware of)
     
    #170
  11. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    742
    Likes Received:
    227
    Picked up a couple of these after seeing this post.
    Installed one in my main rig... ASUS Z10PE-D16 - Windows 10 Pro... it just worked... 10Gbe, Woot :)
    I have the other in a 4-Bay NAS which I have to get setup to test... <crosses fingers> it works.
    I am going to grab a couple more to have on hand.

    Thanks for posting this !
     
    #171
  12. Jannis Jacobsen

    Jannis Jacobsen Active Member

    Joined:
    Mar 19, 2016
    Messages:
    304
    Likes Received:
    49
    Is this adapter working properly in Windows Server 2016?
    I need rdma (roce) support for a storage spaces direct 2-node hyperv cluster.

    -jannis
     
    #172
  13. rune-san

    rune-san Member

    Joined:
    Feb 7, 2014
    Messages:
    78
    Likes Received:
    15
    The Windows Server 2016 driver only works with ConnectX-3 / ConnectX-3 Pro cards for WinOF, and Connect4 and up for WinOF-2.
     
    #173
  14. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,657
    Likes Received:
    399
    Server 2016 supports them out of the box o_O
    If you want to have more configuration options install the winof package, it works without problems with the connect-x2 cards.
     
    #174
    Jannis Jacobsen likes this.
  15. rune-san

    rune-san Member

    Joined:
    Feb 7, 2014
    Messages:
    78
    Likes Received:
    15
    As a basic card I figured that worked, but what about RDMA? I remember specifically a Microsoft guy saying that ConnectX-2 could be buggy at times for SMB Direct because it does not properly support Priority Flow Control, one of the two necessary components for RoCE to work. I was told that's why the ConnectX-2 cards would often drop out of RDMA and back to standard TCP. That was some years ago though. If it's been rectified to the point it works without dropping out (save for maybe extreme link load) that would be awesome :)
     
    #175
  16. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,657
    Likes Received:
    399
    Rdma was added in a firmware version that was only available as a mlx file on the mellanox website or in a package from hp, it was never shipped as a binary.
    Can't say if it really works as I only have one pc with windows 10 (windows client os doesn't support rdma!) + connectx2 and one server with a connectx3 card.
     
    #176
    Jannis Jacobsen likes this.
  17. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,657
    Likes Received:
    399
    Not sure if the post from @jimmyhocky is legit or spam for that link.
     
    #177
  18. CobaltFire

    CobaltFire New Member

    Joined:
    Nov 7, 2015
    Messages:
    18
    Likes Received:
    0
    Can confim, running a Connectx-2 on a Windows Server 2016 machine. Worked straight out of the box.
     
    #178
  19. rune-san

    rune-san Member

    Joined:
    Feb 7, 2014
    Messages:
    78
    Likes Received:
    15
    Indeed, as noted, the Inbox driver works fine for base functionality. What the OP still needs confirmation of, and I'd be curious about of, is if RoCE works on the adapters. Without that, the OP won't be able to use these adapters for what he wants effectively (Storage Spaces Direct). According to i386, there appears to have been *some* RDMA support released for one vendor at one time through non-official means. I would be curious to know if anyone is running that particular version and can confirm Priority Flow Control is properly functioning (in other words, that it's not falling out of RDMA under load). Otherwise, while the card will certainly work in 2016, it won't do what the OP wants to do with it. It will just be a basic TCP/IP Ethernet card.
     
    #179
  20. Jannis Jacobsen

    Jannis Jacobsen Active Member

    Joined:
    Mar 19, 2016
    Messages:
    304
    Likes Received:
    49
    I just flashed 2 hp branded and 2 mellanox branded connectx-2 with firmware supporting rdma, in server 2016

    -jannis
     
    #180

Share This Page