ConnectX-3 vs ConnectX-3 Pro?

Discussion in 'Networking' started by frogtech, Aug 4, 2018.

  1. frogtech

    frogtech Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    1,048
    Likes Received:
    100
    What's the difference between these two feature-wise (EN models only)? It seems like only the ConnectX-3 Pro does RDMA over converged ethernet (RoCE) whereas standard ConnectX-3 does standard RDMA? I'm not sure there's even a difference between just RDMA or RoCE.

    Thanks.
     
    #1
  2. markpower28

    markpower28 Active Member

    Joined:
    Apr 9, 2013
    Messages:
    384
    Likes Received:
    96
  3. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    #3
  4. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    11
    #4
  5. frogtech

    frogtech Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    1,048
    Likes Received:
    100
    While we're still on the topic...I'm getting a little confused by the whole RDMA, RoCE v1, v2, iWARP thing. It seems like some of these run on different protocols. In common scenarios you would want RDMA for a Storage Spaces Direct or other hyper-converged environments. But how do you know which protocol to go with and do you really need data center bridging on a switch for it to work? If you do routing at a switch with static routes do you need RoCE v2 which has routable RoCE packets? Does VMWare VSAN benefit from RDMA?
     
    #5
  6. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,450
    Likes Received:
    331
    I think the CX-3 Pro cards have a slightly newer asic to support roce v2.
    On their switches they have an ethernet switch that's basically a sx6036 that can only do roce v1, a newer revision with a new asic can do roce v2!
    RDMA means Remote Direct Memory Access > reading and writing directly to another hosts memory without involving cpu or the tcp/ip stack.

    RoCE = RDMA over Converged Ethernet, it's mellanox implementation of RDMA for ethernet.
    There are currently two versions of that protocol, v1 and v2. v1 is using ethernet frames for RoCE and is limited to the "local" network (not routable), v2 is using ip packets for RoCE making it a routable protocoll.
    RoCE v1 is supported by CX-2 and CX-3 cards.
    RoCE v2 is supported by CX-3 Pro and newer CX cards (they are backward compatible with roce v1)

    iWARP is the standard protocol for RDMA over ethernet networks, but besides Chelsio nobody uses it (not even Intel who helped establishing that standard!)
     
    #6
    Tha_14, abq and frogtech like this.
  7. frogtech

    frogtech Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    1,048
    Likes Received:
    100
    So would local mean devices/subnets that are directly connected on the same switch?

    If RoCE is mellanox's implementation, was there ever "just RDMA", or was iWARP the first implementation? I guess to me it sounds like RDMA is a standard, and these consortiums implement their own value adds or product features based on them.
     
    #7
  8. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    RDMA functions over Infiniband networks. RoCE and iWARP are the implementations that allow RDMA to occur over Ethernet.
     
    #8
    frogtech likes this.
  9. frogtech

    frogtech Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    1,048
    Likes Received:
    100
    Ooh yeah, it's coming back to me now...
     
    #9
  10. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,450
    Likes Received:
    331
    iWARP is the official protocol, only chelsio using it :D
     
    #10

Share This Page