ConnectX-3 Pro MCX354A-FCCT FDR10 40/56GbE 2 PORT QSFP, $109 shipped, 8 available

[arglebargle]

HP 764736-001 QSFP 10GB 40GB 2 Port HBA 764284-B21 764616-001 High Profile | eBay

These seem to hover around $130-150 most of the time, I thought people wanting RoCE v2 hardware might be interested.

These are a newer generation CX-3 card with extra features, they're not the $30-35 CX-3 dual 40Gb cards that we all know and love.

 

[BackupProphet]

Isn't that the same card as this for just 35 USD? HP 649281-B21 Infiniband FDR/EN 10/40GB 2P 544QSFP ADAPTER 656089-001 661685-001 | eBay

 

[arglebargle]

Isn't that the same card as this for just 35 USD? HP 649281-B21 Infiniband FDR/EN 10/40GB 2P 544QSFP ADAPTER 656089-001 661685-001 | eBay

No, the link in the OP is for a CX-3 Pro, a later gen card with a few new features and current support from Mellanox.

The Pro will do RoCE v2 (RDMA over Ethernet without PFC+ECN) -- the standard CX-3 requires very expensive switches to implement a lossless network and support RDMA (v1) as advertised.

The CX-3 is a ludicrously good deal that's hard to beat, I only posted this for people who need the feature set of the Pro for specific use cases.

 

[kiteboarder]

Yeah, unless you need RDMA/RoCE over L3, just save some cash and make a best offer of 30 each.

I bought 4 of those HP branded cards and they're working great. Latest firmware can be found on HP's website.

 

[wardtj]

There was a deal back in January for the same cx-3 pros for 20$ each. I'd hold out or Bo it for max of 40$. These come on fire sale often enough.

I'm now looking for 100gbe cx4. Hovering around 350 for dual cards.. 40gbe rdma was a great upgrade, now just want to push it...

 

[arglebargle]

There was a deal back in January for the same cx-3 pros for 20$ each. I'd hold out or Bo it for max of 40$. These come on fire sale often enough.

I'm now looking for 100gbe cx4. Hovering around 350 for dual cards.. 40gbe rdma was a great upgrade, now just want to push it...

I don't think that was for Pros, I'm pretty sure that was for standard CX-3s.

 

[awedio]

There was a deal back in January for the same cx-3 pros for 20$ each. I'd hold out or Bo it for max of 40$. These come on fire sale often enough.

I'm now looking for 100gbe cx4. Hovering around 350 for dual cards.. 40gbe rdma was a great upgrade, now just want to push it...

Where are you finding 'em for $350?

 

[wardtj]

Where are you finding 'em for $350?

Mellanox ConnectX-4 CX456A PCIe x16 3.0 100GBe MCX456A-ECAT | eBay

Not that I like Chinese resellers, or ones from Israel, but they are out there...

 

[Cutha]

The linked card is a CX-3 Pro with hardware acceleration for RoCE v2 (routable RDMA over Ethernet.) The standard CX-3 only supports RoCE v1 "locally" (without routing) and without hardware acceleration* (someone fact check this last bit here, the docs are confusing on not enough sleep.)

RDMA over Converged Ethernet - Wikipedia
http://www.mellanox.com/page/products_dyn?product_family=79

As I understand it you're buying the ability to run RDMA over more complicated Ethernet topologies with better hardware acceleration on the CX-3 Pro.

I thought I had this figured out but I think I am missing something critical. As I understand it, there are three implementations of RDMA. InfiniBand, iWarp and RoCE. InfiniBand is lossless so it does not require PFC and DCB but for the life of me I can't find any docs on how to keep it pure IB. If I had 3 windows servers with ConnectX-2 or ConnectX-3 cards and an IB switch why would I want RoCE?

RoCE_versus_InfiniBand

I was just about to buy some CX-3 Pro cards for the hardware acceleration but now I don't understand why I would need it. My servers and workstations are connected via a traditional 1gig Ethernet. The servers have ConnectX-2 cards (MHQH29B-XTR, flashed to a firmware that supports RDMA, 2.10.720) and there is a Mellanox switch (4036) connecting them and a few of the workstations. The InfiniBand NIC's are on a different subnet then the Cat6 network and there is no expectations that the InfiniBand network will need to talk to the Cat6.

To get SR-IOV I need ConnectX-3 cards, but do I need the Pro version? Would the Pro version provide any speed benefits. Hopefully my questions aren't so stupid that it leaves people scratching their heads trying to figure out what I am asking.

 

[PnoT]

You're running pure IB right now with that setup you have and the main reason to switch to RoCE is the ability to route over traditional Ethernet. I'm also pretty sure ConnectX-3 cards support SR-IOV but do you have a use case for that? Have you done speeds tests to see what you're getting right now over IB? If you do testing make sure to create RAM drives on the source and destination to get a good guage of speeds unless you have storage on both devices that's up to the task.

 

[Cutha]

Thanks for the reply. Does the extra hardware acceleration of the CX3 Pro cards assist anything other than RoCEv2?

For Hyper-V I wanted SR-IOV cards. When I setup this all up I didn't realize that I couldn't just get RDMA on the virtual machines. Not having to deal with PFC and DCB makes it much less complex. When I first set this up 2 years ago I benchmarked everything as best as I could. I used a RAM drive, here are the results (I am glad I saved them all):

Command Line: diskspd.exe -c500M -d10 -r -w0 -t4 -o8 -b512K -Sh -L \\192.168.10.1\d$\testfile.dat

Input parameters:

    timespan:   1
    -------------
    duration: 10s
    warm up time: 5s
    cool down time: 0s
    measuring latency
    random seed: 0
    path: '\\192.168.10.1\d$\testfile.dat'
        think time: 0ms
        burst size: 0
        software cache disabled
        hardware write cache disabled, writethrough on
        performing read test
        block size: 524288
        using random I/O (alignment: 524288)
        number of outstanding I/O operations: 8
        thread stride size: 0
        threads per file: 4
        using I/O Completion Ports
        IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time:    10.01s
thread count:        4
proc count:        4

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|  18.73%|   1.09%|   17.63%|  81.30%
   1|  14.67%|   0.47%|   14.20%|  85.36%
   2|  22.63%|   0.94%|   21.69%|  77.40%
   3|  13.11%|   0.62%|   12.48%|  86.92%
-------------------------------------------
avg.|  17.28%|   0.78%|   16.50%|  82.75%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      8023179264 |        15303 |     764.19 |    1528.38 |    5.232 |     0.111 | \\192.168.10.1\d$\testfile.dat (500MB)
     1 |      8021082112 |        15299 |     763.99 |    1527.98 |    5.233 |     0.110 | \\192.168.10.1\d$\testfile.dat (500MB)
     2 |      8024227840 |        15305 |     764.29 |    1528.58 |    5.232 |     0.077 | \\192.168.10.1\d$\testfile.dat (500MB)
     3 |      8014790656 |        15287 |     763.39 |    1526.78 |    5.238 |     0.241 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total:       32083279872 |        61194 |    3055.86 |    6111.71 |    5.234 |     0.149

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      8023179264 |        15303 |     764.19 |    1528.38 |    5.232 |     0.111 | \\192.168.10.1\d$\testfile.dat (500MB)
     1 |      8021082112 |        15299 |     763.99 |    1527.98 |    5.233 |     0.110 | \\192.168.10.1\d$\testfile.dat (500MB)
     2 |      8024227840 |        15305 |     764.29 |    1528.58 |    5.232 |     0.077 | \\192.168.10.1\d$\testfile.dat (500MB)
     3 |      8014790656 |        15287 |     763.39 |    1526.78 |    5.238 |     0.241 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total:       32083279872 |        61194 |    3055.86 |    6111.71 |    5.234 |     0.149

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
     1 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
     2 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
     3 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A


  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      3.778 |        N/A |      3.778
   25th |      5.233 |        N/A |      5.233
   50th |      5.234 |        N/A |      5.234
   75th |      5.235 |        N/A |      5.235
   90th |      5.237 |        N/A |      5.237
   95th |      5.240 |        N/A |      5.240
   99th |      5.304 |        N/A |      5.304
3-nines |      6.361 |        N/A |      6.361
4-nines |     14.206 |        N/A |     14.206
5-nines |     17.738 |        N/A |     17.738
6-nines |     17.738 |        N/A |     17.738
7-nines |     17.738 |        N/A |     17.738
8-nines |     17.738 |        N/A |     17.738
9-nines |     17.738 |        N/A |     17.738
    max |     17.738 |        N/A |     17.738

 

[PnoT]

It looks like your speeds are pretty good on that read test at 3055MB/sec and if you want RDMA on VMs you'll need Hyper-V 2016 and setup a SET switch. Remote Direct Memory Access (RDMA) and Switch Embedded Teaming (SET)

I've never had PRO versions of the cards but here's a general comparison: Mellanox Adapters - Comparison Table | Mellanox Interconnect Community

We use ConnectX-3 and RDMA in our production and development environments for Live Migrations and SMB 3.0 shares. I'm currently building a new cluster to leverage the new capabilities in 2016 for RDMA and VMs.

 

[funkywizard]

Thanks for the reply. Does the extra hardware acceleration of the CX3 Pro cards assist anything other than RoCEv2?

Accelerates VXLAN as well.

 

[arglebargle]

I don't think there's any extra acceleration on the Pro cards beyond the hw offloads for RoCE v2. Sorry if I gave that impression, I was operating on about 2 hours of sleep when I read through the product documentation.

AFAIK the Pro model buys you the ability to run RoCE over standard ethernet without PFC + ECN. The Pro is also a currently supported product, if you care about that.

Edit: ^ see comment above, I stand corrected.

Can someone point out the supposed $30 deal on CX-3 Pros that happened in January? I can't find a link anywhere and I'm pretty sure any $30 deal was for standard CX-3s

 

[funkywizard]

I don't think there's any extra acceleration on the Pro cards beyond the hw offloads for RoCE v2. Sorry if I gave that impression, I was operating on about 2 hours of sleep when I read through the product documentation.

AFAIK the Pro model buys you the ability to run RoCE over standard ethernet without PFC + ECN but not much more. The Pro model is also a currently supported product, if you care about that.

Edit: ^ see comment above, I stand corrected.

Can someone point out the $30 deal on CX3 Pros that happened in January? I can't find a link anywhere. I see plenty of deals on CX3s but I'm not seeing the Pros.

I'm wondering the same. In addition to pros, any deals on dual port 40g C3 or better?

 

[arglebargle]

I'm wondering the same. In addition to pros, any deals on dual port 40g C3 or better?

The regular CX3 VPI dual 40Gb card? Definitely, try these:

("cx3 vpi dual 40gb",mcx354a,7046442,649281-b21) -bracket in "Computers, Tablets, and Networking" | eBay

("cx3 vpi dual 40gb",00d9550,00d9551,00d9552,sn30m27771,95y3456,95y3457,540-bbkh,6rknm,06rknm,0231a7d1,0231a7d2) -bracket in "Computers, Tablets, and Networking" | eBay

That's most of the common CX3 VPI dual 40Gb OEM cards I could find, I have alerts for those at <$25/ea. I have two of the HP 649281-B21 and they're rebadged MCX354A-FCBT's.

 

[Cutha]

Thanks for the info guys.

I just picked up 5 in a lot for $175USD. Hopefully I bought what I think I bought.

HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Lot of 5 LowProfile Bracket 886111482092 | eBay

 

[KC8FLB]

I bought 3 from this seller:

HP 649281-B21 Infiniband FDR/EN 10/40GB 2P 544QSFP ADAPTER 656089-001 661685-001 | eBay

seller accepter BO of $28 @ QTY:3

 

[William]

Isn't that the same card as this for just 35 USD? HP 649281-B21 Infiniband FDR/EN 10/40GB 2P 544QSFP ADAPTER 656089-001 661685-001 | eBay

Do you know by chance if there is Windows 10 drivers for these cards ?
I was trying to search them out but have only come up with Server 2008, 2012 drivers.
Maybe 2012 drivers would work ?

 

[Rain]

Do you know by chance if there is Windows 10 drivers for these cards ?
I was trying to search them out but have only come up with Server 2008, 2012 drivers.
Maybe 2012 drivers would work ?

Where are you looking? On Windows, ConnectX-3 and ConnectX-3 Pro cards are supported by WinOF (not WinOF 2) which can be downloaded on this page: http://www.mellanox.com/page/products_dyn?product_family=32&menu_section=34 (Select the WinOF tab at the bottom).