ConnectX-3 Pro MCX354A-FCCT FDR10 40/56GbE 2 PORT QSFP, $109 shipped, 8 available

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
No, the link in the OP is for a CX-3 Pro, a later gen card with a few new features and current support from Mellanox.

The Pro will do RoCE v2 (RDMA over Ethernet without PFC+ECN) -- the standard CX-3 requires very expensive switches to implement a lossless network and support RDMA (v1) as advertised.

The CX-3 is a ludicrously good deal that's hard to beat, I only posted this for people who need the feature set of the Pro for specific use cases.
 
Last edited:

kiteboarder

Active Member
May 10, 2016
101
48
28
45
Yeah, unless you need RDMA/RoCE over L3, just save some cash and make a best offer of 30 each. :)

I bought 4 of those HP branded cards and they're working great. Latest firmware can be found on HP's website.
 

wardtj

Member
Jan 23, 2015
91
28
18
47
There was a deal back in January for the same cx-3 pros for 20$ each. I'd hold out or Bo it for max of 40$. These come on fire sale often enough.

I'm now looking for 100gbe cx4. Hovering around 350 for dual cards.. 40gbe rdma was a great upgrade, now just want to push it...
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
There was a deal back in January for the same cx-3 pros for 20$ each. I'd hold out or Bo it for max of 40$. These come on fire sale often enough.

I'm now looking for 100gbe cx4. Hovering around 350 for dual cards.. 40gbe rdma was a great upgrade, now just want to push it...
I don't think that was for Pros, I'm pretty sure that was for standard CX-3s.
 
Last edited:

awedio

Active Member
Feb 24, 2012
776
225
43
There was a deal back in January for the same cx-3 pros for 20$ each. I'd hold out or Bo it for max of 40$. These come on fire sale often enough.

I'm now looking for 100gbe cx4. Hovering around 350 for dual cards.. 40gbe rdma was a great upgrade, now just want to push it...
Where are you finding 'em for $350?
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
The linked card is a CX-3 Pro with hardware acceleration for RoCE v2 (routable RDMA over Ethernet.) The standard CX-3 only supports RoCE v1 "locally" (without routing) and without hardware acceleration* (someone fact check this last bit here, the docs are confusing on not enough sleep.)

RDMA over Converged Ethernet - Wikipedia
http://www.mellanox.com/page/products_dyn?product_family=79

As I understand it you're buying the ability to run RDMA over more complicated Ethernet topologies with better hardware acceleration on the CX-3 Pro.
I thought I had this figured out but I think I am missing something critical. As I understand it, there are three implementations of RDMA. InfiniBand, iWarp and RoCE. InfiniBand is lossless so it does not require PFC and DCB but for the life of me I can't find any docs on how to keep it pure IB. If I had 3 windows servers with ConnectX-2 or ConnectX-3 cards and an IB switch why would I want RoCE?

RoCE_versus_InfiniBand

I was just about to buy some CX-3 Pro cards for the hardware acceleration but now I don't understand why I would need it. My servers and workstations are connected via a traditional 1gig Ethernet. The servers have ConnectX-2 cards (MHQH29B-XTR, flashed to a firmware that supports RDMA, 2.10.720) and there is a Mellanox switch (4036) connecting them and a few of the workstations. The InfiniBand NIC's are on a different subnet then the Cat6 network and there is no expectations that the InfiniBand network will need to talk to the Cat6.

To get SR-IOV I need ConnectX-3 cards, but do I need the Pro version? Would the Pro version provide any speed benefits. Hopefully my questions aren't so stupid that it leaves people scratching their heads trying to figure out what I am asking.
 

PnoT

Active Member
Mar 1, 2015
650
162
43
Texas
You're running pure IB right now with that setup you have and the main reason to switch to RoCE is the ability to route over traditional Ethernet. I'm also pretty sure ConnectX-3 cards support SR-IOV but do you have a use case for that? Have you done speeds tests to see what you're getting right now over IB? If you do testing make sure to create RAM drives on the source and destination to get a good guage of speeds unless you have storage on both devices that's up to the task.
 

Cutha

Member
Sep 24, 2016
75
7
8
50
Canada
Thanks for the reply. Does the extra hardware acceleration of the CX3 Pro cards assist anything other than RoCEv2?

For Hyper-V I wanted SR-IOV cards. When I setup this all up I didn't realize that I couldn't just get RDMA on the virtual machines. Not having to deal with PFC and DCB makes it much less complex. When I first set this up 2 years ago I benchmarked everything as best as I could. I used a RAM drive, here are the results (I am glad I saved them all):

Code:
Command Line: diskspd.exe -c500M -d10 -r -w0 -t4 -o8 -b512K -Sh -L \\192.168.10.1\d$\testfile.dat

Input parameters:

    timespan:   1
    -------------
    duration: 10s
    warm up time: 5s
    cool down time: 0s
    measuring latency
    random seed: 0
    path: '\\192.168.10.1\d$\testfile.dat'
        think time: 0ms
        burst size: 0
        software cache disabled
        hardware write cache disabled, writethrough on
        performing read test
        block size: 524288
        using random I/O (alignment: 524288)
        number of outstanding I/O operations: 8
        thread stride size: 0
        threads per file: 4
        using I/O Completion Ports
        IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time:    10.01s
thread count:        4
proc count:        4

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|  18.73%|   1.09%|   17.63%|  81.30%
   1|  14.67%|   0.47%|   14.20%|  85.36%
   2|  22.63%|   0.94%|   21.69%|  77.40%
   3|  13.11%|   0.62%|   12.48%|  86.92%
-------------------------------------------
avg.|  17.28%|   0.78%|   16.50%|  82.75%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      8023179264 |        15303 |     764.19 |    1528.38 |    5.232 |     0.111 | \\192.168.10.1\d$\testfile.dat (500MB)
     1 |      8021082112 |        15299 |     763.99 |    1527.98 |    5.233 |     0.110 | \\192.168.10.1\d$\testfile.dat (500MB)
     2 |      8024227840 |        15305 |     764.29 |    1528.58 |    5.232 |     0.077 | \\192.168.10.1\d$\testfile.dat (500MB)
     3 |      8014790656 |        15287 |     763.39 |    1526.78 |    5.238 |     0.241 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total:       32083279872 |        61194 |    3055.86 |    6111.71 |    5.234 |     0.149

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      8023179264 |        15303 |     764.19 |    1528.38 |    5.232 |     0.111 | \\192.168.10.1\d$\testfile.dat (500MB)
     1 |      8021082112 |        15299 |     763.99 |    1527.98 |    5.233 |     0.110 | \\192.168.10.1\d$\testfile.dat (500MB)
     2 |      8024227840 |        15305 |     764.29 |    1528.58 |    5.232 |     0.077 | \\192.168.10.1\d$\testfile.dat (500MB)
     3 |      8014790656 |        15287 |     763.39 |    1526.78 |    5.238 |     0.241 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total:       32083279872 |        61194 |    3055.86 |    6111.71 |    5.234 |     0.149

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
     1 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
     2 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
     3 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A


  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      3.778 |        N/A |      3.778
   25th |      5.233 |        N/A |      5.233
   50th |      5.234 |        N/A |      5.234
   75th |      5.235 |        N/A |      5.235
   90th |      5.237 |        N/A |      5.237
   95th |      5.240 |        N/A |      5.240
   99th |      5.304 |        N/A |      5.304
3-nines |      6.361 |        N/A |      6.361
4-nines |     14.206 |        N/A |     14.206
5-nines |     17.738 |        N/A |     17.738
6-nines |     17.738 |        N/A |     17.738
7-nines |     17.738 |        N/A |     17.738
8-nines |     17.738 |        N/A |     17.738
9-nines |     17.738 |        N/A |     17.738
    max |     17.738 |        N/A |     17.738
 

PnoT

Active Member
Mar 1, 2015
650
162
43
Texas
It looks like your speeds are pretty good on that read test at 3055MB/sec and if you want RDMA on VMs you'll need Hyper-V 2016 and setup a SET switch. Remote Direct Memory Access (RDMA) and Switch Embedded Teaming (SET)

I've never had PRO versions of the cards but here's a general comparison: Mellanox Adapters - Comparison Table | Mellanox Interconnect Community

We use ConnectX-3 and RDMA in our production and development environments for Live Migrations and SMB 3.0 shares. I'm currently building a new cluster to leverage the new capabilities in 2016 for RDMA and VMs.
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
I don't think there's any extra acceleration on the Pro cards beyond the hw offloads for RoCE v2. Sorry if I gave that impression, I was operating on about 2 hours of sleep when I read through the product documentation.

AFAIK the Pro model buys you the ability to run RoCE over standard ethernet without PFC + ECN. The Pro is also a currently supported product, if you care about that.

Edit: ^ see comment above, I stand corrected.

Can someone point out the supposed $30 deal on CX-3 Pros that happened in January? I can't find a link anywhere and I'm pretty sure any $30 deal was for standard CX-3s
 
Last edited:

funkywizard

mmm.... bandwidth.
Jan 15, 2017
848
402
63
USA
ioflood.com
I don't think there's any extra acceleration on the Pro cards beyond the hw offloads for RoCE v2. Sorry if I gave that impression, I was operating on about 2 hours of sleep when I read through the product documentation.

AFAIK the Pro model buys you the ability to run RoCE over standard ethernet without PFC + ECN but not much more. The Pro model is also a currently supported product, if you care about that.

Edit: ^ see comment above, I stand corrected.

Can someone point out the $30 deal on CX3 Pros that happened in January? I can't find a link anywhere. I see plenty of deals on CX3s but I'm not seeing the Pros.
I'm wondering the same. In addition to pros, any deals on dual port 40g C3 or better?
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
I'm wondering the same. In addition to pros, any deals on dual port 40g C3 or better?
The regular CX3 VPI dual 40Gb card? Definitely, try these:

("cx3 vpi dual 40gb",mcx354a,7046442,649281-b21) -bracket in "Computers, Tablets, and Networking" | eBay

("cx3 vpi dual 40gb",00d9550,00d9551,00d9552,sn30m27771,95y3456,95y3457,540-bbkh,6rknm,06rknm,0231a7d1,0231a7d2) -bracket in "Computers, Tablets, and Networking" | eBay

That's most of the common CX3 VPI dual 40Gb OEM cards I could find, I have alerts for those at <$25/ea. I have two of the HP 649281-B21 and they're rebadged MCX354A-FCBT's.
 
Last edited:

Rain

Active Member
May 13, 2013
276
124
43