Mellanox MCX311A-XCAT 10 Gigabit Ethernet PCIe Adapter Card ConnectX-3 CX311A $12 obo + $15 Shipping

well, i received my order of 10 today. they are tiny, and indeed half height brackets. they were just wrapped in bubble wrap; would really have preferred them to individually placed in anti-static bags, but oh well... will try to test them out over the weekend of next week.

 

The latest sale where it states 50 items per lot is actually a mistake. I just got notified by the seller asking if I wanted a refund or the individual cards. That being said, if these are truly the ConnectX-3 cards and are fully functional, then even at $23 a card with the shipping is still a great price.

 

Hopefully for $23 you'll get it shipped properly (anti-static bags etc). People should give feedback on that even when the price was low as $12. I would prefer packed inside a newspaper than inside a bubble wrap.

 

@hakabe, have you tested your cards yet? with the normal price of ~$80-$100 a card on ebay for something comparable, I am wondering if this is just an awesome deal or if someone made a different mistake or sold a bunch of untested cards.

 

No, my package is on its way. Probably it takes a week to get into the EU. On eBay there's Dual port 10/40Gbps Mellanox NICs for ~$60.

I am keen to see the results from the users who has already received the NICs.

 

I received my cards yesterday and did some quick testing this morning on one of the cards.

As previously noted, these are half height cards with low profile brackets. Compared to the previous gen ConnectX2 card (MNPA19) the MCX311A card is approx 3/4 length, and has a x4 pci-e connector compared to the x8 pci-e connector on the MNPA19. The 2 bracket mounting holes appear to be identical. The "status LEDs" are different between the MNPA19 and MCX311A .

I swapped out the tall bracket from a MNPA19-XTR card that I have in order to test the MCX311A card in my S2012R2 4U test box (which I have previously had all sorts of Mellanox cards in) and it was recognized right away.

I grabbed the latest drivers/firmware/MFT/documentation from the Mellanox site.Previous generations of Mellanox VPI/Eth cards required FW to be downloaded and burnt but in reading the docs for these cards they apparently can be flashed "online". The FW on the card was older so I flashed using the "online" ability ..

Code:

PS C:\downloads\mellanox cx311a s2012r2> mst status
MST devices:
------------
  mt4099_pci_cr0
  mt4099_pciconf0
PS C:\downloads\mellanox cx311a s2012r2> mlxfwmanager -d mt4099_pci_cr0 --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX311A-XCA_A1
  Description:      ConnectX-3 EN network interface card; 10GigE; single-port SFP+; PCIe3.0 x4 8GT/s; RoHS R6
  PSID:             MT_1170110023
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        0002c9239a10
  Port2 MAC:        0002c9239a11
  Versions:         Current        Available
     FW             2.10.4290      N/A
     PXE            3.4.0000       N/A

  Status:           No matching image found

PS C:\downloads\mellanox cx311a s2012r2> mlxfwmanager --online -u -d mt4099_pci_cr0
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX311A-XCA_A1
  Description:      ConnectX-3 EN network interface card; 10GigE; single-port SFP+; PCIe3.0 x4 8GT/s; RoHS R6
  PSID:             MT_1170110023
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        0002c9239a10
  Port2 MAC:        0002c9239a11
  Versions:         Current        Available
     FW             2.10.4290      2.40.7000
     PXE            3.4.0000       3.4.0746

  Status:           Update required

Release notes for the available Firmware:
-----------------------------------------

  For more details, please refer to the following FW release notes:
    1- ConnectX3 (2.40.7000):    http://www.mellanox.com/pdf/firmware/ConnectX3-FW-2_40_7000-release_notes.pdf
    2- ConnectX3Pro (2.40.7000): http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_40_7000-release_notes.pdf
    3- Connect-IB (10.16.1020):  http://www.mellanox.com/pdf/firmware/ConnectIB-FW-10_16_1020-release_notes.pdf
    4- ConnectX4 (12.20.1010):   http://www.mellanox.com/pdf/firmware/ConnectX4-FW-12_20_1010-release_notes.pdf
    5- ConnectX4Lx (14.20.1010): http://www.mellanox.com/pdf/firmware/ConnectX4Lx-FW-14_20_1010-release_notes.pdf
    6- ConnectX5 (16.20.1010):   http://www.mellanox.com/pdf/firmware/ConnectX5-FW-16_20_1010-release_notes.pdf

---------
Found 1 device(s) requiring firmware update...

Perform FW update? [y/N]: y

Please wait while downloading MFA(s) 100%
Device #1: Updating FW ...
Done

Restart needed for updates to take effect.
PS C:\downloads\mellanox cx311a s2012r2>

The tall bracket for the MCX311A appears to be available at the Mellanox Store ..

So all in all I rate this under the "awesome deal" category !

 

Great news @britinpdx, I purchased 2 of the cards under the first deal and then regretted that I didn't get more even if I couldn't use them all, I could have at least have some spares. When I realized that the same seller was selling more cards, I got more of those cards through a offer so I got them a bit cheaper than $23 and now I am happy that I did.
With these cards supporting RDMA, they should do great for using them for SDS(Software Defined Storage) testing which requires really low latency network configuration.

 

Verified with the Finisar FTLX8571D3BCL.

 

Got mine in. All but one were made in China. One was made in Israel. Looks identical to the Chinese cards.
All came wrapped nicely in pink antistatic bubble wrap. This wrap will protect cards from static buildup but not static discharge, just have to ground yourself before handling.

 

One foot to the heater radiator and you're all set

 

Only Connectx-3 Pro cards and Connectx-4 and above cards support RDMA aka ROCE for these Eth cards.

 

I sure hope you are wrong in that statement.
Reading documentation it doesn't state anything regarding only Pro versions having the RDMA capability:
http://www.mellanox.com/related-docs/prod_adapter_cards/ConnectX3_EN_Card.pdf

 

I'm right. The Brochure caused me to think the same thing thus why I own some pro cards and non pro cards. Here's the details you're looking for : http://www.mellanox.com/page/products_dyn?product_family=79 .. Scroll down to hardware support at the bottom.
discussion :
Difference between PRO VPI and PRO EN adapter (... | Mellanox Interconnect Community

I too wish that features were detailed on a concise splash page for equal comparison. Don't even get me started on roce v1 vs. v2. So, there it is.. Roce is only available on connectx-3Pro and Connectx-4 and up. There is a software emulation for Roce for cards that don't support it but you're ofc looking at added latency as it wont be done in hardware on the Connectx card? Here's linkage to the software implementation :
Software-based RoCE, A New Way to Experience RDMA - RoCE Initiative

 

@0xbit, I guess I am still hopeful and here is why:

1. User ophirmaor in the community thread is explicitly stating that RDMA is supported on both types (VPI and EN) and that Pro supports RoCE v2 but non-Pro supports v1
2. Reading the WinOF (not WinOF-2) user manual which can be found here: LINK
   is stating that RoCE is supported on ConnectX-3 and ConnectX-3 Pro with firmware version 2.30.3000 or higher (page 44 under System Requirements)

What I never even though about is the SoftROCE which sounds kind of interesting and I am wondering if it is a cheap solution for the home lab configurations as long as your hardware is beefy enough. Unfortunately it is only supported for Linux OSs which doesn't help with Hyper-V scenarios.

Regarding lack of RoCE v2 support, the only time I see this really causing me an issue is if I decide to do nested virtualization and expose storage directly to some of those nested hypervisors which would most likely require traffic routing and this RoCE routing is only supported in v2.

I really need to plug these cards in and play with them a bit and see if I can gain any benefit over my normal non RoCE storage access that I have going on at the moment.

 

RDMA over converged ethernet works with mellanox adapters since the cx-2 ethernet adapters (the "cheap as dirt" cards ).
For cx-2 you will need a certain firmware version that's only available through hp, cx-3 and newer adapters should support it with any firmware version.
For rdma on windows you need to enable "NDK Global Pause" in thre driver settings.

 

Fudge

Double Fudge

So, now I have super pricey Pro cards that I didn't even need .. FML
Any other benefit of these pro cards? do they do more hardware level processing of network traffic data? Time to fire up ebay, offload these cards I have and wait for a mcx311a-xcat-10-gbit sale .

Also, I never found that Connect-x2 ETH HP firmware website link even though everybody mentions it? Can you provide linkage please?

 

He has reposted and there's 3 left. Mellanox MCX311A-XCAT 10 Gigabit Ethernet PCIe Adapter Card ConnectX-3 MCX311A | eBay

 

So, seeing as how I will be 3d printing myself a dozen full height brackets for these cards, if anyone else needs some send me a PM.

 

finally had some time to test these out last night. some basic 2-way testing with iperf3 revealed two things to me:

1) if you setup the IP wrong and end up iperf3 connecting to yourself, it'll show you 20Gbps bandwidth!!!
2) if you didn't know your motherboard has 1 slot that is PCI-E 1.0, this card will only operate around 5Gbps in that slot!

after all that silliness, all 10 of my order worked perfectly fine with FS SFP+ modules and Cisco DACs. these don't seem to be too picky about SFP+ or cables.

 

Awesome! Could I trouble u to print say 4 for me?