Mellanox MCX311A-XCAT 10 Gigabit Ethernet PCIe Adapter Card ConnectX-3 CX311A $12 obo + $15 Shipping
- Thread starter zeynel
- Start date
The latest sale where it states 50 items per lot is actually a mistake. I just got notified by the seller asking if I wanted a refund or the individual cards. That being said, if these are truly the ConnectX-3 cards and are fully functional, then even at $23 a card with the shipping is still a great price.
Hopefully for $23 you'll get it shipped properly (anti-static bags etc). People should give feedback on that even when the price was low as $12. I would prefer packed inside a newspaper than inside a bubble wrap.
No, my package is on its way. Probably it takes a week to get into the EU. On eBay there's Dual port 10/40Gbps Mellanox NICs for ~$60.
I am keen to see the results from the users who has already received the NICs.
I received my cards yesterday and did some quick testing this morning on one of the cards.
As previously noted, these are half height cards with low profile brackets. Compared to the previous gen ConnectX2 card (MNPA19) the MCX311A card is approx 3/4 length, and has a x4 pci-e connector compared to the x8 pci-e connector on the MNPA19. The 2 bracket mounting holes appear to be identical. The "status LEDs" are different between the MNPA19 and MCX311A .
I swapped out the tall bracket from a MNPA19-XTR card that I have in order to test the MCX311A card in my S2012R2 4U test box (which I have previously had all sorts of Mellanox cards in) and it was recognized right away.
I grabbed the latest drivers/firmware/MFT/documentation from the Mellanox site.Previous generations of Mellanox VPI/Eth cards required FW to be downloaded and burnt but in reading the docs for these cards they apparently can be flashed "online". The FW on the card was older so I flashed using the "online" ability ..
Code:
PS C:\downloads\mellanox cx311a s2012r2> mst status
MST devices:
------------
mt4099_pci_cr0
mt4099_pciconf0
PS C:\downloads\mellanox cx311a s2012r2> mlxfwmanager -d mt4099_pci_cr0 --query
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX311A-XCA_A1
Description: ConnectX-3 EN network interface card; 10GigE; single-port SFP+; PCIe3.0 x4 8GT/s; RoHS R6
PSID: MT_1170110023
PCI Device Name: mt4099_pci_cr0
Port1 MAC: 0002c9239a10
Port2 MAC: 0002c9239a11
Versions: Current Available
FW 2.10.4290 N/A
PXE 3.4.0000 N/A
Status: No matching image found
PS C:\downloads\mellanox cx311a s2012r2> mlxfwmanager --online -u -d mt4099_pci_cr0
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX311A-XCA_A1
Description: ConnectX-3 EN network interface card; 10GigE; single-port SFP+; PCIe3.0 x4 8GT/s; RoHS R6
PSID: MT_1170110023
PCI Device Name: mt4099_pci_cr0
Port1 MAC: 0002c9239a10
Port2 MAC: 0002c9239a11
Versions: Current Available
FW 2.10.4290 2.40.7000
PXE 3.4.0000 3.4.0746
Status: Update required
Release notes for the available Firmware:
-----------------------------------------
For more details, please refer to the following FW release notes:
1- ConnectX3 (2.40.7000): http://www.mellanox.com/pdf/firmware/ConnectX3-FW-2_40_7000-release_notes.pdf
2- ConnectX3Pro (2.40.7000): http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_40_7000-release_notes.pdf
3- Connect-IB (10.16.1020): http://www.mellanox.com/pdf/firmware/ConnectIB-FW-10_16_1020-release_notes.pdf
4- ConnectX4 (12.20.1010): http://www.mellanox.com/pdf/firmware/ConnectX4-FW-12_20_1010-release_notes.pdf
5- ConnectX4Lx (14.20.1010): http://www.mellanox.com/pdf/firmware/ConnectX4Lx-FW-14_20_1010-release_notes.pdf
6- ConnectX5 (16.20.1010): http://www.mellanox.com/pdf/firmware/ConnectX5-FW-16_20_1010-release_notes.pdf
---------
Found 1 device(s) requiring firmware update...
Perform FW update? [y/N]: y
Please wait while downloading MFA(s) 100%
Device #1: Updating FW ...
Done
Restart needed for updates to take effect.
PS C:\downloads\mellanox cx311a s2012r2>So all in all I rate this under the "awesome deal" category !
Great news @britinpdx, I purchased 2 of the cards under the first deal and then regretted that I didn't get more even if I couldn't use them all, I could have at least have some spares. When I realized that the same seller was selling more cards, I got more of those cards through a offer so I got them a bit cheaper than $23 and now I am happy that I did.
With these cards supporting RDMA, they should do great for using them for SDS(Software Defined Storage) testing which requires really low latency network configuration.
With these cards supporting RDMA, they should do great for using them for SDS(Software Defined Storage) testing which requires really low latency network configuration.
Got mine in. All but one were made in China. One was made in Israel. Looks identical to the Chinese cards.
All came wrapped nicely in pink antistatic bubble wrap. This wrap will protect cards from static buildup but not static discharge, just have to ground yourself before handling.
All came wrapped nicely in pink antistatic bubble wrap. This wrap will protect cards from static buildup but not static discharge, just have to ground yourself before handling.
Only Connectx-3 Pro cards and Connectx-4 and above cards support RDMA aka ROCE for these Eth cards.
I sure hope you are wrong in that statement.
Reading documentation it doesn't state anything regarding only Pro versions having the RDMA capability:
http://www.mellanox.com/related-docs/prod_adapter_cards/ConnectX3_EN_Card.pdf
I'm right. The Brochure caused me to think the same thing thus why I own some pro cards and non pro cards. Here's the details you're looking for : http://www.mellanox.com/page/products_dyn?product_family=79 .. Scroll down to hardware support at the bottom.
discussion :
Difference between PRO VPI and PRO EN adapter (... | Mellanox Interconnect Community
I too wish that features were detailed on a concise splash page for equal comparison. Don't even get me started on roce v1 vs. v2. So, there it is.. Roce is only available on connectx-3Pro and Connectx-4 and up. There is a software emulation for Roce for cards that don't support it but you're ofc looking at added latency as it wont be done in hardware on the Connectx card? Here's linkage to the software implementation :
Software-based RoCE, A New Way to Experience RDMA - RoCE Initiative
@0xbit, I guess I am still hopeful and here is why:
What I never even though about is the SoftROCE which sounds kind of interesting and I am wondering if it is a cheap solution for the home lab configurations as long as your hardware is beefy enough. Unfortunately it is only supported for Linux OSs which doesn't help with Hyper-V scenarios.
Regarding lack of RoCE v2 support, the only time I see this really causing me an issue is if I decide to do nested virtualization and expose storage directly to some of those nested hypervisors which would most likely require traffic routing and this RoCE routing is only supported in v2.
I really need to plug these cards in and play with them a bit and see if I can gain any benefit over my normal non RoCE storage access that I have going on at the moment.
- User ophirmaor in the community thread is explicitly stating that RDMA is supported on both types (VPI and EN) and that Pro supports RoCE v2 but non-Pro supports v1
- Reading the WinOF (not WinOF-2) user manual which can be found here: LINK
is stating that RoCE is supported on ConnectX-3 and ConnectX-3 Pro with firmware version 2.30.3000 or higher (page 44 under System Requirements)
What I never even though about is the SoftROCE which sounds kind of interesting and I am wondering if it is a cheap solution for the home lab configurations as long as your hardware is beefy enough. Unfortunately it is only supported for Linux OSs which doesn't help with Hyper-V scenarios.
Regarding lack of RoCE v2 support, the only time I see this really causing me an issue is if I decide to do nested virtualization and expose storage directly to some of those nested hypervisors which would most likely require traffic routing and this RoCE routing is only supported in v2.
I really need to plug these cards in and play with them a bit and see if I can gain any benefit over my normal non RoCE storage access that I have going on at the moment.
RDMA over converged ethernet works with mellanox adapters since the cx-2 ethernet adapters (the "cheap as dirt" cards 
).
For cx-2 you will need a certain firmware version that's only available through hp, cx-3 and newer adapters should support it with any firmware version.
For rdma on windows you need to enable "NDK Global Pause" in thre driver settings.
).For cx-2 you will need a certain firmware version that's only available through hp, cx-3 and newer adapters should support it with any firmware version.
For rdma on windows you need to enable "NDK Global Pause" in thre driver settings.
Fudge
Double FudgeRDMA over converged ethernet works with mellanox adapters since the cx-2 ethernet adapters (the "cheap as dirt" cards
For cx-2 you will need a certain firmware version that's only available through hp, cx-3 and newer adapters should support it with any firmware version.
For rdma on windows you need to enable "NDK Global Pause" in thre driver settings.
So, now I have super pricey Pro cards that I didn't even need .. FML
Any other benefit of these pro cards? do they do more hardware level processing of network traffic data? Time to fire up ebay, offload these cards I have and wait for a mcx311a-xcat-10-gbit sale .
Also, I never found that Connect-x2 ETH HP firmware website link even though everybody mentions it? Can you provide linkage please?
Last edited:
He has reposted and there's 3 left. Mellanox MCX311A-XCAT 10 Gigabit Ethernet PCIe Adapter Card ConnectX-3 MCX311A | eBay
finally had some time to test these out last night. some basic 2-way testing with iperf3 revealed two things to me:
1) if you setup the IP wrong and end up iperf3 connecting to yourself, it'll show you 20Gbps bandwidth!!!
2) if you didn't know your motherboard has 1 slot that is PCI-E 1.0, this card will only operate around 5Gbps in that slot!
after all that silliness, all 10 of my order worked perfectly fine with FS SFP+ modules and Cisco DACs. these don't seem to be too picky about SFP+ or cables.
1) if you setup the IP wrong and end up iperf3 connecting to yourself, it'll show you 20Gbps bandwidth!!!
2) if you didn't know your motherboard has 1 slot that is PCI-E 1.0, this card will only operate around 5Gbps in that slot!
after all that silliness, all 10 of my order worked perfectly fine with FS SFP+ modules and Cisco DACs. these don't seem to be too picky about SFP+ or cables.