Mellanox MCX311A-XCAT 10 Gigabit Ethernet PCIe Adapter Card ConnectX-3 CX311A $12 obo + $15 Shipping

_alex

Active Member
Jan 28, 2016
874
94
28
Bavaria / Germany
For rdma on windows you need to enable "NDK Global Pause" in thre driver settings.
I think for stable/reliable Performance the whole network should be lossless.
This means also the Switch should be configured for 'Global Pause' or 'PFC' on the Ports used for RoCE - what can be quite painful and a limit on usable Switches.

Never setup RoCE because of this but would be interested in some reports if it's worth the efforts involved.
 

Spartus

Active Member
Mar 28, 2012
170
25
28
Ottawa, Canada
I got a few requests for printed brackets. Will do, but haven't designed them yet, because I haven't received the cards yet. If you are in a rush I would suggest ordering from Mellanox. I'll report back once I have designed and printed mine.

Cheers, all.
 

iLya

Member
Jun 22, 2016
48
5
8
I am starting to play with the cards as well and so far I am finding that the firmware is slightly outdated.
If you are interested in updating the firmware, you can download the MFT package from Mellanox using this link:

Mellanox Firmware Tools
MFT User Manual

In the manual they have a section that talks about being able to check the firmware directly against their website and perform the update.
However what I've discovered is that the firmware that appears to be available from Mellanox website directly is actually newer than what the tool is able to detect.
Here is the link to the latest firmware that I was able to find for these cards:
FW-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin.zip

I also think its possible to enable RDMA with these cards using the Quanta LB6M 10GB switch because it supports DCB and PFC which should enable the lossless network. Now I just need to learn how to setup VLANs, enable Trunk Ports and I should be a happy camper :)
 

iLya

Member
Jun 22, 2016
48
5
8
Fudge :(


Double Fudge :(

So, now I have super pricey Pro cards that I didn't even need .. FML
Any other benefit of these pro cards? do they do more hardware level processing of network traffic data? Time to fire up ebay, offload these cards I have and wait for a mcx311a-xcat-10-gbit sale .

Also, I never found that Connect-x2 ETH HP firmware website link even though everybody mentions it? Can you provide linkage please?

@0xbit,

Here is a link to the simple comparison that I found on the Mellanox website:

Adapter comparison

The biggest things that I see are related to RoCE v2 support (Routing) and hardware offloading for Overlay Networks (VXLAN/NVGRE) which are important for SDN (Software Defined Networking).
So it all depends on how crazy you want to go with your setup. If you are playing with things like AzureStack, then this is probably important and in fact you probably want to go to ConnectX-4 or 5 cards. However if you are doing a smaller home setup, then these cards should be plenty.

For me, I am going to figure out how to get RDMA working and in order to get around the RoCE v1 limitation of routing, I am going to change my network config to probably leverage the /16 address space because I am planning to build out a SDN based lab using SCVMM with Network Controllers, etc....

Once I get this sorted, I really want to get the ConnectX-5 cards because of the eSwitch capability and the fact that you can link up to 4 servers together without a switch by looping the dual ports together. Here is a LINK
 

i386

Well-Known Member
Mar 18, 2016
2,047
542
113
31
Germany
If these cards are vanilla mellanox cards just install the (win)ofed package from mellanox and it will update to the latest firmware.
 

iLya

Member
Jun 22, 2016
48
5
8
If these cards are vanilla mellanox cards just install the (win)ofed package from mellanox and it will update to the latest firmware.
I did install the WinOF package but the firmware was still older than what was available.
I ended up having to download the firmware directly from their site and using flint tool to flash it to the latest version.
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,528
975
113
artofserver.com
hey, how many of you guys need these brackets and how many? shipping from Mellanox might be expensive for small orders, but if I order in bulk, the high shipping costs is reduced per bracket and then I can probably re-send them back out 1st class post for cheap, likely $3 or so within USA. So, for small quantities, this might make sense to do.... i'm happy to donate my time as long as my costs are covered. the postman picks up packages from my place every day anyway and I've got some used envelopes I can reuse. let me know if that's something you guys want to consider and we can then figure out if there's enough people and the details.
 

homelabusr

Member
Feb 7, 2017
49
4
8
35
Just FYI,
Mellanox told me that the perforated ones would take 4-6 weeks to ship.
They we're okay with shipping a replacement bracket MTM005078 which was " not perforated, but should fit the MCX312B", those being 16 in stock.

I preferred to wait.

hey, how many of you guys need these brackets and how many? shipping from Mellanox might be expensive for small orders, but if I order in bulk, the high shipping costs is reduced per bracket and then I can probably re-send them back out 1st class post for cheap, likely $3 or so within USA. So, for small quantities, this might make sense to do.... i'm happy to donate my time as long as my costs are covered. the postman picks up packages from my place every day anyway and I've got some used envelopes I can reuse. let me know if that's something you guys want to consider and we can then figure out if there's enough people and the details.
 

antsh

New Member
Sep 14, 2017
21
0
1
hey, how many of you guys need these brackets and how many? shipping from Mellanox might be expensive for small orders, but if I order in bulk, the high shipping costs is reduced per bracket and then I can probably re-send them back out 1st class post for cheap, likely $3 or so within USA. So, for small quantities, this might make sense to do.... i'm happy to donate my time as long as my costs are covered. the postman picks up packages from my place every day anyway and I've got some used envelopes I can reuse. let me know if that's something you guys want to consider and we can then figure out if there's enough people and the details.
I'm in for one.
 

Serhan

Member
Sep 22, 2017
50
11
8
48
hey, how many of you guys need these brackets and how many? shipping from Mellanox might be expensive for small orders, but if I order in bulk, the high shipping costs is reduced per bracket and then I can probably re-send them back out 1st class post for cheap, likely $3 or so within USA. So, for small quantities, this might make sense to do.... i'm happy to donate my time as long as my costs are covered. the postman picks up packages from my place every day anyway and I've got some used envelopes I can reuse. let me know if that's something you guys want to consider and we can then figure out if there's enough people and the details.
I am in for one also


Sent from my Moto G (4) using Tapatalk
 

DavidRa

Infrastructure Architect
Aug 3, 2015
272
122
43
Central Coast of NSW
www.pdconsec.net
hey, how many of you guys need these brackets and how many? shipping from Mellanox might be expensive for small orders, but if I order in bulk, the high shipping costs is reduced per bracket and then I can probably re-send them back out 1st class post for cheap, likely $3 or so within USA. So, for small quantities, this might make sense to do.... i'm happy to donate my time as long as my costs are covered. the postman picks up packages from my place every day anyway and I've got some used envelopes I can reuse. let me know if that's something you guys want to consider and we can then figure out if there's enough people and the details.
Thanks for the offer - I'm seeing USD80 for the parts and USD70 for shipping 6 of them to Australia. Crazy.

What do you reckon it would end up being if I brokered through you? Obviously the parts are the same.
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,528
975
113
artofserver.com
Thanks for the offer - I'm seeing USD80 for the parts and USD70 for shipping 6 of them to Australia. Crazy.

What do you reckon it would end up being if I brokered through you? Obviously the parts are the same.
well, it looks like Mellanox just jacked up the price. it was previously $4 per bracket; shipping was a bit high. they probably saw a sudden demand for it and changed the price...