Mellanox ConnectX-3 Pro EN (MCX314A-BCCT)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

TonyStew

New Member
Aug 17, 2023
1
0
1
I actually just bought a couple from this seller on Ebay for $16 a piece with free 2-4 day shipping (or high-profile). Looking at the datasheet for it vs for non-pro connectx-3 cards it looks like the pro versions only support up to 40gb ethernet, and no version of infiniband. Running MST status on my cards doesn't give a link type setting specifying ib or eth, which supports that. What sets the pros apart is support for RoCE v2 which may be nice to have depending on your setup and what you're doing (clustered VMs in particular).

Something I noticed is my cards came with a newer firmware version than I've seen documented anywhere, 2.43.7010 from 6-5-2019. I need to bump one of the firmware threads and see if that's anything special.
 
Last edited:

Kahooli

Member
Dec 3, 2022
41
26
18
I actually just bought a couple from this seller on Ebay for $16 a piece with free 2-4 day shipping (or high-profile). Looking at the datasheet for it vs for non-pro connectx-3 cards it looks like the pro versions only support up to 40gb ethernet, and no version of infiniband. Running MST status on my cards doesn't give a link type setting specifying ib or eth, which supports that. What sets the pros apart is support for RoCE v2 which may be nice to have depending on your setup and what you're doing (clustered VMs in particular).

Something I noticed is my cards came with a newer firmware version than I've seen documented anywhere, 2.43.7010 from 6-5-2019. I need to bump one of the firmware threads and see if that's anything special.
Pretty sure you can crossflash to any of the pro chip based firmwares and get the functionality you want.
 

86turbodsl

Member
Feb 24, 2020
34
4
8
Cool, but just wondering what use a home lab has for Infiniband? I deal with this sometimes on clusters, which makes sense, but in a home?
 

krista

New Member
Dec 22, 2015
29
12
3
43
Cool, but just wondering what use a home lab has for Infiniband? I deal with this sometimes on clusters, which makes sense, but in a home?
heh, my home lab had fdr-ib on the rack and ran to a couple workstations in other rooms. then i had to move after a couple of decades there.

my apartment doesn't even have ethernet... so i'm going into the crawlspace and adding ethernet... as well as a couple mtp/mpo runs between the office/workshop and my bedroom/studio/workstation.

why?

i deal with large files of various types, from xml to long multichannel audio, 8k video, and other oddities.

plus i can treat my 4 server ”hyperconverged minicluster” as an extremely fast drive, especially using a ramcache on a tiered sas hdd raid with ssd for hot files.

this makes compiling big projects or building unreal 5 projects actually kinda spreedy.

---

- i'm also experimenting with rdma, spatial data, and distributed manipulation thereof (say, using multiple servers to run the physics sim for a very large world multiple local player vr thingy).

- on the back burner is using ib as a way to redirect the gpu framebuffer for a displayport based vr headset from the rendering pc/gpu to a different one. this has a lot of potential applications.

---

plus it's pretty entertaining watching someone's jaw drop when hucking 100gb files around in seconds...
 

maxermaxer

Active Member
Oct 28, 2016
288
48
28
48
I know this is probably a dumb question: is there any difference between cables for 10G and 40G? I am using 10G network but never used 40G before. :)
 
  • Like
Reactions: Samir

Nipp

New Member
May 20, 2022
16
5
3
Haymarket, VA
One more dumb question - can it be used with 10GbE switch? Like a reversed connection - 4x10 breakout cable from NIC to 4 ports on the switch, where it's aggregated to 40GbE. UniFi in particular.

I saw the post on STH main, about the advise to use the QSFP to SFP adapter and that in general it's not working like this. But there are some switches that can "re-aggregate" the QSFP with 5 ports. I wonder if UniFi might be one of these.

Thank you
 
Last edited:
  • Like
Reactions: Samir

blunden

Active Member
Nov 29, 2019
373
105
43
One more dumb question - can it be used with 10GbE switch? Like a reversed connection - 4x10 breakout cable from NIC to 4 ports on the switch, where it's aggregated to 40GbE. UniFi in particular.

I saw the post on STH main, about the advise to use the QSFP to SFP adapter and that in general it's not working like this. But there are some switches that can "re-aggregate" the QSFP with 5 ports. I wonder if UniFi might be one of these.

Thank you
No, those breakout cables usually only work with the QSFP end connected to the switch. I guess there might be some exception somewhere, but I personally haven't seen anyone point to one. :)
 
  • Like
Reactions: Samir

Cruzader

Well-Known Member
Jan 1, 2021
525
528
93
No, those breakout cables usually only work with the QSFP end connected to the switch. I guess there might be some exception somewhere, but I personally haven't seen anyone point to one. :)
Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.
 
Last edited:
  • Like
Reactions: Samir and blunden

blunden

Active Member
Nov 29, 2019
373
105
43
Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.
Interesting. :) It seems to be the exception though, not the rule.
 
  • Like
Reactions: Samir

applepi

Member
Jun 15, 2013
84
66
18
Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.
Yeah I've never seen this from anyone else outside of FPGA NICs. It requires considerations on the FIFO and and MACs that may not have been done at tape out so you generally will only see 4x 10gbps or 40gbps ports. Same deal exists on 100gbps / 25gbps NICs. Usually they are either ganged or not. And when they are broken out they generally need to be the same speed. Though I have a boardcom quad 25gbps NIC that will do both 10gbps and 25gbps at the same time. This requires external clock generation and die area to achieve.
 
  • Like
Reactions: Samir and blunden