Mellanox ConnectX-3 Pro EN (MCX314A-BCCT)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
T

TonyStew

New Member
Aug 17, 2023
1
0
1
I actually just bought a couple from this seller on Ebay for $16 a piece with free 2-4 day shipping (or high-profile). Looking at the datasheet for it vs for non-pro connectx-3 cards it looks like the pro versions only support up to 40gb ethernet, and no version of infiniband. Running MST status on my cards doesn't give a link type setting specifying ib or eth, which supports that. What sets the pros apart is support for RoCE v2 which may be nice to have depending on your setup and what you're doing (clustered VMs in particular).

Something I noticed is my cards came with a newer firmware version than I've seen documented anywhere, 2.43.7010 from 6-5-2019. I need to bump one of the firmware threads and see if that's anything special.
 
Last edited:
K

Kahooli

Member
Dec 3, 2022
54
36
18
TonyStew said:
I actually just bought a couple from this seller on Ebay for $16 a piece with free 2-4 day shipping (or high-profile). Looking at the datasheet for it vs for non-pro connectx-3 cards it looks like the pro versions only support up to 40gb ethernet, and no version of infiniband. Running MST status on my cards doesn't give a link type setting specifying ib or eth, which supports that. What sets the pros apart is support for RoCE v2 which may be nice to have depending on your setup and what you're doing (clustered VMs in particular).

Something I noticed is my cards came with a newer firmware version than I've seen documented anywhere, 2.43.7010 from 6-5-2019. I need to bump one of the firmware threads and see if that's anything special.
Click to expand...
Pretty sure you can crossflash to any of the pro chip based firmwares and get the functionality you want.
 
8

86turbodsl

Active Member
Feb 24, 2020
118
36
28
Cool, but just wondering what use a home lab has for Infiniband? I deal with this sometimes on clusters, which makes sense, but in a home?
 
K

krista

Member
Dec 22, 2015
30
12
8
45
86turbodsl said:
Cool, but just wondering what use a home lab has for Infiniband? I deal with this sometimes on clusters, which makes sense, but in a home?
Click to expand...
heh, my home lab had fdr-ib on the rack and ran to a couple workstations in other rooms. then i had to move after a couple of decades there.

my apartment doesn't even have ethernet... so i'm going into the crawlspace and adding ethernet... as well as a couple mtp/mpo runs between the office/workshop and my bedroom/studio/workstation.

why?

i deal with large files of various types, from xml to long multichannel audio, 8k video, and other oddities.

plus i can treat my 4 server ”hyperconverged minicluster” as an extremely fast drive, especially using a ramcache on a tiered sas hdd raid with ssd for hot files.

this makes compiling big projects or building unreal 5 projects actually kinda spreedy.

---

- i'm also experimenting with rdma, spatial data, and distributed manipulation thereof (say, using multiple servers to run the physics sim for a very large world multiple local player vr thingy).

- on the back burner is using ib as a way to redirect the gpu framebuffer for a displayport based vr headset from the rendering pc/gpu to a different one. this has a lot of potential applications.

---

plus it's pretty entertaining watching someone's jaw drop when hucking 100gb files around in seconds...
 
  • Like
  • Love
Reactions: e97, 86turbodsl and Samir
maxermaxer

maxermaxer

Active Member
Oct 28, 2016
329
61
28
50
I know this is probably a dumb question: is there any difference between cables for 10G and 40G? I am using 10G network but never used 40G before. :)
 
  • Like
Reactions: Samir
N

Nipp

New Member
May 20, 2022
16
5
3
Haymarket, VA
One more dumb question - can it be used with 10GbE switch? Like a reversed connection - 4x10 breakout cable from NIC to 4 ports on the switch, where it's aggregated to 40GbE. UniFi in particular.

I saw the post on STH main, about the advise to use the QSFP to SFP adapter and that in general it's not working like this. But there are some switches that can "re-aggregate" the QSFP with 5 ports. I wonder if UniFi might be one of these.

Thank you
 
Last edited:
  • Like
Reactions: Samir
B

blunden

Well-Known Member
Nov 29, 2019
1,071
373
83
Nipp said:
One more dumb question - can it be used with 10GbE switch? Like a reversed connection - 4x10 breakout cable from NIC to 4 ports on the switch, where it's aggregated to 40GbE. UniFi in particular.

I saw the post on STH main, about the advise to use the QSFP to SFP adapter and that in general it's not working like this. But there are some switches that can "re-aggregate" the QSFP with 5 ports. I wonder if UniFi might be one of these.

Thank you
Click to expand...
No, those breakout cables usually only work with the QSFP end connected to the switch. I guess there might be some exception somewhere, but I personally haven't seen anyone point to one. :)
 
  • Like
Reactions: Samir
C

Cruzader

Well-Known Member
Jan 1, 2021
924
910
93
blunden said:
No, those breakout cables usually only work with the QSFP end connected to the switch. I guess there might be some exception somewhere, but I personally haven't seen anyone point to one. :)
Click to expand...
Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.
 
Last edited:
  • Like
Reactions: Samir and blunden
B

blunden

Well-Known Member
Nov 29, 2019
1,071
373
83
Cruzader said:
Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.
Click to expand...
Interesting. :) It seems to be the exception though, not the rule.
 
  • Like
Reactions: Samir
A

applepi

Member
Jun 15, 2013
88
68
18
Cruzader said:
Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.
Click to expand...
Yeah I've never seen this from anyone else outside of FPGA NICs. It requires considerations on the FIFO and and MACs that may not have been done at tape out so you generally will only see 4x 10gbps or 40gbps ports. Same deal exists on 100gbps / 25gbps NICs. Usually they are either ganged or not. And when they are broken out they generally need to be the same speed. Though I have a boardcom quad 25gbps NIC that will do both 10gbps and 25gbps at the same time. This requires external clock generation and die area to achieve.
 
  • Like
Reactions: mach3.2, Samir and blunden
H

homeuser7665

New Member
Oct 8, 2024
3
2
3
Cruzader said:
Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.
Click to expand...
I was just directed to this post as I had a similiar question elsewhere, the talk in here about FIFO and MACs at tape out is a bit beyond my comprehension.

What I was considering attempting was using this Mellanox QSFP 40Gb NIC in a hypervisor (proxmox) effectively acting as a switch (proxmox linux bridge) and running the 4x10Gb breakout to different machines equipped with mellanox CX312B (10GB SFP+) .
Is that not a possibe useage case when breakout cable originates in MCX314A-BCCT and terminates into muitiple MCX312B-XCCT?
 
  • Like
Reactions: Samir
L

luckylinux

Well-Known Member
Mar 18, 2012
1,444
439
83
homeuser7665 said:
I was just directed to this post as I had a similiar question elsewhere, the talk in here about FIFO and MACs at tape out is a bit beyond my comprehension.

What I was considering attempting was using this Mellanox QSFP 40Gb NIC in a hypervisor (proxmox) effectively acting as a switch (proxmox linux bridge) and running the 4x10Gb breakout to different machines equipped with mellanox CX312B (10GB SFP+) .
Is that not a possibe useage case when breakout cable originates in MCX314A-BCCT and terminates into muitiple MCX312B-XCCT?
Click to expand...
Pretty sure @Cruzader wrote in another Thread that is NOT supported since most NICs CANNOT act like a Switch (when "broken" Out).
 
  • Like
Reactions: Samir
H

homeuser7665

New Member
Oct 8, 2024
3
2
3
Hi do you have a link to that thread please or title of it so that I can dig into it a bit. Im not particularly knowledgeable in these thigns but I know that a standard NON-QSFP regular NIC can be used as part of a software switch (linux bridge) in proxmox. from what I understand its handled by a kernel module

So Im interested to understand what the actual mechanism is that allows a QSFP to break out into 4x10Gb. How the QSFP port presents itself to Proxmox when broken out and how the actual 'breaking-out' happens in terms of network layers. Im not even sure what exact principles/questions to ask to get to the bottom of this.

I have to stress again my knowledge is very limited, so Im just fumbling in the dark pulling at threads until something unravels and leads me to the next thread, until I get to where I need if that makes any sense.

Or woudl you advice its better to start a fresh topic specifically for this question as I dont see any similar post?
 
L

luckylinux

Well-Known Member
Mar 18, 2012
1,444
439
83
homeuser7665 said:
Hi do you have a link to that thread please or title of it so that I can dig into it a bit. Im not particularly knowledgeable in these thigns but I know that a standard NON-QSFP regular NIC can be used as part of a software switch (linux bridge) in proxmox. from what I understand its handled by a kernel module

So Im interested to understand what the actual mechanism is that allows a QSFP to break out into 4x10Gb. How the QSFP port presents itself to Proxmox when broken out and how the actual 'breaking-out' happens in terms of network layers. Im not even sure what exact principles/questions to ask to get to the bottom of this.

I have to stress again my knowledge is very limited, so Im just fumbling in the dark pulling at threads until something unravels and leads me to the next thread, until I get to where I need if that makes any sense.

Or woudl you advice its better to start a fresh topic specifically for this question as I dont see any similar post?
Click to expand...
Search Function and Previous Posts/Threads is really NOT working well on this Forum, I can barely find my own Posts :rolleyes: .

Looking into the Email is a bit better.

Here it is:

 
  • Like
Reactions: homeuser7665
H

homeuser7665

New Member
Oct 8, 2024
3
2
3
luckylinux said:
Search Function and Previous Posts/Threads is really NOT working well on this Forum, I can barely find my own Posts :rolleyes: .

Looking into the Email is a bit better.

Here it is:

Click to expand...
Really appreaciate you digging that out. Seems like I will just have to bite the bullet and upgrade switch for a bit beyond typical chinese white-box switch budget when I eventually absolutely cannot make do with the limited number of SFP+ I currently have.

Youve been a help, sometimes its more humane to thoroughly crush vain hope, than let a creature miserably wander round and round in circles in futile attempts to never find the thing its looking for, looooool. Cheers X-D
 
  • Like
Reactions: luckylinux
You must log in or register to reply here.
Top Bottom