Mellanox ConnectX-3 Pro EN (MCX314A-BCCT)

[gb00s]

Not sure if these Pro versions are still a thing but pricing is going wild. If you take more than 10x seller accepts 15US$ Hope they are genuine. If I'm not mistaken, these can do 56Gbps on IB.

Mellanox ConnectX-3 Pro EN Netzwerkkarte (MCX314A-BCCT) 7290107197144 | eBay

 

[TonyStew]

I actually just bought a couple from this seller on Ebay for $16 a piece with free 2-4 day shipping (or high-profile). Looking at the datasheet for it vs for non-pro connectx-3 cards it looks like the pro versions only support up to 40gb ethernet, and no version of infiniband. Running MST status on my cards doesn't give a link type setting specifying ib or eth, which supports that. What sets the pros apart is support for RoCE v2 which may be nice to have depending on your setup and what you're doing (clustered VMs in particular).

Something I noticed is my cards came with a newer firmware version than I've seen documented anywhere, 2.43.7010 from 6-5-2019. I need to bump one of the firmware threads and see if that's anything special.

 

[klui]

The manual you linked states the 314A-BCCT supports 40 and 56 GbE (page 10).

The model on eBay is ethernet only while 353A-FCCT and 354A-FCCT per the datasheet available at https://download.datasheets.com/pdf...09/mlnx_/manual/17pb_nectx-3_pro_card_vpi.pdf are VPI and will support Infiniband.

 

[Kahooli]

I actually just bought a couple from this seller on Ebay for $16 a piece with free 2-4 day shipping (or high-profile). Looking at the datasheet for it vs for non-pro connectx-3 cards it looks like the pro versions only support up to 40gb ethernet, and no version of infiniband. Running MST status on my cards doesn't give a link type setting specifying ib or eth, which supports that. What sets the pros apart is support for RoCE v2 which may be nice to have depending on your setup and what you're doing (clustered VMs in particular).

Something I noticed is my cards came with a newer firmware version than I've seen documented anywhere, 2.43.7010 from 6-5-2019. I need to bump one of the firmware threads and see if that's anything special.

Pretty sure you can crossflash to any of the pro chip based firmwares and get the functionality you want.

 

[86turbodsl]

Cool, but just wondering what use a home lab has for Infiniband? I deal with this sometimes on clusters, which makes sense, but in a home?

 

[bwahaha]

Cool, but just wondering what use a home lab has for Infiniband? I deal with this sometimes on clusters, which makes sense, but in a home?

Don't ask why, ask why not. Most things here make more sense that way

 

[krista]

Cool, but just wondering what use a home lab has for Infiniband? I deal with this sometimes on clusters, which makes sense, but in a home?

heh, my home lab had fdr-ib on the rack and ran to a couple workstations in other rooms. then i had to move after a couple of decades there.

my apartment doesn't even have ethernet... so i'm going into the crawlspace and adding ethernet... as well as a couple mtp/mpo runs between the office/workshop and my bedroom/studio/workstation.

why?

i deal with large files of various types, from xml to long multichannel audio, 8k video, and other oddities.

plus i can treat my 4 server ”hyperconverged minicluster” as an extremely fast drive, especially using a ramcache on a tiered sas hdd raid with ssd for hot files.

this makes compiling big projects or building unreal 5 projects actually kinda spreedy.

---

- i'm also experimenting with rdma, spatial data, and distributed manipulation thereof (say, using multiple servers to run the physics sim for a very large world multiple local player vr thingy).

- on the back burner is using ib as a way to redirect the gpu framebuffer for a displayport based vr headset from the rendering pc/gpu to a different one. this has a lot of potential applications.

---

plus it's pretty entertaining watching someone's jaw drop when hucking 100gb files around in seconds...

 

[maxermaxer]

I know this is probably a dumb question: is there any difference between cables for 10G and 40G? I am using 10G network but never used 40G before.

 

[Nipp]

One more dumb question - can it be used with 10GbE switch? Like a reversed connection - 4x10 breakout cable from NIC to 4 ports on the switch, where it's aggregated to 40GbE. UniFi in particular.

I saw the post on STH main, about the advise to use the QSFP to SFP adapter and that in general it's not working like this. But there are some switches that can "re-aggregate" the QSFP with 5 ports. I wonder if UniFi might be one of these.

Thank you

 

[msg7086]

I know this is probably a dumb question: is there any difference between cables for 10G and 40G? I am using 10G network but never used 40G before.

Why Use QSFP-40G-UNIV Transceiver for 40G Network? | FS Community This should explain a lot. Previously you'd need to run a MPO 4 pairs cable but you can use a UNIV with a regular 1 pair fiber.

 

[blunden]

One more dumb question - can it be used with 10GbE switch? Like a reversed connection - 4x10 breakout cable from NIC to 4 ports on the switch, where it's aggregated to 40GbE. UniFi in particular.

I saw the post on STH main, about the advise to use the QSFP to SFP adapter and that in general it's not working like this. But there are some switches that can "re-aggregate" the QSFP with 5 ports. I wonder if UniFi might be one of these.

Thank you

No, those breakout cables usually only work with the QSFP end connected to the switch. I guess there might be some exception somewhere, but I personally haven't seen anyone point to one.

 

[Cruzader]

No, those breakout cables usually only work with the QSFP end connected to the switch. I guess there might be some exception somewhere, but I personally haven't seen anyone point to one.

Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.

 

[blunden]

Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.

Interesting. It seems to be the exception though, not the rule.

 

[applepi]

Not sure how common it is, but i can point to an example tho.
The vic1387 2x 40gbe cisco has for m4/m5 servers support using breakout cables.

links nicely as 4x10gbe with a regular cable also if enabling it on both ends.

Yeah I've never seen this from anyone else outside of FPGA NICs. It requires considerations on the FIFO and and MACs that may not have been done at tape out so you generally will only see 4x 10gbps or 40gbps ports. Same deal exists on 100gbps / 25gbps NICs. Usually they are either ganged or not. And when they are broken out they generally need to be the same speed. Though I have a boardcom quad 25gbps NIC that will do both 10gbps and 25gbps at the same time. This requires external clock generation and die area to achieve.

 

[tuntin]

Hello,

Do you know if this cable 90Y3838 will work with this card? Thanks in advance.