40 GbE with Mellanox ConnectX-3 in Ethernet mode?

digity

Member
Jun 3, 2017
48
1
8
52
Can you get the 40 GbE speeds with a Mellanox ConnectX-3 VPI adapter in Ethernet mode or must it be in Infiniband mode? If the latter, can the required subnet manager software run in Linux or a docker?
 

i386

Well-Known Member
Mar 18, 2016
3,403
1,141
113
33
Germany
There are some connect-x 3 cards that are limited in firmware to 10gbe.
But because of the connect-x 3 cards using the same silicon it's possible to flash another firmware and get 40gbe.
 

digity

Member
Jun 3, 2017
48
1
8
52
To answer you. Yes they can and a number of us on here do just that. Both 40g eth point to point. And others with 40 gig eth switches.
My backbone and cluster network is all mellanox X-3. 40gig.
Thanks! Any recommendations on qsfp cables that never give you problems?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,353
1,805
113
CA
To answer you. Yes they can and a number of us on here do just that. Both 40g eth point to point. And others with 40 gig eth switches.
My backbone and cluster network is all mellanox X-3. 40gig.
Are you updating to latest firmware or a specific revision?
I need to update a handful now myself.. fun :D

Thanks!
 

AndreiL

New Member
Jun 30, 2019
23
9
3
There are some connect-x 3 cards that are limited in firmware to 10gbe.
But because of the connect-x 3 cards using the same silicon it's possible to flash another firmware and get 40gbe.
If you get 40Gb VPI cards than it's as easy as flipping a switch in fw using the mst tool to get 40GbE.

But, yes there are 10Gb VPI cards with QSFP cables, so be careful and read the specs carefully.
 
Apr 21, 2016
50
21
8
42
Can you get the 40 GbE speeds with a Mellanox ConnectX-3 VPI adapter in Ethernet mode or must it be in Infiniband mode? If the latter, can the required subnet manager software run in Linux or a docker?
If the card has the QSFP+ interface - advertised as FDR, it will also do 56GbE :)
If the card is only QSFP - advertised as FDR10, it will do only 40GbE.

For the SM, as long as it has an interface (real one) connected to the IB fabric, it'll work.

All of the revisions from A2 and up will do that.

Beware of the A1 revision, I have some inbound and I really hope they are not CX2's packaged in CX3 boards.
They did the same with CX2 A0 - those were CX1's.
I'll post more when I have them.