40 GbE with Mellanox ConnectX-3 in Ethernet mode?

[digity]

Can you get the 40 GbE speeds with a Mellanox ConnectX-3 VPI adapter in Ethernet mode or must it be in Infiniband mode? If the latter, can the required subnet manager software run in Linux or a docker?

 

[i386]

There are some connect-x 3 cards that are limited in firmware to 10gbe.
But because of the connect-x 3 cards using the same silicon it's possible to flash another firmware and get 40gbe.

 

[Terry Wallace]

To answer you. Yes they can and a number of us on here do just that. Both 40g eth point to point. And others with 40 gig eth switches.
My backbone and cluster network is all mellanox X-3. 40gig.

 

[digity]

To answer you. Yes they can and a number of us on here do just that. Both 40g eth point to point. And others with 40 gig eth switches.
My backbone and cluster network is all mellanox X-3. 40gig.

Thanks! Any recommendations on qsfp cables that never give you problems?

 

[Terry Wallace]

I buy theses
https://www.amazon.com/10Gtek-QSFP-H40G-CU1M-Direct-Attach-Passive-1-Meter/dp/B00SIV6PCU/

 

[T_Minus]

To answer you. Yes they can and a number of us on here do just that. Both 40g eth point to point. And others with 40 gig eth switches.
My backbone and cluster network is all mellanox X-3. 40gig.

Are you updating to latest firmware or a specific revision?
I need to update a handful now myself.. fun

Thanks!

 

[Terry Wallace]

Updating to latest firmware for that model.
Just grabbed from the mellanox site.

 

[AndreiL]

There are some connect-x 3 cards that are limited in firmware to 10gbe.
But because of the connect-x 3 cards using the same silicon it's possible to flash another firmware and get 40gbe.

If you get 40Gb VPI cards than it's as easy as flipping a switch in fw using the mst tool to get 40GbE.

But, yes there are 10Gb VPI cards with QSFP cables, so be careful and read the specs carefully.

 

[Gabriel Mateiciuc]

Can you get the 40 GbE speeds with a Mellanox ConnectX-3 VPI adapter in Ethernet mode or must it be in Infiniband mode? If the latter, can the required subnet manager software run in Linux or a docker?

If the card has the QSFP+ interface - advertised as FDR, it will also do 56GbE
If the card is only QSFP - advertised as FDR10, it will do only 40GbE.

For the SM, as long as it has an interface (real one) connected to the IB fabric, it'll work.

All of the revisions from A2 and up will do that.

Beware of the A1 revision, I have some inbound and I really hope they are not CX2's packaged in CX3 boards.
They did the same with CX2 A0 - those were CX1's.
I'll post more when I have them.