HP 649281-B21 ConnectX-3 VPI MCX354A-FCBT $25 shipped Ebay

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
Possibly dumb question: Is a QSFP cable a QSFP cable, or are there differences between, say, a "3 GB/s" QSFP cable from a SAN (specifically, EMC Amphenol 038-003-703 85" 2.2M 3GB/s QSFP to QSFP Male to Male Cable Black | eBay) and one from Mellanox like zer0sum linked?
There are different qsfp revisions with different bandwidths:
qsfp (up to 4x 1gbit/s ethernet, ddr infiniband)
qsfp+ (up to 40gbit/s ethernet, qdr infiniband)
qsfp14 (up to 56gbit/s proprietary ethernet with mellanox nics, fdr infiniband)
qsfp28 (up to 100gbit/s ethernet, edr infiniband, intel omni-path)
 

dontwanna

Member
Dec 22, 2016
90
20
8
There are different qsfp revisions with different bandwidths:
qsfp (up to 4x 1gbit/s ethernet, ddr infiniband)
qsfp+ (up to 40gbit/s ethernet, qdr infiniband)
Now I'm confused. :) Those $8 cables say "qsfp" with no "+".. will they work @ 40 gb/s with connectx-3 cards? Would there be any practical difference between qsfp and qsfp+ cables for connectx-3?
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Now I'm confused. :) Those $8 cables say "qsfp" with no "+".. will they work @ 40 gb/s with connectx-3 cards? Would there be any practical difference between qsfp and qsfp+ cables for connectx-3?
Well...when I said "works perfectly", I didn't pull it out of my rear end... :) I have the exact cables connected to the exact same card to the 40gb ports on a 6610 and they "work perfectly" :)
 
  • Like
Reactions: dontwanna

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
Those $8 cables say "qsfp" with no "+"
Most sellers on ebay don't write the correct revision, I would recommend verifying over part numbers.

"External sas" cables are qsfp14 cables, but they are backward compatible and can be used for 40 gbe.
 
  • Like
Reactions: dontwanna

zer0sum

Well-Known Member
Mar 8, 2013
849
473
63
Well...it seems like maybe my cable isn't the compatible or something as I can't get a link up even with both cards flashed to the most recent firmware etc. :(

New cable coming this week and I'll switch it out Friday and hopefully that's the only issue!

Code:
[root@DarkESXi:/opt/mellanox/bin] esxcli network nic get -n vmnic3                                                                               
   Advertised Auto Negotiation: true
   Advertised Link Modes: 1000None/Half, 1000None/Full, 10000None/Half, 10000None/Full, 40000None/Half, 40000None/Full, Auto
   Auto Negotiation: false
   Cable Type:
   Current Message Level: -1
   Driver Info:
         Bus Info: 0000:03:00:0
         Driver: nmlx4_en
         Firmware Version: 2.42.5000
         Version: 3.16.11.6
   Link Detected: false
   Link Status: Down
   Name: vmnic3
   PHYAddress: 0
   Pause Autonegotiate: false
   Pause RX: true
   Pause TX: true
   Supported Ports:
   Supports Auto Negotiation: true
   Supports Pause: true
   Supports Wakeon: false
   Transceiver: internal
   Virtual Address: 00:50:56:50:80:b3
   Wakeon: None
 

dontwanna

Member
Dec 22, 2016
90
20
8
Well...it seems like maybe my cable isn't the compatible or something as I can't get a link up even with both cards flashed to the most recent firmware etc.
I'm a total noob when it comes to Infiniband stuff, and just recently tried to play with this tech for the 1st time myself, so maybe it's obvious, but still - is your subnet manager up and running? It wasn't obvious to me when I tried these connectx-3 cards and the link wouldn't go up by itself. Apparently you need to run subnet manager software somewhere (on the clients, or the switch, for example), and since you didn't mention it in your post, I suppose there's a chance you're not running any.
 

zer0sum

Well-Known Member
Mar 8, 2013
849
473
63
I'm a total noob when it comes to Infiniband stuff, and just recently tried to play with this tech for the 1st time myself, so maybe it's obvious, but still - is your subnet manager up and running? It wasn't obvious to me when I tried these connectx-3 cards and the link wouldn't go up by itself. Apparently you need to run subnet manager software somewhere (on the clients, or the switch, for example), and since you didn't mention it in your post, I suppose there's a chance you're not running any.
I switched my cards over to ethernet mode on ESXi and Windows...
/opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
Windows > Device Manager > system devices > Mellanox... > Port Protocol tab
 

epicurean

Active Member
Sep 29, 2014
785
80
28
What is the smallest and quietest switch that will enable 40 GB ethernet mode for these cards?
 

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
Not the smallest; the arista switches have a command to set the fans to 30% which is enough to cool the asic and other components.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
I would like a small switch, that consumes as little power as possible and yet enable 40gb ethernet. Any thing cheaper than Arista?
 

Churchill

Admiral
Jan 6, 2016
838
213
43
I would like a small switch, that consumes as little power as possible and yet enable 40gb ethernet. Any thing cheaper than Arista?
These do not exist at this time. In another 2-3 years yes. As of right now the sheer amount of power 40gb needs is the bottleneck. Mikrotek has switches that are 'small' but still take power to use.

If you want to just skip the switch you can do Point to point cabling and connect your servers together over SAS cables instead. Same speed, skip the switch.
 
  • Like
Reactions: dontwanna

epicurean

Active Member
Sep 29, 2014
785
80
28
If you want to just skip the switch you can do Point to point cabling and connect your servers together over SAS cables instead. Same speed said:
Is it difficult to configure for point to point cabling between 3 esxi servers?
 

Dev_Mgr

Active Member
Sep 20, 2014
133
48
28
Texas
That depends on what you're trying to achieve. Anything where one host may need to communicate with both other hosts (at the same time or at different times) won't work and you'd have to have a switch with at least 3 40Gbit ports.

What do you plan to run that needs 40Gbit ports and couldn't work with dual (or maybe quad) 10Gbit ports? 10Gbit connectivity gives you a lot more affordable (and lower noise due to lower power requirements) options.
 
  • Like
Reactions: Churchill

Churchill

Admiral
Jan 6, 2016
838
213
43
You are right. 10GB is plenty.

If you need all 3 ESXI hosts to talk to each other at the same time over the storage network? Yes that would be a problem. spacedinvader one does a good job of talking how to do it with UnRAID but I believe your setup would be similar: