HELP! Tested several Dell Mellanox CX354A adapters and cannot get them to connect with anything!

[waqqas31]

Hello,

This is my first post at STH and I apologize that I am asking for help right out-of-the-gate and before I can help anyone else out, either.

I am upgrading my current lab network from a 10 gig backbone to an 40 gig backbone.

This is a very quick rundown of the equipment in my possession:
1. Cisco Nexus 3132Q-40GE
2. Dell ConnectX-3 CX354A PCIe 3.0 dual QSFP port cards (6RKNM)
3. Dell QSFP-40G-SR4 modules (07TCDN)
4. Genuine Cisco QSFP-40G-SR4 modules
5. Genuine Cisco QSFP-40G-SR4-BD modules
6. 12-fiber Straight Type A MPO/MTP cables (see chart below)
7. 12-fiber Crossover Type B MPO-MTP cables (see chart below)
8. Genuine Cisco DAC (Twinax) QSFP-H40G-CU5M cables
9. OM4 MM LC/LC Fiber Patch Cords

Disclaimer: For all of the following tests, the firmware and drivers used were the latest for the switch, NICs and Windows 2012 R2. I know the QSFP modules and (at least) the crossover MPO and LC cables work because switch-to-switch communication using them works just fine.

The following scenarios (A-F) yielded ZERO results. No lights turned on, and always "Network Cable Unplugged" within Windows.

A. Cisco 3132Q with Dell QSFP --> Straight MPO cable --> Dell QSFP with Dell CX354A
B. Cisco 3132Q with Cisco SR4 --> Straight MPO cable --> Cisco SR4 with Dell CX354A
C. Cisco 3132Q with Cisco SR4-BD --> Straight LC cable --> Cisco SR4-BD with Dell CX354A
D. Cisco 3132Q with Cisco QSFP-H40G-CU5M --> Cisco QSFP-H40G-CU5M with Dell CX354A
E. Server with Dell CX354A with Cisco QSFP-H40G-CU5M --> Cisco QSFP-H40G-CU5M with Dell CX354A
F. Server with Dell CX354A and Dell QSFP --> Straight MPO cable --> Dell QSFP with Dell CX354A

In the next two scenarios (G-H), one light did turn on, but we were still unable to do anything ("network cable unplugged")
G. Server with Dell CX354A and Dell QSFP --> Crossover MPO cable --> Dell QSFP with Dell CX354A
H. Server with Dell CX354A and Cisco SR4 --> Crossover MPO cable --> Cisco SR4 with Dell CX354A

So I am at a loss as to what these Dell cards need to start working. Whatever input anyone can provide would be greatly welcome.

Thanks in advance!

 

[cliffr]

@waqqas31 - the cards are VPI so you need to make sure they are set to Ethernet mode since it sounds like you're not using IB.

What I'd do to troubleshoot is start on two cards and no switch with appropriate optics/ cables get links up. Then start on the switches. Fewer things to go wrong if you do that. You'll also know you have a known good link on one end.

 

[waqqas31]

@cliffr

I had no idea you had to set which mode/media you wanted to use. I am definitely trying to use the Ethernet "side" of the card. Where can I specify this setting?

Thanks!

 

[Rand__]

iirc in the device manager, hardware properties.

Alternatively you could simply start the subnet manager in windows for testing

 

[waqqas31]

iirc in the device manager, hardware properties.

Alternatively you could simply start the subnet manager in windows for testing

@Rand__:

Do you mean that if I am able to start Subnet Manager that directly implies I'm in IB mode?

I think I may have figured out why I've been pulling my hair out this past week.

I have been installing the official WinOF v5.35 package since day 1, but on Win2012 R2 (where the provided options appear to be limited).

First off, for some reason, the ports are automatically configured as "Mellanox ConnectX-3 IPoIB..."

Even though I tried to change the driver before, today it worked when I selected "Mellanox ConnectX-3 Ethernet adapter (Mellanox)" from the list and voila, it came to life. So, there's no Protocol tab in Hardware Properties, and this seems to be the only way of switching to Ethernet.

Second, after changing the driver from IPoIB to Ethernet, the Teaming tab has also disappeared. I'm guessing that Mellanox wants us to use the built-in teaming functionality in Win2012 R2, but I am waiting for my callback from Tech Support to accept that explanation.

Now, the golden question is, have I indeed found the best solution in Windows Server 2012 R2 or am I still doing something wrong?

Thanks everyone for your help and inspiration so far

 

[waqqas31]

Ok, now I feel like a moron.

There is a "Mellanox Connect-X (MT04099) Network Adapter" that appears under System Devices in Device Manager, which is where the "Port Protocol" tab is located and allows for simple switching to Ethernet or IB functionality.

Doh!

 

[Rand__]

Glad you got it to work

Infiniband requires a subnet manager. Open SM is included in the Mlx software package. If you start it you should see link between your two boxes and should be able to ibping the other device.

And regarding the 'best solution' - depends Ethernet will be simple but IB has its advantages as well (latency etc), so based on your target OS's, driver support, switch capabilities etc you will need to choose between Ethernet, IB or even IPoIB

 

[waqqas31]

Many thanks, @Rand__ and @cliffr

 

[Blitzthis]

Zombie thread here...but THANKS waqqas31! I also felt like a moron trying to find that damn port protocol tab, because I kept looking for the option under network connection properties inside of Device Manager. Finally running now.