Mellanox IS5030 Speed Issues: Only DDR?

[alltheasimov]

Hi,

I'm having speed issues that I've isolated to my IS5030 switch.

System: I have two Infiniband switches: 1. A Mellanox IS50XX with 36 ports enabled and the FrabricIT internal subnet manager running, making it an IS5030, with latest firmware (IBM P/N: 98Y3756) 2. A Sun 36 port QDR Infiniband Switch, internal subnet manager. I also have 3 types of HCAs: Sun 375-3696 X4242A (rebranded Mellanox ConnectX-2 card), HP 544FLR-QDR (based on Mellanox ConnectX-3), and Intel/QLogic QLE7340. I haven't figured out how to update the firmware for the Sun products. The HP HCAs have the latest firmware. The QLE7340's don't use firmware. Running CentOS 7 on all nodes.

I've tried the following tests (ibstat):

• Two HP HCAs, back-to-back, opensm: 40 Gb/s
• Sun HCA, HP HCA, back-to-back, opensm: 40 Gb/s
• Two QLE7340, back-to-back, opensm: 40 Gb/s
• Sun and HP HCAs, Sun switch: 40 Gb/s
• Sun and HP HCAs, IS5030 switch: 20 Gb/s
• Two QLE7340, Sun switch: 10 Gb/s
• Two QLE7340, IS5030 switch: 20 Gb/s

I had a long discussion with an Intel rep, and it turns out that these two Infiniband switches are not compatible with the QLE7340's (some of the later Mellanox X series switches are, though), which is probably why they aren't reaching QDR speeds. So let's ignore those for now.

The really weird thing is that the Sun and HP HCAs should really be negotiating to 40 Gb/s with the IS5030, and I have no idea why they are not. Clearly the HCAs, software, and cables (I tested all of these) are fine since they work at 40 Gb/s back-to-back.

Any ideas?

 

[Rand__]

Have you tried forcing the switch port to QDR?

 

[alltheasimov]

How do I do that? I couldn't find anything about that in the manual or in the webUI.

 

[Rand__]

Not sure how it works on 5030 but on my 60xx its under Port Mmgt (here in EN mode)



And cli:
[standalone: master] (config interface ethernet 1/10) # speed ?
1G Port runs at 1Gbps
10G Port runs at 10Gbps
40G Port runs at 40Gbps
56G Port runs at 56Gbps

 

[alltheasimov]

Here's what I see.



It has the option for 10 Gbps under port configuration, with width 4x, which translates to 40Gbps. I can apply that configuration, but I can't seem to force it to 10 Gbps, though. In the "supported speeds" above, it says "2.5 / 5.0 Gbps" with no 10 Gbps.

Any ideas?

 

[i386]

What does it say in the "transceiver information" section about the cable with the DDR speed?

 

[Rand__]

I'd apply that config and then reconnect the client, maybe it then negotiates.
Or you can try to force on client side, but can't remember the command just now, sorry

 

[alltheasimov]

What does it say in the "transceiver information" section about the cable with the DDR speed?

The picture is of the active port, so it says the cable is capable of SDR, DDR, and QDR speeds.

 

[alltheasimov]

I'd apply that config and then reconnect the client, maybe it then negotiates.
Or you can try to force on client side, but can't remember the command just now, sorry

I tried applying the configuration, though I didn't try reconnecting. Just unplug and replug the cable?

Yeah, I was looking for some sort of command to force an HCA to negotiate to a certain speed, but I wasn't able to find anything.

 

[Rand__]

You could try to do admin port off /on on switch side. Not sure if replug helps.

 

[alltheasimov]

I tried saving the applied port configuration, and unplugging/plugging back in, but it still doesn't negotiate to 40.

The ib0 says 40Gb/s, which is odd.


ibstat and the port configuration are still stating 20 Gb/s.

 

[Rand__]

which os do you have client side?

and ib0 is the admin port is it not? why are you on that and not on a dataport?
Are you using IB to manage the switch?

Not sure if that affects anything tbh but...

 

[alltheasimov]

In the Fabric Inspector -> system information -> IB systems -> switch I see the following:



Why all the ports only up to 20 Gbps?

 

[alltheasimov]

which os do you have client side?

and ib0 is the admin port is it not? why are you on that and not on a dataport?
Are you using IB to manage the switch?

Not sure if that affects anything tbh but...

No, I'm using the management ethernet port to access the webui. I just showed that ib0 screenshot because it showed 40 Gbps. Not sure what it means.

 

[alltheasimov]

which os do you have client side?

CentOS 7.3

 

[alltheasimov]

Here are the details of the port that's connected. I checked the HCA, and it's capable of 10 4x, so 40 Gbps. This port just shows 5 4x, so 20 Gbps, which is the current link up speed.

 

[alltheasimov]

Are there IS5030 switches that are DDR? I've only ever seen them advertised as QDR.

 

[Rand__]

no I dont think so.
Will need to find that cli command client side ...
@_alex Do you remember it?

 

[_alex]

no I dont think so.
Will need to find that cli command client side ...
@_alex Do you remember it?

No, and snowboarding until tomorrow. But must be in ,the document' somewhere on the last few pages. Doing this in-band requires to know some things about the ports that can be seen with ibnetdiscover (needs sane fabric/ running sm)

 

[alltheasimov]

So there is a cli command to force the port speed?

But must be in ,the document' somewhere on the last few pages.

Would that be the mlnx OS manual?

I have the management port, so I can ssh into it and run cli commands.