Mellanox SX1012 breakout not behaving

[bob_dvb]

Hi,

I have a Mellanox SX1012 which I got on a good deal, it works fine from one of the ports with a DAC breakout cable, but then I put the breakout into another port and it doesn't get a link.

Are breakouts restricted to certain ports?

Bob

 

[NablaSquaredG]

Are breakouts restricted to certain ports?

Nope, only on the 36 Port models.

2.6.1.1 SX1012/SX1012X In this switch, all ports can be split to 4x10GbE, no limitation.

according to the

Mellanox SwitchX and SwitchX®-2 1U Switch Systems Hardware User Manual Models: SX1012/SX1012X/SX1016/SX1024/SX1036/SX1400 Rev 1.4

 

[bob_dvb]

Nope, only on the 36 Port models.

2.6.1.1 SX1012/SX1012X In this switch, all ports can be split to 4x10GbE, no limitation.

according to the

Mellanox SwitchX and SwitchX®-2 1U Switch Systems Hardware User Manual Models: SX1012/SX1012X/SX1016/SX1024/SX1036/SX1400 Rev 1.4

Thanks, I thought this was the case.

Now I am confused as to why it's not behaving...

 

[Rand__]

I assume u split /configured the ports identically?

From which to which did u move? Is a third/fourth port working?

 

[i386]

Is that cable from fs.com?

 

[bob_dvb]

Okay, so I have half fixed the issue.

1. I changed port, split that port and now it works when I plug that into another switch and other devices.
2. My Epyc server does NOT like it and won't link!

I may have misconfigured the split before, but on my other switch I did need to set it to not auto-negotiate, I am now confident it's not really the switch that's having an issue. FYI these are Molex DACs, I have two, one is working fine with a SAN and the other is NOW working fine with everything except the Epyc server.

So, I might need to create a new thread, but while I have your attention:

It's a known good ConnextX-4LX EN running in ETH mode, I have one port connected at 25G and previously had the other port connected fine to one of those little Mikrotik CRS305 4+1 switches via an single lane 10G IOPlex DAC. The 25g link from the server is to my NAS which is also running a QNAP CX4 (QXG-25G2SF-CX4) and having no issues negotiating with the same DAC and switch on its other port!

I've set ethtool -s <dev> 10000 autoneg off and that doesn't fix it?! Nothing else wants flow control, or any other settings other than the switch which wanted me to turn off autoneg and set 10G.

> Speed: Unknown!
> Link detected: no (Cable issue, unsupported cable)

I'm still confused, and I'd suspect the DAC, except it's working fine with another non-breakout DAC to another off-brand switch. I haven't yet tried a single lane 10G DAC connected to a (Mellanox) QSFP to SFP+ adapter, that's my next diagnostic step. It would be weird if that works.

 

[bob_dvb]

Checked two other good DACs, into the QSFP to SFP+ adapter and that didn't work either!