Intermittent 4OGbit Link on Mikrotik CRS326-24S+2Q+

[mattlach]

Hi Everyone,

Just picked up a new CRS326-24S+2Q+ and I am struggling a little bit.

I'm using it as a layer2 switch + VLAN's with no routing, so I opted to boot and use it in SwOS mode.

I am using 3M Molex DAC cables to connect the Fortville (Intel XL710-QDA2) NIC's in my servers to the switch. I went with Molex cables as - at least the 10gbit SFP+ varieties have always worked in Mikrotik switches for me, and I needed them quickly, and could not find Mikrotik branded cables locally.

These immediately establish a link when connected directly, NIC to NIC, suggesting the cables are fine, but when plugging into the switch, there is no link light.

Last night, after repeatedly unplugging and re-plugging the cable, and waiting, eventually a link light appeared, and once it did, the connection was solid, and worked perfectly, until the server rebooted, at which point there was - again - no link light and no connection.

I have now been trying for 20-30 minutes, and have been unable to get another link light on the QSFP+ port. I have no idea what finally got it to link last night, but I cannot seem to replicate it again today.

It's notable that the switch detects that the DAC cable is connected, and displays the information on the Link page as follows:

QSFP+2.1    Molex     1002971301         215021347     22-05-30    3m copper

...it just doesn't result in a link.

Does anyone have any suggestions here?

1.) Do I potentially have a bad switch?

2.) Was I just silly to try to use a Molex DAC cable instead of the Mikrotik branded one?

3.) I have read that in some cases disabling auto-negotiation helps with the 40Gbit ports on Mikrotik hardware. Is this something I should try? If so, how do I do it? I clicked around the settings menu but if I disable auto-negotiation, I don't get a 40Gbit option, only 10M, 100M and 1G.

4.) Anything else I can try?


I appreciate any suggestions!

 

[RecursiveG]

My CRS309 do have a 40Gbps options (even though the hardware doesn't support that.)
Maybe try upgrading your system (and firmware)? I'm on RouterOS 7.11.2.

 

[mattlach]

My CRS309 do have a 40Gbps options (even though the hardware doesn't support that.)
Maybe try upgrading your system (and firmware)? I'm on RouterOS 7.11.2.
View attachment 34854

That screen dopes not look familiar? Are you in RouterOS or Swos? I always use Swos, as I do not intend to use my switch for anything Layer 3 related, just flat switching with some VLAN's.

Here is what I am seeing:

 

[RecursiveG]

That screen dopes not look familiar? Are you in RouterOS or Swos? I always use Swos, as I do not intend to use my switch for anything Layer 3 related, just flat switching with some VLAN's.

Here is what I am seeing:

View attachment 34855

I'm on RouterOS. I have no idea for SwOS. Maybe check the command line?

 

[mattlach]

I'm on RouterOS. I have no idea for SwOS. Maybe check the command line?

I'm not sure Swos has the ability to enter a command line. I will have to look at that.

 

[mattlach]

And just like that, after hours sitting plugged in without a link, suddenly there is a link

I guess if I just never reboot the server, everything will be fine

 

[mattventura]

SwOS unfortunately doesn't expose all of the SFP-related settings that ROS does, nor some of the logging.

My first step with these is to disable auto-negotiation.

 

[mattlach]

SwOS unfortunately doesn't expose all of the SFP-related settings that ROS does, nor some of the logging.

My first step with these is to disable auto-negotiation.

Interestingly enough version 2.16 of SwOS was released today, and while I still can't manually select 40G in the drop downs, 10G is now available. (they also added 2.5G in this update, I think.) I wonder if I am supposed to set 10G on each of the constituent QSFP links?



Either way, that doesn't work. Still no link.

I have decided to (at least for now) blame this on the DAC cable. I have ordered a couple of Mirkotik transceivers for the switch, a couple of 2M lengths of OM4 fiber, and a couple of Finisar transducers for the Intel NIC's and see if that solves it.

 

[mattlach]

So I wound up resolving this by running a short 2M fiber run instead of DAC cables.

The generic FS.com 40Gbit SR40 transceiver worked just fine on the Mikrotik end. I had ordered the Mikrotik variant, but they sent the generic instead telling me it was equivalent and compatible, and it worked. I used the Intel variant on the NIC side.

Between them I used 2M of MTP OM4 cabling, which had some sticker shock associated with it compared to LC connector cables.

Winds up being pricier, but at least there was an instant link and I didn't have to resort to turning off auto-negotiation or anything like that.

 

[ano]

mikrotik usually eats all codings, so should have worked
with that said, Ive had issues on the CCR2216, it wouldnt take my SMF LC QSFP28 optics, but took the SR bidi ones.

with DACs, more often than not you will have to force speeds. so that was more likely your issue

 

[mattlach]

mikrotik usually eats all codings, so should have worked
with that said, Ive had issues on the CCR2216, it wouldnt take my SMF LC QSFP28 optics, but took the SR bidi ones.

with DACs, more often than not you will have to force speeds. so that was more likely your issue

I tried forcing speeds, but there was no 40gbit setting. I could set 10gbit for each constituent QSFP channel but that didn't work.

THeir support suggested I try switching back to RouterOS and try forcing the speed there, which may have worked, but I didn't feel like running RouterOS. SwOS feels just right for my application, which is flat, layer2 with VLAN's and no routing.

I'm glad the transcievers did the trick!