10gig - Random disconnects

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fmatthew5876

Member
Mar 20, 2017
80
18
8
38
I've noticed some random disconnects with my 10g network connection.

The setup is:

FreeBSD 11.0 Server <-> Chelsio T520-SO-CR 10g nic <-> FiberStore DAC cable <-> TP Link T1700G-28TQ

I see these in my dmesg on FreeBSD

cxl0: link state changed to DOWN
cxl0: link state changed to UP

I also see similar error logs on the switch:

3 2017-05-16 17:45:21 LinkScan level_5 Te1/0/25 changed state to up.
4 2017-05-16 17:45:20 LinkScan level_5 Te1/0/25 changed state to down.

It seems to happen at least once a day.

This was causing a big problem on my FreeBSD server as the disconnects were triggering the DHCP client. This caused my jails to lose their IP Address and then my network services started failing. I've worked around it by configuring the server with a static IP instead of using DHCP, but I'd still like to figure out whats going on.

Has anyone seen this kind of behavior before?
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Not had the pleasure on BSD, but I reckon it's a physical layer problem of some sort. Check for bad cable/ connection/ port or possibly a low level driver in BSD :)
 

fmatthew5876

Member
Mar 20, 2017
80
18
8
38
or possibly a low level driver in BSD :)
Chelsio is supposed to be one of the best supported nics for FreeBSD. Driver bug like this possible but unlikely.

Does your chassis provide enough airflow for cooling?
I'm using a SuperMicro SC743TQ case. My Chelsio NIC also has a noctua 40mm fan I added to it. The temperature sensor reads 47C in FreeBSD. I don't believe cooling is a problem here.

Can you make sure the DAC cable is OK?
The DAC cable is compatible with the NIC? How about change with another DAC cable?
The DAC cable is Fs.com TP-Link branded (whatever that means, the switch still logs a warning about using a third party cable). Other than these disconnects I have no issues with it. I can get full 10gig speeds on the cable.

I'm guessing its either the DAC cable or the switch. I only have these FS.com cables, so I can only try using a different cable for the same vendor.

I did see some disconnect log messages for other 1gig ports on the switch, but not as often as I see for the 10gig. I'm going to raise an issue with TP-Link tech support and see if its a switch problem.