(SOLVED) Windows - TrueNAS 40GbE NIC connection switches to 1GbE NIC after hibernation - but 40GbE still pingable

[cudak888]

I have a TrueNAS Scale (home) server directly attached to my Windows workstation with a pair of 40GbE NICs (Chelsio T580-LP-CR in the TrueNAS box, Mellanox Connect-X3 MCX354A-FCBT on the Windows machine) connected together with a QSFP+ cable. Each have permanently assigned IPs pointing to each other.

A 1GbE connection also exists between my home network, an un-managed switch, and a 1GbE connection to the motherboard of the server for GUI access and the like.

The server has SMB configured on it with two pools on it; nothing fancy.

For the most part, this combination has behaved itself, but I've been running into an issue where - usually after hibernation, but sometimes during normal use - the 40GbE connection stops being used, and defaults to the 1GbE connection.

The odd part about this is that the 40GbE connection is neither dropped or lost when this happens. I can still ping each other's 40GbE NIC by IP without any delay or packet drops. However, if I keep the Status window (in Windows) open for either the 1GbE connection or the 40GbE connection, any activity between the workstation and the server clearly RX/TX'es itself over the 1GbE connection.

Disabling and restarting the NIC in the Windows box (via the Windows GUI; apologies if this offends anyone who would CLI ipconfig /renew and /release as the first step instead) has no effect, and the NIC is also seen in the TrueNAS GUI as connected (but not actively sharing packets unless pinged).

Leaving the Windows box on and restarting the server always works to renew the 40GbE connection, but this is (obviously) becoming a massive pain in the butt.

I'm at a loss at how to approach troubleshooting this and would greatly appreciate any advice what to do and where to look.

-Kurt

 

[Sean Ho]

DNS/NetBIOS? How is the Windows workstation accessing the TNS server? Via SMB by NetBIOS name? If you mount the SMB share via IP, does it work and use the 40GbE interface?

 

[cudak888]

DNS with NetBIOS enabled; the mapped share is by NetBIOS name (at present; there have been times I've had it mapped by IP).

I'll try mapping it again by IP to see what the results are.

-Kurt

 

[klui]

Make sure 40G is your default route on both systems. On Windows, also change interface metric so 40G has lower number.

 

[cudak888]

Thank you, @klui - though I have my answer after leaving the computer hibernating for about two hours: The the IP-mapped drive remains at full 40GbE, while the NetBIOS-mapped drive has fallen back to the 1GbE connection.

Any recommendation for the interface metric # of the 40GbE vs the 1GbE?

Delighted that it was that easy and straightforward a fix too. Off to remap them again.

-Kurt

 

[klui]

Up to you. You can use something in the 200s for your gig interfaces and 100s for 40G. Set lower values for faster interfaces. Windows should prioritize based on link speed but if you have interfaces of the same speed that's when manually setting the metric will help.

 

[Sean Ho]

If your 40GbE is a direct-connect, don't point your default route to it, otherwise traffic will just bounce between the two hosts. Leave your default route on the gigabit link that goes to your switch and router, and use the 40GbE only for traffic that needs it between the two hosts. You can add a static hosts entry if you like. Remember to use a different subnet for the 40GbE link.

 

[cudak888]

Global config on the TrueNAS server does have the default route/gateway pointed to the 1GbE network, but the gateway on the Windows box for the 40GbE link is pointed specifically to the subnet of the 40GbE link.

Would you recommend adding the specific gateway on TrueNAS as well?

-Kurt

P.S.: I set the interface metrics per Klui's advice. No change in operation so far, but no apparent downsides either, so calling it a win!