Q: What Linux distributions support Intel's new X557-AT 10GBASE-T chip?

[trippehh]

I then decided to go with what has worked for others, and found this blog post by Major Hayden where he fired up Fedora 23 and got the 10Gb NICs to work. Downloaded Fedora Server 23 Beta but the result was the same. Module ixgbe loads but the link is not active, higher layers fails.

I have the same board and the 10Gbase-T ports are misbehaving on it as well. Its nearly impossible to negotiate a 10G link with the switch. It works by chance from time to time, otherwise it either fails completely (most likely) or drop to 1G (if lucky). This is with cat6a cabling of course. If it manages to get a 10G link up and running, it is completely stable until the link is taken down.

Locking both ends to 10G/Full makes it no better.

I've tested with the driver from kernel 4.2, as well as 4.3-rc7 ("4.0.1-k") and the one that is pending for inclusion in 4.4 ("4.2.1-k", from net-next, which has a bunch of X550 fixes), but no luck.

The times it actually works performance is very good. 9.4Gbps in iperf using 3% CPU, and 9Gbps up/9Gbps down at the same time as well, no jumbos and no kernel tuning. The PCIe warning is obviously bogus, and it also went away with kernel 4.3.

May take it for a spin with Windows in a few days..

Are you using a Zyxel XGS1920-12 switch by any chance?

 

[trippehh]

Well, it seems link comes up reliably if you make the switch only announce 10G to prevent 1G fallback, and just wait long enough. Waiting for up to an hour to get link is not great though

Behaves the same way with FreeBSD (10.2-STABLE)

 

[jimmy_1969]

Are you using a Zyxel XGS1920-12 switch by any chance?

@trippehh: I am using a Dell PowerConnect 5324. Different switch, but same symptom.

 

[jimmy_1969]

UPDATE: Getting ixgbe driver to work on CentOS 7.1

So in my previous update I shared how to get the Intel X557-AT NICs up using Fedora 23 Server Beta.

But for my project I plan to use oVirt 3.6 which does not yet support that Fedora release. This time I knew exactly how to make it work so setting it up in CentOS 7.1 should be a breeze, right? Well, not really...

My initial plan was to install CentOS 7.1 and then add the 4.3 kernel from reposotory elrepo-kernel. It includes the ixgbe driver release 4.0.1-k which can be made to work with a 1 Gb switch using a ethtool work-around. But after installing kernel 4.3.0-1 with expected driver, the link refused to activate. Applying the work-around didn't do anything.

So I went for plan B and installed the driver from source. Downloaded ixgbe 4.1.5 drivers from Intel's site.
Followed this method of procedure to compile, test and install the new driver. Note that if IP forwarding- or bridging features are to be used the driver needs to be compiled to disable the Large Receive Offload feature (LRO).

make CFLAGS_EXTRA="-DIXGBE_NO_LRO"

Once the new driver has been loaded to the kernel, the dmesg happily announced:

[ 8370.459076] ixgbe 0000:03:00.0 eno1: NIC Link is Up 1 Gbps, Flow Control: None
[ 8370.461865] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready

The downside of this approach is that the 10 GBe NICs will break every time there is a kernel update. So automatic kernel yum updates are not recommended. And it is wise to have a spare USB 1G Ethernet adapter as a backup.

The upside is that the link activation happens much faster during boot-up, and you don't need to configure any ETHTOOL_OPTS in ifcfg-<device-name>.

//Jimmy

 

[trippehh]

Update:
I finally got off my ass and pestered Zyxel about the X557 + XS1920 interop problem, they sent over a internal firmware build + updated broadcom phy firmware which fixed my 10Gbase-T problems. Now it goes to 10G on every try, without any delay. Yay! I can finally remove my teaming hack using one 1G and one 10G port in failover configuration

The Zyxel fw was dated late february (from build timestamp) and the broadcom phy fw seems to be dated 17 march (from file timestamp), so they've had the fix for a few months. It seems anything with Broadcom Trident II inside needs a similar fix to be friends with these ports.

I'm also seeing SFP+ interop problems with X552, over DAC, 10G SR and 10G bidi, which is still a problem after this fix. What a PITA this combination has been...