Q: What Linux distributions support Intel's new X557-AT 10GBASE-T chip?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

jimmy_1969

New Member
Jul 22, 2015
24
16
3
54
Jakarta
Hi,

I am doing a project based on Supermicro x10sdv-tln2f Mini-ITX board that uses Intel's new X557-AT 10GBASE-T chip.

My goal is to run this on CentOS 7. However, the CentOS 7 minimal installation ISO isn't able to configure the NICs. Is there anyone running this motherboard on CentOS, and if so, what method did you use to get the kernel to play with the 10 Gb NICs?

Would welcome recommendations of any other Linux distributions that has successfully been deployed with built in kernel support for this chip. (NOTE: I want to avoid compiling drivers from source for obvious reasons.)

Best Regards

//Jimmy
 

mikesteigerwalt

New Member
Aug 27, 2015
8
3
3
North DFW
CentOS isn't my preferred platform. So I'm not too sure what 7 uses in regards to kernel version. Have you tried Debian though? Debian Testing has the 4.2 kernel. Id give that a shot. You should be able to get the minimal network installer, though not sure what version of the kernel that is running.
 
  • Like
Reactions: jimmy_1969

Jeggs101

Well-Known Member
Dec 29, 2010
1,529
241
63
I'm not sure why it is so hard to just download the drivers and install them. Takes ~2 min.
 

vl1969

Active Member
Feb 5, 2014
634
76
28
I'm not sure why it is so hard to just download the drivers and install them. Takes ~2 min.
Probably because you can not connect to Internet without a nic card. I have the same issue but my nic is older. Thus red hat decided to remove kernel support for it. As luck might have it, I also have an Intel nic in the system so I can still get to the internet and load the driver for the second nic. Otherwise I would have been miserable and very pissed.


BTW, you can try Fedora. It should be all cutting edge and all. Might have the support you need and still a red hat family.
 

RTM

Well-Known Member
Jan 26, 2014
956
359
63
According to Intels website support for x550 chips appeared around ixgbe driver 4.03, the driver in RHEL 7.1 (and thus Centos) is 3.19, in RHEL 7.2 (beta) the driver is upgraded to 4.01-k, so there is a chance that you can use the NIC in RHEL 7.2.
 
  • Like
Reactions: Chuckleb

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Why not connect with another machine, drop it on a usb stick and slipstream the drivers
I get frustrated with this too. Here is the easy way to do this.
1. Install OS without driver
2. Make a little ISO with the driver
3. Mount ISO via IPMI
4. Install driver

Takes ~2-3 minutes to make, mount and install.

Much easier with IPMI enabled systems versus non-IPMI systems!
 
  • Like
Reactions: jimmy_1969

vl1969

Active Member
Feb 5, 2014
634
76
28
It is easier when it works. On my system I cannot mount any drives via ipmi. I could when I used windows to run ipmiview on
In Linux client it says drive redirection not supported by the system. Is there other ipmi gui client I can use?
 

jimmy_1969

New Member
Jul 22, 2015
24
16
3
54
Jakarta
Update 15 Oct-15
Thanks for everyone contributed with suggestions.

I had a go at installing rpms for kernel 4.2.3.1 and ixgbe 4.0.1-k-rh7.1 to CentOS7. Result was that module loaded, but there is "no link" (ethtools) and ip addr indicates "carrier not detected". So this lead me to start looking in to BIOS setting (there is none), motherboard jumper settings (confirmed 10GB NICs enabled), auto-negotiation on link level (nope), changing physical cables, changing motherboard port to 2nd 10 GB interface, changing far-end switch port, changing to other switch, deactivating Network Manager and redefining ifcfg from scratch. But the link still will not come up.

I then decided to go with what has worked for others, and found this blog post by Major Hayden where he fired up Fedora 23 and got the 10Gb NICs to work. Downloaded Fedora Server 23 Beta but the result was the same. Module ixgbe loads but the link is not active, higher layers fails.

Code:
[root@unas-host ~]# dmesg | grep -e 'ixg\|eno'
[    3.188577] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
[    3.188582] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[    4.576844] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8
[    4.643474] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
[    4.643481] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    4.643485] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[    4.643489] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    4.643491] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
[    4.683420] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.683428] ixgbe 0000:03:00.0: 0c:c4:7a:78:06:70
[    4.748924] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[    6.614575] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8
[    6.681286] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[    6.681294] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    6.681299] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[    6.681302] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[    6.681306] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
[    6.725760] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000
[    6.725767] ixgbe 0000:03:00.1: 0c:c4:7a:78:06:71
[    6.791236] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[    6.793337] ixgbe 0000:03:00.1 eno2: renamed from eth1
[    6.803464] ixgbe 0000:03:00.0 eno1: renamed from eth0
[   27.355358] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
[   27.784699] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
[   27.797231] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[   28.259135] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
Code:
[root@unas-host ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 0c:c4:7a:78:06:70 brd ff:ff:ff:ff:ff:ff
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdis
Code:
[root@unas-host ~]# modinfo ixgbe | more
filename:       /lib/modules/4.2.0-300.fc23.x86_64/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko.xz
version:        4.0.1-k
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     E08010C0D72963989E9A05E
...
...
depends:        mdio,ptp,dca,vxlan
intree:         Y
vermagic:       4.2.0-300.fc23.x86_64 SMP mod_unload
signer:         Fedora kernel signing key
sig_key:        B9:82:4D:E9:E3:5E:FB:22:C6:F2:3A:3A:B9:12:37:57:FE:17:26:B3
sig_hashalgo:   sha256
parm:           max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)
parm:           debug:Debug level (0=none,...,16=all) (int)
Code:
ethtool eno1
Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: Unknown!
    Duplex: Unknown! (255)
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: no

[root@unas-host ~]# ethtool -t eno1
The test result is FAIL
The test extra info:
Register test  (offline)     0
Eeprom test    (offline)     0
Interrupt test (offline)     0
Loopback test  (offline)     0
Link test   (on/offline)     1
Code:
cat /etc/sysconfig/network-scripts/ifcfg-eno1
HWADDR=0C:C4:7A:78:06:70
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eno1
UUID=03fc2535-42aa-46be-9a29-35044ea4f37f
ONBOOT=yes
IPADDR=10.0.69.18
PREFIX=24
GATEWAY=10.0.69.1
DNS1=10.0.69.1
Code:
network restart in journalctl:
-- Unit network.service has begun starting up.
Oct 15 05:55:04 unas-host network[1628]: Bringing up loopback interface:  Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Oct 15 05:55:04 unas-host network[1628]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Oct 15 05:55:04 unas-host kernel: cfg80211: World regulatory domain updated:
Oct 15 05:55:04 unas-host kernel: cfg80211:  DFS Master region: unset
Oct 15 05:55:04 unas-host kernel: cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
Oct 15 05:55:04 unas-host kernel: cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)
Oct 15 05:55:04 unas-host network[1628]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Oct 15 05:55:05 unas-host network[1628]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Oct 15 05:55:05 unas-host network[1628]: [  OK  ]
Oct 15 05:55:05 unas-host network[1628]: Bringing up interface eno1:  Error: Connection activation failed: No suitable device found for this connection.
Oct 15 05:55:05 unas-host network[1628]: [FAILED]
Oct 15 05:55:05 unas-host network[1628]: Bringing up interface eno2:  Error: Connection activation failed: No suitable device found for this connection.
Oct 15 05:55:05 unas-host network[1628]: [FAILED]
Oct 15 05:55:05 unas-host systemd[1]: network.service: Control process exited, code=exited status=1
Oct 15 05:55:05 unas-host systemd[1]: Failed to start LSB: Bring up/down networking.
I havn't yet tried compiling the driver from source. Found another blog post suggesting this method has been successful.

But there are a few things I don't understand. Why does the link failure only indicate IPv6 "IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready"? And what could possibly prevent the link layer from being connected?

Any suggestions would be appreciated.

Best Regards

//Jimmy
 

jimmy_1969

New Member
Jul 22, 2015
24
16
3
54
Jakarta
Update
Making progress and have finally managed to get the link up. Still need to stabilise my configuration and ensure it can survive reboots.

One thing that I noticed when booting up the kernel is that it complains that the PCIe buss on the Supermicro board is too slow to get optimal 10 GBe performance.

Code:
[    3.188577] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
[    3.188582] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[    4.576844] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8
[    4.643474] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
[    4.643481] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    4.643485] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[    4.643489] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    4.643491] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
[    4.683420] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.683428] ixgbe 0000:03:00.0: 0c:c4:7a:78:06:70
[    4.748924] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
Would be interested to hear from anyone who has tested this board in a real 10 GBe set-up if there was any unexpected performance hits during practical testing. Also would appreciate thoughts about the implications of running PCIe on lower than recommended optimal speed.

Best Regards

//Jimmy
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
Interesting that it thinks there's only a 1x link at PCIe 1.0 speeds (2.5GT/s), thus a maximum of 250MB/s; that definitely sounds like it's being detected wrong, but need a block diagram of the x10sdv-tln2f to prove it...

Assuming that isn't just some debug mode used in early initialisation that is. What does lspci -vv show you for PCIe speed and lane width for the NICs?

Edit: GIMS found me a block diagram at tweaktown and apparently shows that the 10GbE NICs aren't connected over PCIe and are over some sort of bus called KR...?
 
  • Like
Reactions: miraculix

miraculix

Active Member
Mar 6, 2015
116
25
28
Interesting that it thinks there's only a 1x link at PCIe 1.0 speeds (2.5GT/s), thus a maximum of 250MB/s; that definitely sounds like it's being detected wrong, but need a block diagram of the x10sdv-tln2f to prove it...

Assuming that isn't just some debug mode used in early initialisation that is. What does lspci -vv show you for PCIe speed and lane width for the NICs?

Edit: GIMS found me a block diagram at tweaktown and apparently shows that the 10GbE NICs aren't connected over PCIe and are over some sort of bus called KR...?
Good find! 10GBASE-KR is a standard for getting 10GE across backplanes/midplanes (so PCB copper traces generally speaking). I've seen it used by blade server vendors so a blade is able to present an SFP+ port elsewhere on the chassis (or the blade can connect to an internal/modular blade switch elsewhere on the chassis).

Since X550 is integrated in the Xeon-D SoC, yet PHY is an external chip, I guess it's a similar case here in miniature.

The -KR spec has a single 10gig data lane. I don't think the -KR link is the source of the bottleneck but I could be wrong.
 

jimmy_1969

New Member
Jul 22, 2015
24
16
3
54
Jakarta
Network Configuration Issue Solved!
Basically, my the reason my network configuration did not work out-of-the-box came down to two root causes:
  1. Incorrect ixgbe driver version
  2. Incorrect link level configuration to local switch
Issue #1 was resolved by moving to Fedora Server 23 Beta which has a working driver and kernel. That ensures it will survive kernel updates. Also disabled NetworkManager, in favour of old-school, manual network ifcfg configuration.

Issue #2 was resolved by limiting how the NIC advertises its link capabilities to the switch:
Code:
ethtool -s eno1 speed 1000 duplex full
To make this configuration permanent, just add the following lines to the network-script ifcfg file (note that the parameter sequence matters - must be specified as per below or will fail loading):
Code:
...
ETHTOOL_OPTS=”autoneg on duplex full speed 1000”
...
//Jimmy
 
Last edited:

canta

Well-Known Member
Nov 26, 2014
1,012
216
63
43
[ 4.643481] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[ 4.643485] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[ 4.643489] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[ 4.643491] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
----------
I am worried about the bold line..

KR is just the PHY backplane as already mentioned


ethtool -s eno1 speed 1000 duplex full
------
you have to force to 1gbe , autodetect does not work?
I remember with old switch 100Mb that gave headache when connected with 1gbe nic with autodetect. the only way was forcing 100Mbe to nic :D
 

jimmy_1969

New Member
Jul 22, 2015
24
16
3
54
Jakarta
Nice! Are you seeing the same PCIe bandwidth reported in dmesg?
PCIe issue remains. The interface throughput is only detected at 2.5 GT/s. If this really is a motherboard design limitation it would be a major dent to the value proposition to this 700 USD product. For a company like SuperMicro I expect that the individual components and overall design is aligned to reach peak performance.

Right now I don't have any 10GBe equipment to do any testing. Would be great if someone who has the environment could do some real performance benchmarking to verify the 10 GBe performance.

you have to force to 1gbe , autodetect does not work?
I remember with old switch 100Mb that gave headache when connected with 1gbe nic with auto detect. the only way was forcing 100Mbe to nic :D
Removing advertisement of 10 GBe speeds is the only way I could get the link up. My old Dell PowerConnect 5324 switch proberbly got confused by the 10 GB speed, I guess...

//Jimmy
 
Last edited:

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
That's why I brought up the issue of the KR bus above - according to the block diagram, the 10GbE PHYs aren't connected to anything via PCIe, they're only going over KR. MACs are meant to be internal so I wouldn't expect those to be using PCIe either. Possible that the kernel is seeing the controller and knowing it's connected somehow, but is showing the wrong info as it's not on PCIe?

Testing the throughput would of course be the easiest way to test but didn't realise you don't yet have any other 10GbE kit...

In any case... seems like the driver for these chips is still in its infancy somewhat.
 

miraculix

Active Member
Mar 6, 2015
116
25
28
Right now I don't have any 10GBe equipment to do any testing. Would be great if someone who has the environment could do some real performance benchmarking to verify the 10 GBe performance.
I have a couple of SYS-5018D-FN4Ts come in recently that I can test this weekend before I install ESXi on them. I haven't tested Linux network performance in ages but am assuming iperf is still the way to go. I can try testing back-to-back and through a 1RU 10GE switch.