Asrockrack D1540D4U-2O8R - Onboard NIC not working

[RandellH]

I wasn't sure if this belong here or in the networking section:

The board has 2 Intel X552 SFP+ ports exposed by a Cortina CS4227 (or something like that). I'm connecting it to a Brocade switch. Now, part of my troubleshooting, I put in X520-DA1 card and it worked, so I used that DAC cable (fs.com) with each of the onboard ports and it doesn't work. The interfaces are recognized, I can see them with IP A, I can see them with ethtool, I don't see any messages about unsupported transceiver in dmesg, but I do see lines like:

ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): port expander access failed with -18
ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): CS4227 reset failed: -18

I've tried Debian (Proxmox), TrueNas Scale/Core, and Ubuntu Desktop and nothing works. When I power on, I see the lights blinking on the nic up until the point it looks like the kernel starts loading drivers and then they go out and that's the end of it.

Ethtool: Link detected: no
IP A: Down

I reached out to AsRockRack but I don't expect a response. I might end up having to use a pcie nic, but I wanted this board because of the built in sas3 controller and sfp 10gb ports.

I can grab more information if anything thinks it'll help.

 

[RolloZ170]

maybe it fails because the board was started without SFP plugged.
power down. reboot with the SFP plugged in.

 

[RandellH]

maybe it fails because the board was started without SFP plugged.
power down. reboot with the SFP plugged in.

I thought I tried that (I did see some linux patch requests about that) but I'll have to try again this evening to be 100% sure.

 

[RolloZ170]

I did see some linux patch requests about that

indeed
"On some hardware platforms, the CS4227 does not initialize properly.
Detect those cases and reset it appropriately."

 

[RandellH]

Unfortunately, that didn't seem to help. I even tried Windows 2022. From a cold boot, the lights blink on the nic until about the time the operating system loads the driver, then they go out, and no link

 

[NateS]

That sounds suspiciously like a hardware problem. Reset failures and inability to access a port expander *could* be a driver issue, but if you've tried multiple different OSes and the behavior is the same, most likely the message is pointing to a real failure.

My bet is that it's using the port expander pins to detect the cable and read its eeprom, and since it can't access the port expander, it can't see that the cables are getting plugged into it.

 

[RandellH]

That sounds suspiciously like a hardware problem. Reset failures and inability to access a port expander *could* be a driver issue, but if you've tried multiple different OSes and the behavior is the same, most likely the message is pointing to a real failure.

My bet is that it's using the port expander pins to detect the cable and read its eeprom, and since it can't access the port expander, it can't see that the cables are getting plugged into it.

That was along the lines of what I was thinking. ethtool -m interface does dump information fwiw. This is new territory for me, so I'll keep poking.

Edit: I think -18 is:
#define IXGBE_ERR_I2C -18

Edit 2: I tried setting tftpd64 up as a pxe server and enabled pxe booting from the BIOS. While I have no idea what I'm actually doing (never did that before) I did see entries in my tftpd64 log, (and the pxe boot client showed it had a connection), so the issue isn't the cable/dac.

I played with ethtool and when I do ethtool -r enp3s0f0 I get all those port expander and cs4227 reset failure messages. I wonder if there is a bad firmware, with something like an intel nvm update that could fix it. Problem is, can't use the normal intel one, would need Asrockrack to provide one, and that probably isn't going to happen.

 

[RandellH]

So, on a whim, I tried Ubuntu 14.04 (since it is on the old motherboard web page as a "supported os") and it works. I see the same dmesg logs about port expander/cs4227 reset failing, but I get a link and an ip address.

I'm going to try a couple newer versions of Ubuntu and see

Ubuntu 16.04 (upgraded from the command line) works, but then after I upgrade to 18.04, it fails. I noticed the ixgbe driver changes from 4.x to 5.x. I'll see about trying the latest 5 series driver maybe...

Edit: I reinstalled Ubuntu 16.04 and verified everything worked. ixgbe was version 4.2.1-k. I then installed 5.14.6 (Ubuntu 18.04 installed a 5.x, I don't remember which exactly) and verified that broke 16.04. Rolled back to 4.2.1-k and it started working again.

 

[RolloZ170]

I see the same dmesg logs about port expander/cs4227 reset failing, but I get a link and an ip address.

there is a workarround inside for the cs4227 "reset takes too long" issue, the msg stays there but the situation is corrected.
looks to me the latest drivers lack this workarround.

 

[RandellH]

there is a workarround inside for the cs4227 "reset takes too long" issue, the msg stays there but the situation is corrected.
looks to me the latest drivers lack this workarround.

I'm an amateur at this at best, but I look at the source for ixgbe_phy.c and ixgbe_x550.c and I didn't notice much difference between the latest 4.x and 5.x drivers. I'm still looking and playing around with making changes to the 5.x to see if I can stumble across a "fix".

 

[RolloZ170]

i think the problem is that one the D1540D4U-2O8R the CS4227 is connected by iXFI.


This patch removes X550em SFP iXFI setup for the drivers since there
is no released hardware production with SFP iXFI.

 

[RandellH]

You might be on to something. I was so hung up on 4.x vs 5.x that I assumed it was introduced with 5.x. The installed version that works is 4.2.1-k. I just tried 4.6.4 and it failed as well.

I'll try 4.3.15 (the last version before that patch removed the code) and 4.4.6 (the next one after that code) and see if that is it.

Edit: 4.4.6 works -> 4.5.4 fails

 

[RandellH]

i think the problem is that one the D1540D4U-2O8R the CS4227 is connected by iXFI.


This patch removes X550em SFP iXFI setup for the drivers since there
is no released hardware production with SFP iXFI.

Thanks @RolloZ170. That lead me to finding a was the solution.

[dpdk-dev] net/ixgbe: fix link never come up problem with x552 - Patchwork

I applied that patch to both the latest 4.6.4 and 5.14.6 versions of the Intel ixgbe drivers and verified they work. It looks like the patch was never merged or something? These patches are a little hard follow, especially when they are 6 years old.

Edit: You know, I just re-read the patch you linked to, and I must have read it wrong. If I'd just looked at it closer and rolled the changes back, I could have saved myself some heartache.

 

[Victor Fernandes]

2 HBA's HP X520-DA2 Intel 10GB HBA
Cisco SFP+ SFP-10G-SR 10-2415-03

This is for a home Lab and not production.

I've got them working on TrueNAS-SCALE-22.02.4 and Proxmox Virtual Environment 7.2-11
They both worked ok under Windows 10 and Server 2019 without any config changes, but needed to add the below

Added ixgbe.allow_unsupported_sfp=1 to /etc/default/grub.d/truenas.cfg


OLD
root@hp-two[~]# dmesg |grep ixgbe
[ 3.970290] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 3.970617] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 4.010961] ixgbe 0000:02:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 4.011426] ixgbe 0000:02:00.0: Reload the driver after installing a supported module.
[ 5.205059] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2 XDP Queue count = 0
[ 5.205832] ixgbe 0000:02:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 5.206252] ixgbe 0000:02:00.1: MAC: 2, PHY: 1, PBA No: E68785-003
[ 5.206542] ixgbe 0000:02:00.1: 00:1b:21:91:27:d9
[ 5.213498] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
[ 5.215450] ixgbe 0000:02:00.1 enp2s0f1: renamed from eth0

New
root@hp-two[~]# dmesg |grep ixgbe
[ 3.726366] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 3.726707] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 3.918139] ixgbe 0000:02:00.0: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2 XDP Queue count = 0
[ 3.919441] ixgbe 0000:02:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 3.919895] ixgbe 0000:02:00.0: MAC: 2, PHY: 19, SFP+: 5, PBA No: E68785-003
[ 3.920215] ixgbe 0000:02:00.0: 00:1b:21:91:27:d8
[ 3.925242] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
[ 5.114023] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2 XDP Queue count = 0
[ 5.114763] ixgbe 0000:02:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 5.115184] ixgbe 0000:02:00.1: MAC: 2, PHY: 1, PBA No: E68785-003
[ 5.115473] ixgbe 0000:02:00.1: 00:1b:21:91:27:d9
[ 5.117418] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
[ 5.119362] ixgbe 0000:02:00.0 enp2s0f0: renamed from eth1
[ 5.136983] ixgbe 0000:02:00.1 enp2s0f1: renamed from eth0
[ 31.811876] ixgbe 0000:02:00.1: registered PHC device on enp2s0f1


root@hp-two[~]# cat /etc/default/grub.d/truenas.cfg
GRUB_DISTRIBUTOR="TrueNAS Scale"
GRUB_CMDLINE_LINUX_DEFAULT="libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 ixgbe.allow_unsupported_sfp=1"

update-grub


For Proxmox

cat /etc/kernel/cmdline
quiet amd_iommu=on iommu=pt ixgbe.allow_unsupported_sfp=1


proxmox-boot-tool refresh


See full Instructions similar for editing the config's
How to Pass-through PCIe NICs with Proxmox VE on Intel and AMD (servethehome.com)

 

[formularity]

I used 2 TP-Link 10G RJ45 transceivers, and connected Netgear GS305 (1GbE unmanaged switch) via Cat5e patch cable. When tested in Win10 PE environment, the 2 SFP+ driver successfully installed, but neither ports can obtain valid IPs.
It is the problem of TP-Link transceiver or the Cat5e cable or Netgear GS305?