10gb NIC issues OmniOS

manxam

Active Member
Jul 25, 2015
235
50
28
Hi, I have 2x Intel AF DA cards (Intel 82598EB). One of which is installed in a Windows Server 2012 R2 box. The other is in a Supermicro X8DTH OmniOS box.

These are directly attached by confirmed working Twinax cables but both sides show no link when connected to each other. All interfaces are in the same subnet.

When looping the cable on the Windows machine the link comes up as expected. When performing the same on OmniOS the link stays down (lights off on NIC).

I have swapped the NICs between servers and the issue stays with the OmniOS box.

Modprobe shows the ixgbe driver
Checking the logs shows that it correctly detects the card
dladm shows two interfaces with static IP addresses configured BUT also shows the link as down even with the cable looped back. The link lights on the card do not turn on.

I've verified that this card is in the Illumos HCL but cannot, for the life of me, determine why I can't get a link.
I've very new to Solaris but using Napp-it could easily get my 1GB cards detected and working but not my 10GB.

If anyone can offer me suggestions on things to check/try/kick I'd appreciate it.

Thanks,
M

EDIT: I just did an 'lspci' and noticed that the onboard NICs are showing in output but not the 10Gb. I assume that this means something despite OmniOS loading the module and showing in its dmesg?

EDIT2: lspci doesn't show one of my 3 M1015 HBAs. The one that it doesn't display is not currently attached to anything. sas2ircu DOES detect it though. The HBA and network card are in slots 5/6 of my motherboard. This is a dual processor motherboard with one processor. Is there a limit to PCI devices in a single CPU instance? *grasping at straws*
 
Last edited:

cperalt1

Active Member
Feb 23, 2015
180
55
28
41
Don't know how to help you but I would go on the #omnios channel on freenode irc as the OmniOS developers hang out there and might be able to help you out. Thats what I would do. Just be prepared to have a few pastebins of what you have seen and found and that will increase your chances of a good outcome.
 

manxam

Active Member
Jul 25, 2015
235
50
28
Thank you both for your assistance.
I've moved the NIC to slot 3 though I don't think it matters as my board is the one listed in Gea's link and clearly shows all 8 PCI buses being active regardless of CPU qty.
I've reset my bios to default and only changed a couple small settings unrelated to PCI.
Unfortunately, this card is still not working in OmniOS.

I see the following in dmesg:

Code:
Mar 30 19:46:12 fileserver pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10f1 (ixgbe) instance 3 irq 0x52 vector 0x64 ioapic 0xff intin 0xff is bound to cpu 6
Mar 30 19:46:12 fileserver pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10f1 (ixgbe) instance 3 irq 0x53 vector 0x65 ioapic 0xff intin 0xff is bound to cpu 7
Napp-it enumerates my ixgbe3 and ixgbe4 (was 1 and 2 until I moved slots).
I've set static IP with NO GATEWAY as these will be direct connected to another machine and are on a different subnet than my 1GB interface.

Code:
 ixgbe2       ixgbe2       Ethernet       down       0       unknown       static       10.10.10.10       255.255.255.0       0:1b:21:29:8b:71       9000       inaccessible                
ixgbe3       ixgbe3       Ethernet       down       0       unknown       static       10.10.10.15       255.255.255.0       0:1b:21:29:8b:70       9000       inaccessible
Links still show down and link lights are off.
I've never been more confused...

EDIT: It has to be a config issue.. scanpci and prtdiag list the network card.
My config:
Code:
ipadm create-if ixgbe2
ipadm create-if ixgbe3
ipadm create-addr -T static -a 10.10.10.10/24 ixgbe2/v4
ipadm create-addr -T static -a 10.10.10.15/24 ixgbe3/v4
 
Last edited:

manxam

Active Member
Jul 25, 2015
235
50
28
Thank you Gea. As stated, OmniOS detects the card and I see the output in "dladm show-link" as well as in Napp-IT.
Just all attempts at creating a direct to other computer network on it fail but I know the card is good as is the Twinax.
 

its

New Member
Mar 14, 2017
10
2
3
55
I think I might have hit the same error as you. It looks like there is a bug in the illumos PCIe device discovery code at least in older dual processor systems that fails to attach a driver on slots connected to the second processor. In my case, it is a Dell r420. The card works when I use the single processor riser, but it doesn’t with the dual processor riser. In both cases it is listed in PCIe (using pcieadm and prtconf) but no driver attaches when connected to the second processor with the dual processor riser.