Just our luck, about a year ago we sourced some refurb 10gb cards (Intel X520 fibre cards) along with Ubiquiti 10GB SPF+ modules - We previously used Emulex cards with the modules and they worked fine with 10GB but was unable to source so considering "Intel" we saw no issue. Popped them into a new host and at surface all looked well so when we went into production, we realized backups took longer than expected and noted that the host only gave 1GB instead of 10GB. After extensive testing (first thinking cable, maybe a driver or some setting during boot), I stumbled upon a worst case scenario that the card does not support all SFP+ modules.
As the host is already in production and we had a spare card, I popped the card into another server to see if I can reproduce the issue and maybe figure out what I will be able to do without having to request downtime. Note that the test server has the integrated 2 eth ports, and we have another 4 port eth port (so in total 6 ports eth).
I went ahead and installed XCP, but found that during installation the card was not picked up at all. After installing (first boot), I saw the dreaded message:
I also picked up this little message just before the XCP loader showed:
Going into the network devices, I could not see any of my fibre ports at all (which leads me to believe the above message failed to actually load the fcoe_driver hence why the interface did not show up at all:
So I did what any other admin would do and went ahead and did a yum update and rebooted. This time around I saw the "Unsupported message" but did not see the critical error for the FCOE driver, and I could see in the XCP TTY dashboard that it now shows the 10GB card port, but only the one that is unplugged.
Here is a screengrab of the dmesg which iterates the unsupported SFP+ module
I then followed this guide to enable unsupported modules (How To Fix The Unsupported SFP+ Issue in Linux ? - ServerAdminz Blog | Who Knows Server Better Than Adminz?). In a nutshell:
1. [root@xyz ~]# vi /etc/modprobe.d/ixgbe.conf
2. options ixgbe allow_unsupported_sfp=1
3. [root@xyz ~]# rmmod ixgbe; modprobe ixgbe [ we then remove the module and reload it again with these commands ]
4. [root@xyz ~]# ip a [to check whether the missing interface with the uplink is showing]
After performing these steps, dmesg now shows a different message and at face value it appears that both ports on the card is listed:
I can also see 2 new interfaces (side-3923-eth2 and side-5525-eth2) and it appears that eth2 has "dissapeared":
Doing an ethtool on both these new interfaces "appears" that side-3923-eth2 has the modules plugged in whereas the other interface is likely the one that is unplugged:
So here we are, on the switch side it shows the port is connected, but ethtool does not appear to pick up a link. I also expected the interfaces to be renamed to the standard "ens1f1" etc, not sure what the "side" name is. At face value it appears that they are both associated to eth2. In XCP, eth2 is available as an interface (and shows the intel 10gb card" but it shows as unplugged.
This leads me to the only community I believe I might get answers
Has anyone seem something similar? I know the modules work as the same server was used with the same modules in opnsense. I have already dived much deeper into this than what I would have wanted (it appears that getting other modules might just do the trick) but because I have come this far, I dont want to just quit without getting some sort of solution or a "what are you doing, why haven't you done that" from the community so that I can learn something from this ordeal
Anyone able to help a brother out? I'm happy to run / test / give output of commands, let me know
As the host is already in production and we had a spare card, I popped the card into another server to see if I can reproduce the issue and maybe figure out what I will be able to do without having to request downtime. Note that the test server has the integrated 2 eth ports, and we have another 4 port eth port (so in total 6 ports eth).
I went ahead and installed XCP, but found that during installation the card was not picked up at all. After installing (first boot), I saw the dreaded message:
I also picked up this little message just before the XCP loader showed:
Going into the network devices, I could not see any of my fibre ports at all (which leads me to believe the above message failed to actually load the fcoe_driver hence why the interface did not show up at all:
So I did what any other admin would do and went ahead and did a yum update and rebooted. This time around I saw the "Unsupported message" but did not see the critical error for the FCOE driver, and I could see in the XCP TTY dashboard that it now shows the 10GB card port, but only the one that is unplugged.
Here is a screengrab of the dmesg which iterates the unsupported SFP+ module
I then followed this guide to enable unsupported modules (How To Fix The Unsupported SFP+ Issue in Linux ? - ServerAdminz Blog | Who Knows Server Better Than Adminz?). In a nutshell:
1. [root@xyz ~]# vi /etc/modprobe.d/ixgbe.conf
2. options ixgbe allow_unsupported_sfp=1
3. [root@xyz ~]# rmmod ixgbe; modprobe ixgbe [ we then remove the module and reload it again with these commands ]
4. [root@xyz ~]# ip a [to check whether the missing interface with the uplink is showing]
After performing these steps, dmesg now shows a different message and at face value it appears that both ports on the card is listed:
I can also see 2 new interfaces (side-3923-eth2 and side-5525-eth2) and it appears that eth2 has "dissapeared":
Doing an ethtool on both these new interfaces "appears" that side-3923-eth2 has the modules plugged in whereas the other interface is likely the one that is unplugged:
So here we are, on the switch side it shows the port is connected, but ethtool does not appear to pick up a link. I also expected the interfaces to be renamed to the standard "ens1f1" etc, not sure what the "side" name is. At face value it appears that they are both associated to eth2. In XCP, eth2 is available as an interface (and shows the intel 10gb card" but it shows as unplugged.
This leads me to the only community I believe I might get answers
Has anyone seem something similar? I know the modules work as the same server was used with the same modules in opnsense. I have already dived much deeper into this than what I would have wanted (it appears that getting other modules might just do the trick) but because I have come this far, I dont want to just quit without getting some sort of solution or a "what are you doing, why haven't you done that" from the community so that I can learn something from this ordeal Anyone able to help a brother out? I'm happy to run / test / give output of commands, let me know