Hey everyone!
I recently upgraded my homelab's network to utilise the 40gig ports on the ICX6610. I purchased 2 different 40gig NICs off ebay and I cannot manage to get either to work on Ubuntu Server 23.04.
The models of cards I have:
Connectx3 CX311A 10gig NIC
Connectx3 MCX354A-FCBT 40gig VPI
ConnectX4 MCX4131A-BCAT 40gig
My 10gig NIC works flawlessly without issues. If I add either of the 40gig cards it causes the 10gig card to not function(I confirmed the machine has enough PCI lanes). When lspci is ran all the cards are seen by the system. They allow me to flash their firmware and reload the drivers. However, They are still not being recognized. ifconfig -a does not show either card. If 10gig card is removed and either 40gig card is installed I can get a light to show on the card when a DAC is plugged in but Linux still does not see the card.
I put both cards into a known good Windows machine with the correct drivers and the same symptoms present. Device manager see's card but cannot access it.
Did I managed to get 2 bad NICs in a row or am I just plan old missing something?
Edit: Purchased a Connectx3 MCX313A-BCCT which is an Ethernet only 40gig card to test a couple theories. This card functions using the same driver as the existing 10gig card. Using ifconfig -a I confirmed the mac address of the card. A second person who experienced this concern did some testing on their end by attempting to use a connectx4 card on 23.04 install and had no luck until they downgraded their distro to 22.04. On 22.04 the Connectx4 card works without manually installing any drivers. This issue would appear to be a busted driver for Ubuntu Server 23.04. Both the Infiband drivers and the Connectx4 driver appear to not function on this distro version.
Hope this helps someone! Let me know if you have any different experiences.
I recently upgraded my homelab's network to utilise the 40gig ports on the ICX6610. I purchased 2 different 40gig NICs off ebay and I cannot manage to get either to work on Ubuntu Server 23.04.
The models of cards I have:
Connectx3 CX311A 10gig NIC
Connectx3 MCX354A-FCBT 40gig VPI
ConnectX4 MCX4131A-BCAT 40gig
My 10gig NIC works flawlessly without issues. If I add either of the 40gig cards it causes the 10gig card to not function(I confirmed the machine has enough PCI lanes). When lspci is ran all the cards are seen by the system. They allow me to flash their firmware and reload the drivers. However, They are still not being recognized. ifconfig -a does not show either card. If 10gig card is removed and either 40gig card is installed I can get a light to show on the card when a DAC is plugged in but Linux still does not see the card.
I put both cards into a known good Windows machine with the correct drivers and the same symptoms present. Device manager see's card but cannot access it.
Did I managed to get 2 bad NICs in a row or am I just plan old missing something?
Edit: Purchased a Connectx3 MCX313A-BCCT which is an Ethernet only 40gig card to test a couple theories. This card functions using the same driver as the existing 10gig card. Using ifconfig -a I confirmed the mac address of the card. A second person who experienced this concern did some testing on their end by attempting to use a connectx4 card on 23.04 install and had no luck until they downgraded their distro to 22.04. On 22.04 the Connectx4 card works without manually installing any drivers. This issue would appear to be a busted driver for Ubuntu Server 23.04. Both the Infiband drivers and the Connectx4 driver appear to not function on this distro version.
Hope this helps someone! Let me know if you have any different experiences.
Last edited:
