Hoping someone can help to keep the rest of my hair from falling out...
I have the following set up:
FreeNAS 11.3U2 on Supermicro X9DR3-LNF4+
Two Mellanox (reflashed HPE) ConnectX-3 adapters on 2.42.5000 firmware
4 3m FS.com 40Gb DAC's with Brocade branding in the EEPROM's
Brocade ICX 6650 on latest firmware
Cables connected to the 6650 on ports 1/2/1 through 1/2/4
The issues I'm having is that only one port comes up out of all four on the Mellanox cards. I have some Mellanox 2m DAC cables that I've tried as well, and none of the ports come up when connected to the switch (even the one that comes up with the FS cable).
Observations:
If I take the FS cable from the port that works on the Mellanox side and move it to the other 3 ports on the Mellanox side, none of them come up.
If I take the Mellanox cable and connect the Mellanox ports together, they come up (same cable that won't come up on the switch).
I've used mlxfwconfig to set all ports to 2 (ETH) and confirmed with querying after reboot they report 2 (ETH)
PCICONF -lv shows both cards with ethernet as the class.
I cannot force any of the ports up with ifconfig.
I can take the DAC's and connect any of the module 2 ports on the switch to each other and they temporarily come up (yes, I know this creates a loop but just want to confirm they come up)
Running "show media" on the switch confirms copper passive cables when any cable is plugged in and empty when unplugged.
I am unable to find any difference between the port that comes up and the ones that don't. The behavior doesn't make sense. I can't determine if the issue is on the switch or the cards. The switch isn't complaining about anything. Anyone have any ideas?
I have the following set up:
FreeNAS 11.3U2 on Supermicro X9DR3-LNF4+
Two Mellanox (reflashed HPE) ConnectX-3 adapters on 2.42.5000 firmware
4 3m FS.com 40Gb DAC's with Brocade branding in the EEPROM's
Brocade ICX 6650 on latest firmware
Cables connected to the 6650 on ports 1/2/1 through 1/2/4
The issues I'm having is that only one port comes up out of all four on the Mellanox cards. I have some Mellanox 2m DAC cables that I've tried as well, and none of the ports come up when connected to the switch (even the one that comes up with the FS cable).
Observations:
If I take the FS cable from the port that works on the Mellanox side and move it to the other 3 ports on the Mellanox side, none of them come up.
If I take the Mellanox cable and connect the Mellanox ports together, they come up (same cable that won't come up on the switch).
I've used mlxfwconfig to set all ports to 2 (ETH) and confirmed with querying after reboot they report 2 (ETH)
PCICONF -lv shows both cards with ethernet as the class.
I cannot force any of the ports up with ifconfig.
I can take the DAC's and connect any of the module 2 ports on the switch to each other and they temporarily come up (yes, I know this creates a loop but just want to confirm they come up)
Running "show media" on the switch confirms copper passive cables when any cable is plugged in and empty when unplugged.
I am unable to find any difference between the port that comes up and the ones that don't. The behavior doesn't make sense. I can't determine if the issue is on the switch or the cards. The switch isn't complaining about anything. Anyone have any ideas?
Last edited: