Struggling with Mellanox ConnectX-3 in FreeNAS

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

MoMeanMugs

Member
Apr 16, 2018
60
19
8
74
Hoping someone can help to keep the rest of my hair from falling out...

I have the following set up:

FreeNAS 11.3U2 on Supermicro X9DR3-LNF4+
Two Mellanox (reflashed HPE) ConnectX-3 adapters on 2.42.5000 firmware
4 3m FS.com 40Gb DAC's with Brocade branding in the EEPROM's
Brocade ICX 6650 on latest firmware
Cables connected to the 6650 on ports 1/2/1 through 1/2/4

The issues I'm having is that only one port comes up out of all four on the Mellanox cards. I have some Mellanox 2m DAC cables that I've tried as well, and none of the ports come up when connected to the switch (even the one that comes up with the FS cable).
Observations:
If I take the FS cable from the port that works on the Mellanox side and move it to the other 3 ports on the Mellanox side, none of them come up.
If I take the Mellanox cable and connect the Mellanox ports together, they come up (same cable that won't come up on the switch).
I've used mlxfwconfig to set all ports to 2 (ETH) and confirmed with querying after reboot they report 2 (ETH)
PCICONF -lv shows both cards with ethernet as the class.
I cannot force any of the ports up with ifconfig.
I can take the DAC's and connect any of the module 2 ports on the switch to each other and they temporarily come up (yes, I know this creates a loop but just want to confirm they come up)
Running "show media" on the switch confirms copper passive cables when any cable is plugged in and empty when unplugged.

I am unable to find any difference between the port that comes up and the ones that don't. The behavior doesn't make sense. I can't determine if the issue is on the switch or the cards. The switch isn't complaining about anything. Anyone have any ideas?
 
Last edited:

MoMeanMugs

Member
Apr 16, 2018
60
19
8
74
BTW, the ICX 6650 has the licenses installed and confirmed they are active.

Let me know if any output from switch or FreeBSD commands would be helpful.
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
What happens if you direct connect the 2 cards together using the DACs? Does the port light up?
Are these passive or active DAC cables?
Does ifconfig -v on FreeBSD show the media/vendor info?

Trying to see if the DACs are being rejected on the card side due to the encoding.
 

MoMeanMugs

Member
Apr 16, 2018
60
19
8
74
Yes, this is one of the things I tried. The ports do come up. If I run ifconfig -vvv mlxenX (0 through 3), each reports the DAC. The info between the working port and the 3 non-working ports is the same (other than serial numbers):

Code:
root@XXX[~]# ifconfig -vvv mlxen0
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:02:c9:39:e7:80
        hwaddr 00:02:c9:39:e7:80
        inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        [b]status: no carrier[/b]
        plugged: QSFP+ 40GBASE-CR4 (No separable connector)
        vendor: FS PN: QSFP-40G-DAC SN: S1807460810-1 DATE: 2018-08-03
        compliance level: Unspecified
        nominal bitrate: 10300 Mbps

        SFF8436 DUMP (0xA0 128..255 range):
        0D 00 23 08 00 30 00 40 00 80 D5 00 67 00 00 00
        00 00 03 A0 46 53 20 20 20 20 20 20 20 20 20 20
        20 20 20 20 07 00 40 20 51 53 46 50 2D 34 30 47
        2D 44 41 43 20 20 20 20 41 20 08 0B 00 00 46 08
        00 00 00 00 53 31 38 30 37 34 36 30 38 31 30 2D
        31 20 20 20 31 38 30 38 30 33 20 20 00 00 00 88
        37 34 30 2D 30 33 38 36 32 34 20 52 45 56 20 30
        31 00 00 00 00 00 00 00 00 00 FF FF FF FF FF FF

        SFF8436 DUMP (0xA0 0..81 range):
        0D 00 23 FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF
root@XXX[~]# ifconfig -vvv mlxen3
mlxen3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: VMWare SAN Host iSCSI Path 2
        options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:02:c9:39:af:61
        hwaddr 00:02:c9:39:af:61
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (40Gbase-CR4 <full-duplex,rxpause,txpause>)
        [b]status: active[/b]
        plugged: QSFP+ 40GBASE-CR4 (No separable connector)
        vendor: FS PN: QSFP-40G-DAC SN: S1807460809-1 DATE: 2018-08-03
        compliance level: Unspecified
        nominal bitrate: 10300 Mbps

        SFF8436 DUMP (0xA0 128..255 range):
        0D 00 23 08 00 30 00 40 00 80 D5 00 67 00 00 00
        00 00 03 A0 46 53 20 20 20 20 20 20 20 20 20 20
        20 20 20 20 07 00 40 20 51 53 46 50 2D 34 30 47
        2D 44 41 43 20 20 20 20 41 20 08 0B 00 00 46 08
        00 00 00 00 53 31 38 30 37 34 36 30 38 30 39 2D
        31 20 20 20 31 38 30 38 30 33 20 20 00 00 00 90
        37 34 30 2D 30 33 38 36 32 34 20 52 45 56 20 30
        31 00 00 00 00 00 00 00 00 00 FF FF FF FF FF FF

        SFF8436 DUMP (0xA0 0..81 range):
        0D 00 23 FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
        FF FF
I have a couple Brocade QSFP's, but I don't have any MPO cables. Just to see, I plugged this into a port to see if it would be detected:

Code:
root@XXX[~]# ifconfig -vvv mlxen2
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: VMWare SAN Host iSCSI Path 1
        options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:02:c9:39:af:60
        hwaddr 00:02:c9:39:af:60
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
        plugged: QSFP+ Unknown (MPO Parallel Optic)
        vendor: BROCADE PN: 57-1000267-01 SN: HTA21617000009P DATE: 2016-04-26
        compliance level: Unspecified
        nominal bitrate: 14000 Mbps
        module temperature: 28.72 C voltage: 3.29 Volts
        lane 1: RX: 0.00 mW (-inf dBm) TX: 0.38 mW (-4.11 dBm)
        lane 2: RX: 0.00 mW (-inf dBm) TX: 0.38 mW (-4.18 dBm)
        lane 3: RX: 0.00 mW (-inf dBm) TX: 0.00 mW (-inf dBm)
        lane 4: RX: 0.00 mW (-inf dBm) TX: 0.00 mW (-inf dBm)

        SFF8436 DUMP (0xA0 128..255 range):
        0D 00 0C 00 00 00 00 00 00 00 00 05 8C 00 00 21
        00 00 32 00 42 52 4F 43 41 44 45 20 20 20 20 20
        20 20 20 20 00 00 05 1E 35 37 2D 31 30 30 30 32
        36 37 2D 30 31 20 20 20 41 20 42 68 07 D0 46 3F
        00 00 0F DE 48 54 41 32 31 36 31 37 30 30 30 30
        30 39 50 20 31 36 30 34 32 36 20 20 08 00 00 DF
        41 46 42 52 2D 37 39 46 34 45 5A 2D 42 52 31 20
        20 32 20 20 20 20 20 20 20 20 00 00 00 00 00 00

        SFF8436 DUMP (0xA0 0..81 range):
        0D 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00
        00 00 00 00 00 00 1C B9 00 00 80 24 00 00 00 00
        00 00 00 00 00 00 00 00 00 00 0F 24 0E CA 0F 25
        0F 1C 1F D9 25 77 23 7B 24 4D 00 00 00 00 00 00
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        00 00
All the ports detect the DAC's (they are all passive) and the QSFP module. What I don't understand is that the working cable doesn't work on other ports. I'm wondering if there's some bug that is causing only the last port to come up (mlxen3). Going to try to install Windows to see if the behavior is different...
 

MoMeanMugs

Member
Apr 16, 2018
60
19
8
74
Show media on the 6650 (I plugged the other QSFP module into 1/2/1 to see if it would be detected):

Code:
Port 1/2/1:  Type : 40GBASE-Passive Copper
Port 1/2/2:  Type : 40GE-Passive Copper
Port 1/2/3:  Type : 40GE-Passive Copper
Port 1/2/4:  Type : 40GE-Passive Copper
 

-MoNsTeRRR

New Member
May 24, 2020
5
1
3
Hey, I just faced the same issue today with the icx6650, mellanox X3 flashed and 40G FS DAC with an Intel server running ESXI 6.7. Did you find a workaround ?
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
40G FS DAC
I think in the last 6 months or so almost all reported problems on sth in combination with connect-x3 nics and brocade/arista switches used fs.com cables or transceivers. I think they have a problem with their QA...
Try some other known and working brand (the icx megathread had some links to dirt cheap cables/transceivers).
 

-MoNsTeRRR

New Member
May 24, 2020
5
1
3
I think in the last 6 months or so almost all reported problems on sth in combination with connect-x3 nics and brocade/arista switches used fs.com cables or transceivers. I think they have a problem with their QA...
Try some other known and working brand (the icx megathread had some links to dirt cheap cables/transceivers).
Okay thanks for the info I will try with differents QSFP+ :)
 

MoMeanMugs

Member
Apr 16, 2018
60
19
8
74
Okay thanks for the info I will try with differents QSFP+ :)
I think @i386 is spot on. I ended up ditching the DAC's and going with Brocade QSFP's and MPO cables. Everything works as expected with these. I can give you the part number on eBay I got, but I think I mentioned it to someone else on these forums if you want to search my posts.
 

-MoNsTeRRR

New Member
May 24, 2020
5
1
3
I think @i386 is spot on. I ended up ditching the DAC's and going with Brocade QSFP's and MPO cables. Everything works as expected with these. I can give you the part number on eBay I got, but I think I mentioned it to someone else on these forums if you want to search my posts.
If you tell me it works I will buy the same ref as your, dont want to lose time and money :D

Thanks for your quick reply
 
  • Like
Reactions: MoMeanMugs

itronin

Well-Known Member
Nov 24, 2018
1,234
794
113
Denver, Colorado
  • Like
Reactions: -MoNsTeRRR