ConnectX3 40GbE only works at 10G in FreeNAS

[metag]

I am having a strange issue in FreeNAS: I have a few HP MCX354A-QCBT cards with 10GbE. I had one installed in my FreeNAS box. It connects successfully to my SX6012 switch port with speed set at 10G. Recently I setup a Ubuntu box to flash all of them to FCBT. After that, they can connect to the SX6012 at 40G from Ubuntu. However, I could not get it to connect at 40G from FreeNAS. The media type shown by ifconfig is 10Gbase-CX4. If I set the switch port at 40G, the link will be broken. How can I get it to work as 40G? Manually setting media type would fail:

root@ZFS01[~]# ifconfig mlxen1
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: mlxen1
        options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:02:c9:41:f6:91
        hwaddr 00:02:c9:41:f6:91
        inet 192.168.70.12 netmask 0xffffff00 broadcast 192.168.70.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex>)
        status: active
root@ZFS01[~]# ifconfig mlxen1 down
root@ZFS01[~]# ifconfig mlxen1 media 40GBase-SR4 mediaopt full-duplex
ifconfig: SIOCSIFMEDIA (media): Device not configured
root@ZFS01[~]# ifconfig mlxen1 up
root@ZFS01[~]# ifconfig mlxen1 media 40GBase-SR4 mediaopt full-duplex
ifconfig: SIOCSIFMEDIA (media): Device not configured
root@ZFS01[~]# ifconfig mlxen1 media 40GBase-LR4 mediaopt full-duplex
ifconfig: SIOCSIFMEDIA (media): Device not configured
root@ZFS01[~]# ifconfig mlxen1 media 40GBase-CR4 mediaopt full-duplex
ifconfig: SIOCSIFMEDIA (media): Invalid argument

Another unrelated question: is there additional configuration necessary on FreeNAS side to support PFC enabled switch port? If PFC is enabled on the port (either for iperf client or server or both), I can only get to 2-3G from iperf. Only if PFC is off for both client and server, I can get to about 9.5Gbps.

 

[herby]

What version of FreeNAS, and what optics?
I might not be able to help you troubleshoot but I can confim that what you have should work.

My FreeNAS (11.2-U8)

# ifconfig mlxen1
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:02:c9:37:b4:f1
        hwaddr 00:02:c9:37:b4:f1
        inet 192.168.11.28 netmask 0xffffff00 broadcast 192.168.11.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (40Gbase-CR4 <full-duplex,rxpause,txpause>)
        status: active
# ifconfig mlxen0
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:02:c9:37:b4:f0
        hwaddr 00:02:c9:37:b4:f0
        inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (40Gbase-CR4 <full-duplex,rxpause,txpause>)
        status: active

I'm using those cheap NetApp QSFP-QSFP 112-00177 cables, the same HP NICs flashed to Connectx-3s, and a flashed SX6012

 

[metag]

What version of FreeNAS, and what optics?
I might not be able to help you troubleshoot but I can confim that what you have should work.

Thanks for confirming. I am on 11.3. Silly me: I never gave it enough time to negotiate the link speed it seems. This afternoon, I set the port speed 40G then completely forgot about it. Now it is connected at 40G!

The remaining question: can FreeBSD work properly with PFC on switch ports, or am I making another dumb mistake? I will have a mixed environment. IIRC, RoCE requires PFC on switch to work. So I would like to have PFC on for all ethernet ports on the SX6012.