Gnodal Switch Status Not Connected

[darkconz]

Hi there,

I recently purchased a Gnodal GS0018 and I was fairly excited to set it up. However, I was trying to utilize my old QSFP cables I've used in my IB setup. I have Mellanox ConnectX-3 VPI adapters.

However, Gnodal must not like the cable because none of the adapters show connected. The NIC was working when connected to my IB switch but not on Gnodal. Could it be my cable that is the problem?

I have this cable: Mellanox MC2206130-002 Passive Copper Cable IB QDR/FDR10 40Gb/s QSFP 2m

Do I need to replace the cables as well?

Thanks

 

[Patrick]

Did you set the switch up using the console cable first?

 

[Scott Laird]

I had a couple Mellanox ConnectX-3s that seemed to work with IB but wouldn't sync up with 40 Gbe (with a GS0018) until I did a firmware upgrade.

 

[darkconz]

I had time yesterday to do further digging for a few hours but could not solve why. It seems like it is not the switch that is the problem but the NIC itself. Please allow me to explain a little bit on the situation.

The Gnodal switch is updated to firmware 2.6.3 and my NICs are actually onboard ConnectX-3 (Supermicro | Products | SuperServers | 2U | 6027TR-DTFRF) which I cross flashed to Mellanox 2.40 and I have installed the Mellanox drivers on my Hyper-V 2016.

Now, here are the outputs:

CA 'ibv_device0'
        CA type:
        Number of ports: 1
        Firmware version: 2.40.5030
        Hardware version: 0x0
        Node GUID:
        System image GUID:
    Port 1:
        State: Down
        Physical state: Disabled
        Rate: 0
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask:
        Port GUID:
        Link layer: Ethernet
        Transport: RoCE v1.25

        hca_idx=0
        uplink={BUS=PCI_E Gen3, SPEED=8.0 Gbps, WIDTH=x8, CAPS=8.0*x8}
        MSI-X={ENABLED=1, SUPPORTED=128, GRANTED=34, ALL_MASKED=N}
        vendor_id=0x02c9
        vendor_part_id=4099
        hw_ver=0x0
        fw_ver=2.40.5030
        PSID=MT_1100120019
        node_guid=
        num_phys_ports=1
                port=1
                port_guid=
                port_state=PORT_DOWN (1)
                link_speed=NA
                link_width=NA
                rate=NA
                port_phys_state=DISABLED (3)
                active_speed=1.00 Gbps
                sm_lid=0x0000
                port_lid=0x0000
                port_lmc=0x0
                transport=RoCE v1.25
                max_mtu=2048 (4)
                active_mtu=256 (1)
                GID[0]=
                GID[1]=

I've switched the type of port from VPI to ETH through command line. When it is in VPI or IB, the port_phys_state or Physical State would always be in Polling (which is what I expect) but when it is in ETH mode, the Physical State would always be in Disabled.

Does anyone know why this NIC doesn't like to operate in ethernet?

I have a few of these nodes which I would like to utilize the onboard ConnectX-3 NICs.

Thanks

 

[darkconz]

I had a couple Mellanox ConnectX-3s that seemed to work with IB but wouldn't sync up with 40 Gbe (with a GS0018) until I did a firmware upgrade.

Oh I also tried different versions of the firmware from Mellanox and SuperMicro, 2.49 and 2.35 respectively.


Sent from my iPhone using Tapatalk

 

[darkconz]

I actually got this working. It seems like the mlxconfig has something to not presisting the config. I installed a trial version of Windows and used GUI from Device Manager, it worked.

Now I am facing a different problem.. iperf is giving me 40gbps but file copy is capped at 2.5 gbps (~290MB/s) to RAMDisk. Anyone may have some input? I have MTUs at 4K at both switch and NIC level.