100GbE Mellanox 180 USD

[llowrey]

The slow boat from China was extremely slow, but the card finally arrived.

I installed the card as-is and mget_temp reports 91C. Apparently the max safe is 105C so this is cutting it a little close. Ambient is 32C at the moment though.

The card is loaded with a Lenovo firmware and is stuck in IB mode. I was not able to change the port to Ethernet using mlxconfig. Hopefully I can load the regular Mellanox firmware and get this thing switched over.

I did switch my switch port to IB and it linked up at FDR-10. I'm not seeing an option in my switch config for FDR even though I have the option for 56GbE when in Ethernet mode. Anyway, I think that has more to do with my switch than the card.

Details...

Product Name: CX455A - ConnectX-4 QSFP
Read-only fields:
        [PN] Part number: 00MW478
        [EC] Engineering changes: A1
        [SN] Serial number: 00MW478YK502065K00C
        [V0] Vendor specific: PCIeGen3 x16
        [RV] Reserved: checksum good, 0 byte(s) reserved

hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         12.100.6440
        node_guid:                      248a:0703:001e:0644
        sys_image_guid:                 248a:0703:001e:0644
        vendor_id:                      0x02c9
        vendor_part_id:                 4115
        hw_ver:                         0x0
        board_id:                       LNV2180110032
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               65535
                        port_lmc:               0x00
                        link_layer:             InfiniBand

Device #1:
----------

  Device Type:      ConnectX4
  Part Number:      00KH925_Ax
  Description:      Mellanox ConnectX-4 EDR IB VPI Single-port x16 PCIe 3.0 HCA
  PSID:             LNV2180110032
  PCI Device Name:  /dev/mst/mt4115_pciconf0
  Base GUID:        248a0703001e0644
  Base MAC:         248a071e0644
  Versions:         Current        Available
     FW             12.0100.6440   N/A

  Status:           No matching image found

# mlxconfig -d /dev/mst/mt4115_pciconf0  q

Device #1:
----------

Device type:    ConnectX4
Name:           N/A
Description:    N/A
Device:         /dev/mst/mt4115_pciconf0

Configurations:                              Next Boot
-E- Device doesn't support any configuration changes.

 

[fohdeesha]

I'd imagine idling at 91C will reduce that ASIC's life to a couple months

 

[llowrey]

I'd imagine idling at 91C will reduce that ASIC's life to a couple months

Idle is down to 88C but I'm definitely going to remove the heatpipe assembly and slap on a finned heatsink. The heatpipe assembly interfaces through an absurdly thick thermal pad. It's probably fine if the other end of the heatpipe is kept cold via a water cooling system.

Anyway, I loaded up the stock Mellanox firmware and linked up at 40GbE. It would not link when the switch was set to 56GbE. Oddly, the datasheet for the MCX455a-ECAT does list 56GBASE-R4 as supported for Ethernet but does not list 56Gb/s as a supported Ethernet speed.

Settings for enp65s0:
        Supported ports: [ Backplane ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
                                50000baseCR2/Full
                                50000baseKR2/Full
                                100000baseKR4/Full
                                100000baseSR4/Full
                                100000baseCR4/Full
                                100000baseLR4_ER4/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
                                50000baseCR2/Full
                                50000baseKR2/Full
                                100000baseKR4/Full
                                100000baseSR4/Full
                                100000baseCR4/Full
                                100000baseLR4_ER4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  Not reported
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000004 (4)
                               link
        Link detected: yes

 

[awedio]

Idle is down to 88C but I'm definitely going to remove the heatpipe assembly and slap on a finned heatsink.

Who/where is a good source for the heatsinks?

 

[awedio]

Is there anything such as an "inexpensive" 100GBe switch?

 

[LukeP]

2nd hand if youre lucky. inexpensive 10gbe is still rare.

 

[Rand__]

Is there anything such as an "inexpensive" 100GBe switch?

Depends what is inexpensive for you - most used I've seen go for around 3-4k

 

[awedio]

@Rand__ Which brands/model have you seen?

 

[Rand__]

Mostly Mellanox 7700/7790s

 

[llowrey]

Who/where is a good source for the heatsinks?

I salvaged a heatsink from an old GPU.



These nylon zip-ties are only good to 85C so I'll need to acquire some hi-temp zip-ties.

After my ambient temp got back to normal, the temp with heatpipe was 88C and with the above heatsink it's down to 77C.

I'm likely going to try the folllowing active solution next:

Sunon Super Slim 50x7mm Aluminum Heatsink 40x40x5mm Fan 795508170842 | eBay

 

[llowrey]

The active cooling solution is working well and has the temperature down to 69C. Ambient is an invigorating 30C.

As you can see from the bottom photo, my chassis is very cramped and there's almost no airflow between the cards. The passive heatsink had to rely on convection to get the hot air up and out. Ordinarily, the stock passive heatsink is probably quite adequate but in my case I do benefit from an active cooling solution.

The Sunon heatsink plus a pack of high temperature zip-ties ran me just shy of $20. I had to salvage a molex power connector from my scrap pile since the fan leads were bare. Still, not a bad deal at $200 (180+20).