[Solved] ESX turns off Mellanox MT26448 optical after reboot

[high]

ESX 6.7, Mellanox MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s], Cisco 3750E, X2-10GB-SR

The Mellanox (HP brand card) is connected via fiber to the X2-10GB-SR transceiver. After th ESX host reboots, the nic is fully operational and traffic is going both ways and the Cisco is blinking. But after a few minutes, the connection seems to drop. ESX is saying the link is up, but the Cisco is saying that the 10GB line protocol is down "(not connect)" and that the Optical Receive Power is -40. No more traffic flows. Disconnecting the transceiver or the fiber cable and connecting doesn't bring the link back, only a reboot of the ESX host. Then it works for a few minutes before shutting down again.

I can't see any reference in the ESX or Cisco logs. ESX still thinks the link is connected, probably because the Ciso Optical Transmit Power is -4.4 and the Mellanox is receiving light, but no light from the Mellanox to the Cisco switch. Any thoughts on why ESX or the NIC is shutting down the transmit side?

Thanks!

 

[high]

esxcli network nic down -n vmnic6
esxcli network nic up -n vmnic6

Doens't bring the link back to life either. Only rebooting the host, then it goes down after a few minutes.

I can see the light in the end of the fibre cable from the switch, but don't see the light transmitted from the NIC .I've tried it with two separate MT26448 cards and both exhibit the same issue. Could the issue be on the switch of X2-10GB-SR module on the switch? I suspect the problem is on the NIC/ESX side rather than the switch side since ESX still thinks the link is up, but the transceiver doesn't look to be transmitting.

 

[high]

I tried with another X2-10GB-SR and a different fiber cable, but no joy. Just waiting for delivery of a genuine Cisco SFP+ to see if the transceiver is the issue. I currently have an HP SFP+ for the Mellanox, which is actually a HP card.

The system I have is a SuperMicro SYS-7047R-TRF with a Super X9DRi-F motherboard. It has been pretty reliable over the years.

Perhaprs there is an incompatability between the 10gb card, ESX, and this motherboard?

 

[high]

Received another HP SR transceiver and this one had no problems. Same model number so the old one must be faulty. Replaced the SFP+ transceiver and now have my 10gb uplink!

 

[kent05]

I have same problem with yours and I'm uncertain whether it is fault on my third party SPF+ or cisco X2-10GB-SR transceiver.