Network foo Gone :-(

[wildchild]

Hi All,

must be loosing my mind, but just in case i missed something:

Recently bought a few of the Mellanox connectx-2 EN 10G , and Cisco's DAC cable
Also bought an Intel x520 DA2

so just for fun and games ( and while rebuilding,silencing and sorting my home cabinet) i wanted my OmniOS storage servre and my ESX 5.5 to start running 10G

OmniOS doesn't like Mellanox Connectx-2 , so the intel is in there, running ip 10.x.4.1/24 for port 1
Esx has the Connectx-2 EN card, and has been configured at 10.x.4.200/24
DAC is running in between the OmniOS box and the ESX, so they are directly connected
Both have no gateway, and i have enabled the maximum MTU of 15500
Both links show up, no errors, both can ping themselves, however no joy ping each other..

i updated the ESX driver to MEL-mlnx-en-1.9.9.0-1OEM.510.0.0.799733-1551173, left the firmware of both devices as is.

still no joy

did i miss something ?

 

[Scott Laird]

A few things:
- Try a smaller MTU. Try 1500 bytes, because that always works on all Ethernet gear. If that works, try 9,000. If that works, try larger until it fails.
- Someone claimed that some of Intel's 10G cards are locked to only use Intel optics. I'd expect that to show up as the link being down, not up, though. Do you have any other options for cables?
- Does tcpdump show any inbound traffic over the link? If so, does ARP work?
- Do you have any other 10G hardware that you could use for testing? A cheap ConnectX-2 card in a desktop would make it much easier to see which end is broken.

 

[wildchild]

turns out my foo wasn't broken
somehow OmniOS had put some pci ( not the NIC) in faulty, after flushing it, when... there was ISCSI

updated all firmwares just to be safe, but everything has been running great since saturday

 

[wildchild]

a note worth mentioning ...
eventhough the card was disabled in OmniOS , it did show link up lights ...
So seeing lights doesn't mean it's enabled!