Mellanox ConnectX-3 lost Connection after Hyper-V Installation on other NIC

[cuco]

Hello,

my MellanoX ConnectX-3 worked well the last Months. But last month i installed a HyperV Machine with an external Switch connected to one of my four Intel i350 Nics.
Since then my Mellanox 40G connection doesnt work any more.

The Server reports a IP Adresse. But i cant ping it,
It also says it is connceted but has no Internet.
Windows tells the network is not identified. But when i open network in explorer i dont get the chance to switch to private network.

I use a Supermicro M11sdv-8ct-ln4f board. It is a bit tricky with this board. I only get into bios when i remove my x8x8riser with the mallanox and my samsung PM1725 SSD.

The local Server Manager reported Problems with SR-IOV initialization. So i enabled it in Bios and turned it on in Firmware. but it didnt solve the problem. The next boot it reported Roce issues. driver loaded v1.5 and system required v2. so i switched to v2. the next reboot the mellanox had no valid IP-Adress. so i switched back.
And as soon as i disable SR-IOV in the driver menu i also lose my normal IP Adress.

But right now i am out of luck and need some help.
Any idea how to fix this?

Thank you in advance!
regards,
cuco

 

[Rand__]

I don't think that SRIOV and bifurcation work properly at the same time.
Do you really need it? Are you passing through virtual functions or only using virtual nics?

 

[cuco]

No i dont need it. It was disabled in BIOS and i enabled it to test if it works then. But didnt solve the issue.
It was enabled in the Virtual Switch i created. So i erased it and created a new one. but this didnt help.
This is so strange. Getting an ip but no connection to internet etc.
The solution center tells me it has no default gateway. But the config menu looks good.

Any other idea?

 

[Rand__]

so can you ping its own ip? the gw you assigned?

 

[elag]

Hello,

my MellanoX ConnectX-3 worked well the last Months. But last month i installed a HyperV Machine with an external Switch connected to one of my four Intel i350 Nics.
Since then my Mellanox 40G connection doesnt work any more.

The Server reports a IP Adresse. But i cant ping it,
It also says it is connceted but has no Internet.
Windows tells the network is not identified. But when i open network in explorer i dont get the chance to switch to private network.

I use a Supermicro M11sdv-8ct-ln4f board. It is a bit tricky with this board. I only get into bios when i remove my x8x8riser with the mallanox and my samsung PM1725 SSD.

The local Server Manager reported Problems with SR-IOV initialization. So i enabled it in Bios and turned it on in Firmware. but it didnt solve the problem. The next boot it reported Roce issues. driver loaded v1.5 and system required v2. so i switched to v2. the next reboot the mellanox had no valid IP-Adress. so i switched back.
And as soon as i disable SR-IOV in the driver menu i also lose my normal IP Adress.

But right now i am out of luck and need some help.
Any idea how to fix this?

Thank you in advance!
regards,
cuco

Your problem may be related to STP: is there a loop between the i350 and the Mellanox where a switch decides to block the Mellanox?

 

[cuco]

the i350 with two cables and the mellanox are hocked to a brocade icx6610.

I had this running with one cable on the i350 for WOL usage and the mellanox for fast connection.
This worked and had no issues. Should i use a sort of link aggregation here? I thought this is not necessary cause i dont need this feature with a 40gb connection as main link.

Since i added the second cable and installed the Hyper-V i have the problems.
My second server uses a similar setup with windows 10 as operation system for steam remote gaming.
It has a i210 dual port nic and aalso a mellanox. this works just fine. The other server worked fine as well.

I can ping the servers i350 nic from my laptop and from the server itself. but the mellanox card just pingable from the server itself.

I think it is not a stp issue. but i will check that by removing the i350s cable. then the mellanox card should just work.

 

[cuco]

So i resolved the issue. The Card works well. no Problems any more.

I checked the device manager this morning and recognized that there are a few devices that were no more installed.
Probably because i made a bios update before i installed the hyper-v machine.
after reinstalling the drivers of the chipset the card works well again.

Thank you all for yor help!