10GBe direct connection issues

[talsit]

I am trying to direct connect 10GBe cards and having a heck of a time.

I'm connecting short runs via SFP, two ConnectX2 cards. They occasionally see each other.

1 GB network is 192.168.3.xxx
10GB network is 192.168.10.xxx with manual IPs set.

In the 10GBe network all I have specified in Windows Networking is the IP and netmask, I left gateway and DNS blank.

When I run pathping on my 10GBe connection it's going out on my 192.168.3.xxx network and searching the internet.

I have another server with 10GBe cards but they are Intel 520's and I'm trying to eliminate any brand incompatibility. I haven't been able to get the ConnectX2 cards to talk to the Intel cards, but I want to make sure my setup is correct before I try to dig into that anymore. The Intel cards are in an Ubuntu machine and I've done the same thing with the /etc/network/interfaces file, the 10GBe cards have a static IP and netmask with no specified gateway or DNS.

 

[ttabbal]

Is the link up on the cards? Even without passing traffic, you should be able to see the cards linked up. ifconfig on linux should show it. The network card status in Windows usually shows the link speed as well...

What's the IP and netmask set to on both interfaces? I used these and it worked well...

1G - 10.0.0.110/24 (255.255.255.0)
10G - 10.0.1.110/24

On the other machine, .111 as the last octet. With that, you should be able to ping between them using IP addresses.

No reason it should be different for 198.x.x.x...

 

[talsit]

I'm getting link speeds in Windows and it passes some traffic.

I ran ping 4 times and I got 75% packet loss, one 25% packet loss and two failure to connect. That's when I did pathping and it's going out to my 1G router, the internet and then coming back. It seems to be using my 1G gateway but there is no gateway set for the 10G subnet.

 

[acquacow]

Have you set your interface metric to be lower on the 10gig network?

If you have automatic metric set, you're going to have a bad time.

 

[talsit]

I tried adjusting it. This one is Win 7 and Server 2012.

 

[acquacow]

10gig network should be something like 10, and gig-e should be more than 10... I use 20 for my gig-e network.

This will ensure that communications try the 10gig network first, if applicable (by route).

 

[talsit]

Set 10G to 10, 1G to 100, still can't get a good ping.

 

[acquacow]

Set 10G to 10, 1G to 100, still can't get a good ping.

Well, pings will be independent of weighting. If you can't get a stable connection, something is up with the path... I'd go back over all of your mtu settings and check your cables.

Sent from my XT1650 using Tapatalk

 

[namike]

We are going to need a ipconfig and route print on both boxes.

 

[talsit]

For the 1G box 1

Address 192.168.3.181
Netmask 255.255.255.0
Gateway 192.168.3.1
DNS 192.168.3.1 8.8.8.8

Box 2
Address 192.168.3.192
Netmask 255.255.255.0
Gateway 102.168.3.1
DNS 192.168.3.1 8.8.8.8

10g box 1
Address 192.168.10.4
Netmask 255.255.255.0

No gateway or DNS

Box 2
Address 192.168.10.5
Netmask 255.255.255.0

No gateway or DNS

Route from 192.168.10.5 is:
192.168.3.1 (router)
192.168.1.1 (FIOS router)
Death

 

[fractal]

It will go out the default route if it thinks the 10G link is down.

Are you using a DAC cable? Do you have a spare? This smells more like a physical layer issue than a link layer.

 

[talsit]

Thanks for the help. I had four cables apparently three were bad...


I got the Mellanox cards talking, good link speeds. I took my known good cable and known working Mellanox card and tried to connect to a SolarFlare dual port card and that's not working. I'll play with that next weekend, it has something to do with the drivers, the SolarFlare Adapter Manager says the drivers aren't loaded even though Device Manager says they are. I may pull that card and put it in a Linux machine and see what I can make happen.