10GBe direct connection issues

Discussion in 'Networking' started by talsit, Feb 24, 2017.

  1. talsit

    talsit Member

    Joined:
    Aug 8, 2013
    Messages:
    112
    Likes Received:
    20
    I am trying to direct connect 10GBe cards and having a heck of a time.

    I'm connecting short runs via SFP, two ConnectX2 cards. They occasionally see each other.

    1 GB network is 192.168.3.xxx
    10GB network is 192.168.10.xxx with manual IPs set.

    In the 10GBe network all I have specified in Windows Networking is the IP and netmask, I left gateway and DNS blank.

    When I run pathping on my 10GBe connection it's going out on my 192.168.3.xxx network and searching the internet.

    I have another server with 10GBe cards but they are Intel 520's and I'm trying to eliminate any brand incompatibility. I haven't been able to get the ConnectX2 cards to talk to the Intel cards, but I want to make sure my setup is correct before I try to dig into that anymore. The Intel cards are in an Ubuntu machine and I've done the same thing with the /etc/network/interfaces file, the 10GBe cards have a static IP and netmask with no specified gateway or DNS.
     
    #1
  2. ttabbal

    ttabbal Active Member

    Joined:
    Mar 10, 2016
    Messages:
    704
    Likes Received:
    186
    Is the link up on the cards? Even without passing traffic, you should be able to see the cards linked up. ifconfig on linux should show it. The network card status in Windows usually shows the link speed as well...

    What's the IP and netmask set to on both interfaces? I used these and it worked well...

    1G - 10.0.0.110/24 (255.255.255.0)
    10G - 10.0.1.110/24

    On the other machine, .111 as the last octet. With that, you should be able to ping between them using IP addresses.

    No reason it should be different for 198.x.x.x...
     
    #2
  3. talsit

    talsit Member

    Joined:
    Aug 8, 2013
    Messages:
    112
    Likes Received:
    20
    I'm getting link speeds in Windows and it passes some traffic.

    I ran ping 4 times and I got 75% packet loss, one 25% packet loss and two failure to connect. That's when I did pathping and it's going out to my 1G router, the internet and then coming back. It seems to be using my 1G gateway but there is no gateway set for the 10G subnet.
     
    #3
  4. acquacow

    acquacow Active Member

    Joined:
    Feb 15, 2017
    Messages:
    375
    Likes Received:
    164
    Have you set your interface metric to be lower on the 10gig network?

    If you have automatic metric set, you're going to have a bad time.
     
    #4
  5. talsit

    talsit Member

    Joined:
    Aug 8, 2013
    Messages:
    112
    Likes Received:
    20
    I tried adjusting it. This one is Win 7 and Server 2012.
     
    #5
  6. acquacow

    acquacow Active Member

    Joined:
    Feb 15, 2017
    Messages:
    375
    Likes Received:
    164
    10gig network should be something like 10, and gig-e should be more than 10... I use 20 for my gig-e network.

    This will ensure that communications try the 10gig network first, if applicable (by route).
     
    #6
  7. talsit

    talsit Member

    Joined:
    Aug 8, 2013
    Messages:
    112
    Likes Received:
    20
    Set 10G to 10, 1G to 100, still can't get a good ping.
     
    #7
  8. acquacow

    acquacow Active Member

    Joined:
    Feb 15, 2017
    Messages:
    375
    Likes Received:
    164
    Well, pings will be independent of weighting. If you can't get a stable connection, something is up with the path... I'd go back over all of your mtu settings and check your cables.

    Sent from my XT1650 using Tapatalk
     
    #8
  9. namike

    namike Member

    Joined:
    Sep 2, 2014
    Messages:
    62
    Likes Received:
    16
    We are going to need a ipconfig and route print on both boxes.
     
    #9
  10. talsit

    talsit Member

    Joined:
    Aug 8, 2013
    Messages:
    112
    Likes Received:
    20
    For the 1G box 1

    Address 192.168.3.181
    Netmask 255.255.255.0
    Gateway 192.168.3.1
    DNS 192.168.3.1 8.8.8.8

    Box 2
    Address 192.168.3.192
    Netmask 255.255.255.0
    Gateway 102.168.3.1
    DNS 192.168.3.1 8.8.8.8

    10g box 1
    Address 192.168.10.4
    Netmask 255.255.255.0

    No gateway or DNS

    Box 2
    Address 192.168.10.5
    Netmask 255.255.255.0

    No gateway or DNS

    Route from 192.168.10.5 is:
    192.168.3.1 (router)
    192.168.1.1 (FIOS router)
    Death
     
    #10
  11. fractal

    fractal Active Member

    Joined:
    Jun 7, 2016
    Messages:
    309
    Likes Received:
    67
    It will go out the default route if it thinks the 10G link is down.

    Are you using a DAC cable? Do you have a spare? This smells more like a physical layer issue than a link layer.
     
    #11
  12. talsit

    talsit Member

    Joined:
    Aug 8, 2013
    Messages:
    112
    Likes Received:
    20
    Thanks for the help. I had four cables apparently three were bad...


    I got the Mellanox cards talking, good link speeds. I took my known good cable and known working Mellanox card and tried to connect to a SolarFlare dual port card and that's not working. I'll play with that next weekend, it has something to do with the drivers, the SolarFlare Adapter Manager says the drivers aren't loaded even though Device Manager says they are. I may pull that card and put it in a Linux machine and see what I can make happen.
     
    #12

Share This Page