Need some help with 10gbE

[xnoodle]

Connected two machines directly to each other, I can't get traffic from one machine to the other. Can't ping the other from either end. Oddly enough, I can see ARP requests on one end, but not the other.

One is a Linux VM on chassis 1 in my signature with a MNPH29B-XTC passed through running in EN mode, set to 192.168.2.2
The other is a C6100 node running FreeBSD with the Dell TCK99 dual port Intel mezzanine card, set to 192.168.2.3

192.168.0.0/24 is my regular network, 192.168.2.0/24 is what I'm using for this point to point connection.

Linux:

root@purple:/etc/network# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:02:c9:04:45:2c
          inet addr:192.168.2.2  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c9ff:fe04:452c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23 errors:2 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:1889 (1.8 KB)

root@purple:/etc/network# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.0.1     0.0.0.0         UG    100    0        0 eth0
10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
10.8.0.2        *               255.255.255.255 UH    0      0        0 tun0
192.168.0.0     *               255.255.255.0   U     0      0        0 eth0
192.168.2.0     *               255.255.255.0   U     0      0        0 eth1

root@purple:/etc/network# ethtool eth1
Settings for eth1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes

FreeBSD:

[root@freenas ~]# ifconfig ix0                                                  
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500       
        options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU
M,TSO4,VLAN_HWTSO>                                                              
        ether 00:8c:fa:01:d0:5a                                                 
        inet 192.168.2.3 netmask 0xffffff00 broadcast 192.168.2.255             
        media: Ethernet autoselect (autoselect <full-duplex>)                   
        status: active      

[root@freenas ~]# netstat -r                                                    
Routing tables                                                                  
                                                                                
Internet:                                                                       
Destination        Gateway            Flags    Refs      Use  Netif Expire      
default            192.168.0.1        UGS         0   205322   igb1             
localhost          link#9             UH          0   158638    lo0             
192.168.0.0        link#2             U           0    64778   igb1             
192.168.0.129      link#2             UHS         0        0    lo0             
192.168.2.0        link#3             U           0      911    ix0             
192.168.2.3        link#3             UHS         0        9    lo0

 

[nitrobass24]

For PPP in linux you need to set the dest address for the link otherwise it tries to route that traffic to the default gateway and since its in another broadcast domain it doesn't know where to go.

 

[xnoodle]

Alright, tried setting some routes. Also changed IPs to 10.0.2.1 and 10.0.2.2. Slight improvement. I can see ARP requests going out from Linux machine, ARP request received by FreeBSD, ARP reply leaving FreeBSD, but no reply reaching the Linux machine.

11:53:23.499010 ARP, Request who-has 10.0.2.2 tell 10.0.2.1, length 28

11:53:23.288564 ARP, Request who-has 10.0.2.2 tell 10.0.2.1, length 46
11:53:23.288574 ARP, Reply 10.0.2.2 is-at 00:8c:fa:01:d0:5a (oui Unknown), length 28

root@purple:/etc/network# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
10.0.2.2 0.0.0.0 255.255.255.255 UH 0 0 0 eth1
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

[root@freenas ~]# netstat -rn
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.0.1 UGS 1 205359 igb1
10.0.2.1 00:8c:fa:01:d0:5a UHS 0 29 ix0
127.0.0.1 link#9 UH 0 216271 lo0
192.168.0.0/24 link#2 U 0 196487 igb1

 

[dba]

Those are both dual port cards. I'd try one server at a time. Set IP/Netmask/Gateway on both ports, connect port A to port B with an Ethernet cable, and then see if you can pass traffic. If that works, do the same with the other server.

I wouldn't be at all surprised if you found that one of the setups was working as expected and the other was not, probably BSD. Be sure to swap out cables if your first server does not work - they do fail occasionally.

 

[cactus]

For PPP in linux you need to set the dest address for the link otherwise it tries to route that traffic to the default gateway and since its in another broadcast domain it doesn't know where to go.

I have never had to add routes to run P2P in an all Linux environment.

I agree with dba's plan of attack, try a loopback or PortA to PortB.