DUP! on ping in ESXi environment (dvSwitch) (solved)

[Rand__]

Hi,
just wanted to share (one possible) root cause of DUP!'s when using dvSwitches. Been looking for the error for a while but never found it, there is little info out there so to help (others and myself the next time) I thought to put it out there.

Problem - running ping to/from *nix based VMs caused weird DUP! messages.
This occurred only when pinging outside the particular ESXi host, locally it was fine (sexigraf and freenas running on the same host, dc on another one).

FROM Freenas BOX
----------------------------
ping mydc  (Running on another ESX host)
PING mydc.mydom.local (10.10.0.2): 56 data bytes
64 bytes from 10.10.0.2: icmp_seq=0 ttl=128 time=0.114 ms
64 bytes from 10.10.0.2: icmp_seq=0 ttl=128 time=0.123 ms (DUP!)
64 bytes from 10.10.0.2: icmp_seq=0 ttl=128 time=0.139 ms (DUP!)
64 bytes from 10.10.0.2: icmp_seq=0 ttl=128 time=0.142 ms (DUP!)
64 bytes from 10.10.0.2: icmp_seq=1 ttl=128 time=0.232 ms
64 bytes from 10.10.0.2: icmp_seq=1 ttl=128 time=0.245 ms (DUP!)
64 bytes from 10.10.0.2: icmp_seq=1 ttl=128 time=0.250 ms (DUP!)
64 bytes from 10.10.0.2: icmp_seq=1 ttl=128 time=0.255 ms (DUP!)
--- mydc.mydom.local ping statistics ---
11 packets transmitted, 11 packets received, +33 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 0.114/0.286/0.412/0.066 ms
root@freenas:~ # ping 10.10.1.34 (ESX host that is not hosting freenas)
PING 10.10.1.34 (10.10.1.34): 56 data bytes
64 bytes from 10.10.1.34: icmp_seq=0 ttl=64 time=0.213 ms
64 bytes from 10.10.1.34: icmp_seq=0 ttl=64 time=0.221 ms (DUP!)
64 bytes from 10.10.1.34: icmp_seq=1 ttl=64 time=0.237 ms
64 bytes from 10.10.1.34: icmp_seq=1 ttl=64 time=0.245 ms (DUP!)
^C


root@freenas:~ # ping 10.10.1.37 (ESX host that is hosting freenas)
PING 10.10.1.37 (10.10.1.37): 56 data bytes
64 bytes from 10.10.1.37: icmp_seq=0 ttl=64 time=0.140 ms
64 bytes from 10.10.1.37: icmp_seq=0 ttl=64 time=0.148 ms (DUP!)
^C


root@freenas:~ # ping freenas (self)
PING freenas.mydom.local (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.028 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.029 ms
^C


root@freenas:~ # ping sexigraf (Running on same ESX host)
PING sexigraf.mydom.local (10.10.1.5): 56 data bytes
64 bytes from 10.10.1.5: icmp_seq=0 ttl=64 time=0.109 ms
64 bytes from 10.10.1.5: icmp_seq=1 ttl=64 time=0.116 ms
64 bytes from 10.10.1.5: icmp_seq=2 ttl=64 time=0.137 ms
^C

From ESX Host 
---------------------
[root@esx2:~] ping sophos (Running on another ESX host)
PING sophos (10.10.0.1): 56 data bytes
64 bytes from 10.10.0.1: icmp_seq=0 ttl=64 time=0.253 ms
64 bytes from 10.10.0.1: icmp_seq=0 ttl=64 time=0.312 ms (DUP!)

[root@esx2:~] ping freenas (Running on this ESX host)
PING freenas (10.10.1.32): 56 data bytes
64 bytes from 10.10.1.32: icmp_seq=0 ttl=64 time=0.168 ms
64 bytes from 10.10.1.32: icmp_seq=0 ttl=64 time=0.234 ms (DUP!)


[root@esx2:~] ping esx1 (another ESX host - no problem when running physical only)
PING esx1 (10.10.1.34): 56 data bytes
64 bytes from 10.10.1.34: icmp_seq=0 ttl=64 time=0.435 ms
64 bytes from 10.10.1.34: icmp_seq=1 ttl=64 time=0.252 ms

[root@esx2:~] ping sexigraf (Running on this ESX host)
PING sexigraf (10.10.1.5): 56 data bytes
64 bytes from 10.10.1.5: icmp_seq=0 ttl=64 time=0.366 ms
64 bytes from 10.10.1.5: icmp_seq=0 ttl=64 time=0.413 ms (DUP!)

The most common hint online was that this might be related to promiscuous mode of the dvSwitch, but that was off. Next on the line was a box that had routing between interfaces active - also didnt check.

In the end it turned out to be the 'Beacon probing' on the Teaming and Failover config. When I turned that back to Link detection all problems went away.