Yea this sounds like Jumbo Frames type of issues... are you on 1500 MTU? ensure that all VMKernels used for vSAN have 1500 MTU cause a mismatch could cause what you are seeing... another easy way to see is in the cmd line.
start with these to start looking at how they are configured...
check the adapters and what MTU they are reading
esxcli network nic list
or
esxcfg-nics -l
[root@SNODE2V:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address M TU Description
vmnic0 0000:04:00.0 igbn Up 1000Mbps Full 0c:c4:7a:84:4a:7e 1 500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:04:00.1 igbn Down 0Mbps Half 0c:c4:7a:84:4a:7f 1 500 Intel Corporation I350 Gigabit Network Connection
vmnic1000202 0000:02:00.0 nmlx4_en Up 10000Mbps Full f4:52:14:60:53:01 9 000 Mellanox Technologies MT27500 Family [ConnectX-3]
vmnic2 0000:02:00.0 nmlx4_en Down 0Mbps Half f4:52:14:60:53:00 1 500 Mellanox Technologies MT27500 Family [ConnectX-3]
check the kernels on all hosts see what MTU they are reading
esxcli network ip interface list
or
esxcfg-vmknic -l
[root@SNODE2V:~] esxcfg-vmknic -l
Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack
vmk0 Management Network IPv4 192.168.0.26 255.255.255.0 192.168.0 .255 0c:c4:7a:84:4a:7e 1500 65535 true STATIC defaultTcpipStack
vmk0 Management Network IPv6 fe80::ec4:7aff:fe84:4a7e 64 0c:c4:7a:84:4a:7e 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk1 VSAN IPv4 192.168.5.2 255.255.255.248 192.168.5 .7 00:50:56:6b:9e:0d 9000 65535 true STATIC defaultTcpipStack
vmk1 VSAN IPv6 fe80::250:56ff:fe6b:9e0d 64 00:50:56:6b:9e:0d 9000 65535 true STATIC, PREFERRED defaultTcpipStack
next thing i would check is to make sure Multicast isnt getting hit as vSAN uses Multicast to communicate against other nodes as well
do this on all 3 hosts... typically if theres a multicast type issue its in the physical switch.
check your vsan multicast settings.
esxcli vsan network list
if your switch has IGMP Querier turned on for the VLAN/Ports you are using. See for example
Quanta LB6M (10GbE) -- Discussion