[SOLVED] ESXI 6.7 one ubuntu VM not performing (slow download; packet loss)

[MartinM]

edit: SOLVED, was a bad cable..

Hi,
i tried googling to no avail. I am running multiple VMs on host on Supermicro X10SDV , Xeon 1540d, with 128GB ECC ram. I have two Ubuntu VMs beside others on this host, one general purpose (running docker, plex and other services - this is the affected VM) and the other not really running anything. I have not performed any configuration or hardware changes in either machine or the host. Both the Ubuntu VMs are 18.04LTS, with patches applied ~10days ago. Neither is under significant load.

Issue #1 (solved, likely caused by #2)
one of the Ubuntu VMs threw couple errors when trying to download updates through sudo apt-get update, sudo apt-get upgrade resulting in Hash Sum mismatch, when trying to download newest updates (see below). I googled for solutions, found couple tips with sudo apt-get clean or removing the apt list. However that did not resolve the issue. I ultimately downloaded the package throwing error on a different Windows VM on the same machine, checked the hash and it was correct, I then manually installed the package on the Ubuntu system with no hiccups using dpkg. I consider this solved.

Issue #2
- secondary issue, which i noticed afterwards was extremely slow WAN speed, simple wget of ubuntu iso (as a test) is approx 200KB/s, while i have 225mbit down/up. Again, using different VM the speed was very close to the usual 200mbit. Tried different mirrors as well.
- I've installed speedtest in cli, running the speedtest back to back in different Ubuntu VMs yields completely different results. (with identical Network setup in VMware and slightly different iptables setup - due to docker being installed in the affected ubuntu; no manual entries added)
- iperf is fine and yields expected results.
- I've run iperf through wireguard tunnel (on different machine in same LAN as affected Ubuntu VM) to a different site (over WAN) and the results are in the range of 200mbit.. So the connection should


This points me to issue in ESXi, but ONLY in one VM. How can ESXi result in packet loss and to WAN only? Is this Ubuntu software issue? I could likely try to restore previous backup of the affected VM and I will (didnt have time yet as i need to make space for the restore) but I would like to find out what is going on.


Thank you for any help or tips!

Speedtest affected Ubuntu VM (packet loss is sometimes there, sometimes not, however speed test is always bad)

    Latency:     2.17 ms   (104.08 ms jitter)
   Download:     6.29 Mbps (data used: 8.7 MB)
     Upload:   145.81 Mbps (data used: 224.9 MB)
Packet Loss:    12.3%

Speedtest unaffected Ubuntu VM

  Latency:     2.27 ms   (0.50 ms jitter)
   Download:   219.78 Mbps (data used: 199.9 MB)
     Upload:   220.00 Mbps (data used: 224.7 MB)
Packet Loss:     0.0%

Update issue

panda@box:~$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  linux-firmware
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 75.1 MB of archives.
After this operation, 10.2 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-firmware all 1.173.18 [75.1 MB]
Err:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-firmware all 1.173.18
  Hash Sum mismatch
  Hashes of expected file:
   - SHA256:099bbc0f958db43fad328f9f09065e448a85dcdb126691c2dab040213af13e41
   - SHA1:31b98ad6e6d2ca49e1a5297d9913d0be41f2a965 [weak]
   - MD5Sum:60b003009991844b6f8cd73bc185989a [weak]
   - Filesize:75056824 [weak]
  Hashes of received file:
   - SHA256:0c404bce54ff28a87300f0ba933726ea7bc7008cc1edd4b8586f2f93a5bab2e8
   - SHA1:d5d35d984788d6e25b10abdaf09f33614dd1fa72 [weak]
   - MD5Sum:94495275aa23904b065cb37185d35b8e [weak]
   - Filesize:75056824 [weak]
  Last modification reported: Wed, 06 May 2020 15:18:25 +0000
Fetched 75.1 MB in 6s (12.6 MB/s)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/l/linux-firmware/linux-firmware_1.173.18_all.deb  Hash Sum mismatch
   Hashes of expected file:
    - SHA256:099bbc0f958db43fad328f9f09065e448a85dcdb126691c2dab040213af13e41
    - SHA1:31b98ad6e6d2ca49e1a5297d9913d0be41f2a965 [weak]
    - MD5Sum:60b003009991844b6f8cd73bc185989a [weak]
    - Filesize:75056824 [weak]
   Hashes of received file:
    - SHA256:0c404bce54ff28a87300f0ba933726ea7bc7008cc1edd4b8586f2f93a5bab2e8
    - SHA1:d5d35d984788d6e25b10abdaf09f33614dd1fa72 [weak]
    - MD5Sum:94495275aa23904b065cb37185d35b8e [weak]
    - Filesize:75056824 [weak]
   Last modification reported: Wed, 06 May 2020 15:18:25 +0000
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

 

[vangoose]

How many uplinks do you have in the vswitch? Could be one bad nic/cable if you have more than one uplinks and one vm is going through that bad nic/cable.

 

[MartinM]

Only one. There are three on the board (IPMI and two Gbit NICs), i am actively and always have used only one.

I have another connectx3 with 2xSFP+ in there as well, but that serves only as direct connection to my workstation and is pass through to Unraid VM.

One thing I remembered is that ESXI / Vcenter switched the NIC to 100mbit speed before. I had to manually switch it back to 1000mbit, but it worked fine and if it really was hardware issue it would affect all VMs wouldnt it? I will try replacing the patch cabel tomorrow.


edit: It seems to have been the cable. after replacing it, everything seems to be OK.. I still don't understand how it could seemingly affect only one VM. Now to recheck what it affected (backups, media uploads etc.. :|)