hyperV Guests Slow as a 486

[AFisher]

I run a pretty extensive lab for work related endeavors, but generally as of late the VMs have come to a screech halt... below are my specs, let me know what you what consider looking into...
Storage
FreeNas 11.3-U4.1 Running on a SuperMicro x9DR3-F with 2 intel E52620 Xeons and 128G of DDR3 Ram, 10G network for storage, 1G network for management. 20, 4TB SAS 6G HGST HUS726040ALS214 drives
Hyperv Hosts
Windows Server 2016, running on SuperMicro X9DRi-LN4+ with 2 Xeon E5-2630 CPUs, and 256G DDR3 ram, boots from Samsung 850 SSD, 10G NIC for storage, 1G for management

Unifi 10G (fiber) switch for storage, cisco 2960 for management

I run anywhere between 10 and 50 hyperV vms, generally I assign 4G of ram, and 2 vCPU... VMs run anything from windows 7 to Ubuntu 20!

over the last six months when I log in, the VMs have been getting more and more unresponsive. I have done several VM rebuilds nothing changed, I have done full patching, and short of a 100% rebuild I'm just lost. I Thought this might be related to 10G bottleneck, but I never hit more than 30 or 40MB on the ports\over the switch. I even moved VMs to local storage and no real change!

so, what are you thoughts? I don't want to change hypervisors due to customer needs(I replicate my customers infrastructure for testing and dev)

 

[zedascuras]

Hi, you give a lot of information about your setup, but you don't mention how your virtual machines are connected to the network, are you using the management network to share the network connectivity to your vm''s, or do you have a dedicated interface for the traffic?
Also, how are you presenting the storage from the freenas to the hyper-v host, is it block storage(iscsi), or file storage(smb, nfs)?

 

[Dreece]

Ok, so we can rule out storage as you've moved to local disks and no difference. Everything is just real slow?

The only thing I can come up with based on your report is that the host has issues. What you can do is first do a backup of the host, then disable hyper-v (not uninstall) via bcdedit, install virtualbox and create a new vm in that, if it runs fine, then we can say the issue is starting to look a hyper-v configuration issue. The next thing I would do is re-install hyper-v, create a new vm and just make sure its fine, if it is, then import the existing vms one at a time and see how they all function.

I recently lost access to hyper-v management, and I was literally pulling what little hair I have left out of me scalp, in the end I found some strange line of text I had to type into powershell which re-configured some registry settings to do with hyper-v, and then management was working again. The strange fact here is that I've had this Windows Hyper-V box running ever since 2008 and has gone through all the upgrades and not once missed a heartbeat, so something (Microshaft done it via update I feel!) screwed something up and then I hit google search on a whole different level.