Poor Low Que Depth Performance with Windows 10 Guest under Proxmox KVM 5.3-8

[am4593]

ok. So I cant make sense of this. I've done some testing running the intel Optane 900p 280gb ssd inside a windows 10 VM under proxmox 5.3-8 and I cant figure out why I'm only getting around 50MB 4K Q1 performance. bare metal performance should be closer to 200Mb/sec, one of the great benefits of running an optane drive is the performance at low Que depths.

I've tried the drive as a ZFS stripe of 1 drive. I've also trying making the pool as LVM. I also directly passed through the drive as a PCI device to the VM and interestingly I still got very similar results.

This system is a Supermicro X9SRL-F with a Xeon E5 1650 and 128GB 1333 Ram.

 

[mbosma]

Hi,

I have similar results on 2 different systems.
First tested on a dual Xeon E5-2670 server. I initialy though it was due to the server being on an older platform.
Also tested on an Atom c3758 system wich gave the same "slow" results.

My Q1t1 caps out around 30-40Mb.

Only thing I have not tried is to lower the ashift on the optane on zfs.
I have been testing on zfs ashift=13 and LVM-thin default.

 

[MiniKnight]

What do you get on a RAM disk?

 

[mbosma]

Very odd.
Mounting a ramdisk from host to the vm via VirtIO gave really similar results to the optane.
Creating a ramdisk in the vm gave way higher results on Q1T1.
c: Optane VirtIO
f: ramdisk VirtIO
h: ramdisk in vm (softperfect)

 

[MiniKnight]

Well at least that helps narrow down the issue.

I wonder what would happen if you created a virtual disk on the Optane drive. If you got better performance, then you could troubleshoot why passthrough is yielding poor performance.

 

[am4593]

So when I ran these original benchmarks I did it by cloning VM's from another storage disk onto the optane 900p. Just for the hell of it I made new win 10 VMs, ran crystaldiskmark and I got much better results. I have no idea why/how the cloned VM's seem to be limited performance wise by the disk that they were cloned from. These are full cloned, not linked. I got around 90/MB a sec on the 4K Q1 which is better, but still below expectations.

 

[mbosma]

I did not experience the same speeds when creating a new vm.
My speeds went down to about 15mb/s on the newly created machine.
Even creating new disks on the machines gave me the same results as the ones originally attached to it.

The only thing I might have done different is grab an older version of the VirtIO drivers since I did not take a look at the version number.
Could that have been an inconsistency in your tests as well?

 

[efschu2]

Which type of VirtIO are you using? I'm getting best results by using VirtIO SCSI Single followed close by VirtIO SCSI, but both gave me way better results in using "only" VirtIO block

 

[mbosma]

I've tries multiple ViortIO setups. VirtIO block, VirtIO SCSI and single.
All gave similar results.
I'll be reinstalling the server again soon for more testing, I'll check how it goes.

 

[mbosma]

Just tried the whole riddle again on zfs and got the same results.

 

[efschu2]

Install ubuntu, configure your kvm command by your own, compare the results to the one proxmox is giving you. I have had some weired performance issues on proxmox too, I doubt about the kvm command proxmox was creating (qm chowcmd <id>) and was playing around by adding my stuff with the args: in the <vmid>.cfg manualy, with this I got rid of my performance issue. But this was about 2 years ago, or so, then after some proxmox updates I recognized on a new created vm my performance issues were gone...
Have you tried in disabling proxmox (cluster) services? I have had some trouble with them making my VMs slow down for some reason.

But yeah, I suggest in doing performance test on an ubuntu w/o all proxmox stuff. Then you may come closer to the problem.

 

[mbosma]

I created a thread on the proxmox forums wich gave me a lot more insight in this problem.

make kinda sense, since qemu schedules the io of the disks with the main thread, and i guess you get cpu bottlenecked by your atom/7-year old cpu
you could try to enable 'iothreads' (do not forget to change the scsi controller to virtio-scsi-single) and see if the result improves

This makes total sense given our 'older' cpu's. It also epxlains why our setups yield different results in speed.

 

[efschu2]

Well a Xeon E5 1650 should still be more then capable? Beside a lil' more clock, a lil' more cache/ramspeed not much changed for Q1T1 compared to new Xeons.

 

[mbosma]

I've tried this using an Intel Atom c3758 as well as a dual E5-2670 system.
The Xeon system did give me better results than the Atom.