Painfully slow Storage vMotion Migrations?

[IamSpartacus]

I'm trying to determine where my bottleneck is as I'm seeing some very poor performance when doing a Storage vMotion migration (migrating VM's to a different host and datastore) between my 2 ESXi hosts. I don't believe it's a networking issue based on the following:

1. iperf between hosts via the vMotion IPs (separate vlan from management) is 9Gbps+
2. While overall the transfers are very slow, there are rare (usually towards the end) bursts over 8Gbps.
3. Copying files from one VM on one host, to another VM on another host is over 400MB/s

And that last fact is what makes it hard to believe it's storage related. Both datastores are RAID10 arrays consisting of 4 SSD's each.

I've tried dual NIC vMotion and I've tried just limiting the transfers to using a single one of my 10Gb NICs (tried both) on each server to no avail.

What else should I be looking at?

 

[Rand__]

What disks are in use?

And when you copy files inside the VM - what filesize?

 

[IamSpartacus]

What disks are in use?

What's the best way to determine this?

And when you copy files inside the VM - what filesize?

Files are several GB in size that I've tested.

 

[Rand__]

so out of cache size I assume.
Is it constantly slow or quick initially and then slows down?

 

[IamSpartacus]

so out of cache size I assume.
Is it constantly slow or quick initially and then slows down?

Slow right off the bat and pretty much the whole way until the very end.

 

[markarr]

The burst you see at the end is the memory sync it does before flipping the vm from host to host, so it sounds like the vm vmotion is fine but the storage side is wonky. If the file transfer between datastores are quick but vmotion is not, then it sounds like some weird config issue.

Is the vmkernel on the same vswitch as everything else?

 

[IamSpartacus]

The burst you see at the end is the memory sync it does before flipping the vm from host to host, so it sounds like the vm vmotion is fine but the storage side is wonky. If the file transfer between datastores are quick but vmotion is not, then it sounds like some weird config issue.

Is the vmkernel on the same vswitch as everything else?

The memory sync will show up across the NIC? Interesting.

I don't know how it could be a config issue as right now my setup is as vanilla as it comes. I literally just installed vsphere, installed vCenter, setup the vMotion vmks and off I went.

 

[realtomatoes]

i generally see these on high RAM and high activity VMs like databases and exchange. vmdks are all copied but the last 30-ish percent takes some time. that's usually when the active ram sync and switch over happens. what's the total svmotion duration like and how huge are the VMs?

 

[IamSpartacus]

i generally see these on high RAM and high activity VMs like databases and exchange. vmdks are all copied but the last 30-ish percent takes some time. that's usually when the active ram sync and switch over happens. what's the total svmotion duration like and how huge are the VMs?

The tests I've done have not been with high use VM's at all. In fact, I get the same results whether the VM's are on or off for the most part. I've tested with small VM's around 20GB all the way up to larger ones around 250GB.

 

[realtomatoes]

great isolation you did there. yeah, looks like something the storage is doing or not doing holding back your speeds.

 

[azev]

have you tried comparing vmotioning vm while they are power off compared to power on ?? I noticed significant better performance when the VM is offline.

 

[IamSpartacus]

have you tried comparing vmotioning vm while they are power off compared to power on ?? I noticed significant better performance when the VM is offline.

See my last post.

 

[Rand__]

Checked this? ESXI 6.5 Storage Performance Issues and Fix - opvizor

Or are you on 6.5 vanilla or updated?

 

[IamSpartacus]

Checked this? ESXI 6.5 Storage Performance Issues and Fix - opvizor

Or are you on 6.5 vanilla or updated?

I'm on the latest 6.5U1. Was this issue patched in the latest update? If not I'll have to look into this.

 

[Rand__]

Hm after rereading I think this was the SATA issue which a.) should have been fixed and b.) would not apply.
But maybe it gives a new search vector...