ESXi 6.5: Server 2016 VM SSD transfer speed issues

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

snipor

New Member
Feb 5, 2018
2
0
1
34
Hello guys.

I'm having some weird problems with my server's data transfer rate and need your help.

When ever I copying a large file (30GB) from one Seagate SSD to the other Seagate , it starts at full speed from about 400 -450MB/s and then suddenly drops off to around 150-250 MB/s after some written gigabytes, sometimes the speed increases again after while.

My setup:

Supermicro X10SDV-TLN4F Xeon-D 1541 8Core / 16Thread
2x Samsung DIMM 32GB, DDR4-2400, CL17-17-17, reg ECC M393A4K40BB1
Dell Perc H310 flashed with LSI IT firmware version 20.00.07.00 now displays LSI 9211-8i, but in ESXi and Windows it is recognized as DELL SAS 6GB
1xSamsung 840 Basic, 250GB connected to one SATA port of the board
2xSeagate Nytro XF1230 1920GB connected to the H310

ESXi 6.5.0 Update 1 (Build 7388607) is installed on the Samsung SSD.

The unused space on the Samsung SSD is used for VMFS6 DataStore1.

The VM Windows Server 2016 is located on this DataStore.

VMFS6 DataStore2: First Seagate Nytro XF1230 (thick lazy zeroed), Windows E:

VMFS6 DataStore3: Second Seagate Nytro XF1230 (thick lazy zeroed), Windows F:

First some CDM Bench Screens:

c bench2.JPG e bench2.JPG f bench.JPG

Copying the 30GB File from the Samsung to the first Seagate....

c nach e.JPG
.... and to the second Seagate

c nach f.JPG

No problem so far.

But when ever I copying the FIle from one Seagate SSD to the other....

e nach f.JPG f nach e.JPG

What I've done / tried so far:

-RDM

no improvement


-eger zeroed provisioning
no improvement

-eger zerod + paravision
no improvment

-H310 passthrough

e nach f passthrough.JPG

same, 10 minutes later.... strange....

e nach f passthrough2.JPG

-H310 P19 Firmware
no improvement

-LSI 9361-8i JBOD
no improvement

f nach e lsi.JPG

-different amount of RAM
strange
4G:
e nach f 4GB.JPG

16G:
e nach f 24GB.JPG

- other combination
First Seagate connected to the H310 with activated Passthrough
Second Seagate connected to one of the Board Sata Ports, eger zeroed + paravirtual
Different RAM Settings:
4GB:
e nach f (f passthrough, e an sata port, thick + para) 4gb.JPG
8GB:
e nach f (f passthrough, e an sata port, thick + para).JPG
10GB:
e nach f (f passthrough, e an sata port, thick + para) 10gb.JPG
12GB:
e nach f (f passthrough, e an sata port, thick + para) 12gb.JPG
16GB:
e nach f (f passthrough, e an sata port, thick + para) 16gb.JPG


The Drop is not always avail able, sometimes constant speed at about 300MB/s.
The Drop does not always take place at the same amount of written data.
From the Samsung SSD to one of the two Seagate I got always a constant speed about 400MB/s.
High CPU Load during the transfer.

Windows installed directly on the physical machine, constant speed about 450-480MB/s, no drops, no CPU Load????

Could it be a problem with the BIOS settings?
PCIe BIOS.JPG

Sorry for the german screens.


Thanks in advance!
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I would have thought that this might be related to the internal cache of the target drive; being only able to cache so much at some point it is saturated and drops to a lower (sustainable) speed...
Now those Seagate are enterprisel level (read centric) so they should be fine.

But in this case I am not so sure, especially the "Samsung-> Seagate always good" case hints that this might not be the case.
Also the high CPU load is weird.

Can you try with another OS (linux), or run a fio test or somthing similar?
 

snipor

New Member
Feb 5, 2018
2
0
1
34
Thanks for your reply.
I have tried it with a Ubuntu VM (4C/8GB), but here I reach only about 80Mb/s, may there is still a driver issue.

Also I did some tests with fio and the Windows VM.

Settings:
fio settings.JPG

First Seagate connected to the H310 with activated Passthrough.
fio f 1.JPG fio f 2.JPG fio f 3.JPG

Second Seagate connected to one of the Board Sata Ports, eger zeroed + paravirtual

fio e 1.JPG fio e 2.JPG fio e 3.JPG

Should I test again with other settings?

If I compare the Windows chart to the esxtop view, I couldn't see this drop??

esxtop.JPG esxtop2.JPG

Any further tips?
 
Last edited: