Speed writing to a nfs share does not match the speed shown in network reporting

[Foxyz]

Hi guys,

On my linux server i mount a dir using nfs from truenas scale.
When i use dd command writing a 1G file to this dir on the linux server, it says the speed is around 650MB/s, but when i look at the network reporting, it says the max speed is above 9000 Mb/s which is about 1.1GB/s (I’m using a 10G network and a SSD pool, the iperf test shown nothing wrong).

so can anyone tells me why the network traffic is about 2 times than the dd command shows?
And can i make the write speed faster? this is a NVME pool and using dd comand in truenas shell can easily reach above 2GB/s.

by the way, it’s strip pool which only has 1 nvme disk

Thanks a lot!

 

[nexox]

You'll need to post your exact dd command, it's not really a good benchmark in most cases.

 

[unwind-protect]

You need a benchmark that does an fsync(2) at the end at the least.

 

[Foxyz]

You'll need to post your exact dd command, it's not really a good benchmark in most cases.

dd if=/dev/zero of=test.img bs=1GB count=1 oflag=dsync

 

[Foxyz]

You need a benchmark that does an fsync(2) at the end at the least.

the code i'm using:

dd if=/dev/zero of=test.img bs=1GB count=1 oflag=dsync

 

[nexox]

O_SYNC/O_DSYNC don't necessarily work the same on NFS as they do on local storage, the network bandwidth spikes you see are likely due to caching somewhere. You'd probably be better off testing performance with a large file copy followed by executing sync and measuring the combined time, though running an actual workload would tell you more than any synthetic benchmark.

 

[unwind-protect]

O_SYNC/O_DSYNC don't necessarily work the same on NFS as they do on local storage, the network bandwidth spikes you see are likely due to caching somewhere. You'd probably be better off testing performance with a large file copy followed by executing sync and measuring the combined time, though running an actual workload would tell you more than any synthetic benchmark.

fsync(2) on the file is more likely to flush on the server. I don't think sync(1) has any meaning for NFS.

 

[nexox]

fsync(2) on the file is more likely to flush on the server. I don't think sync(1) has any meaning for NFS.

It should at least flush everything out from the client's buffers, so you can measure average network bandwidth, beyond that requires way more understanding of the configuration of the server and the client.

 

[unwind-protect]

It should at least flush everything out from the client's buffers, so you can measure average network bandwidth, beyond that requires way more understanding of the configuration of the server and the client.

fsync(2) should also sync on the server.

 

[UhClem]

dd if=/dev/zero of=test.img bs=1GB count=1 oflag=dsync

Out of curiosity, does

dd if=/dev/zero of=test.img bs=1M count=954 oflag=fdatasync

perform any better?