Benchmarking read speed without hitting the network

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

tinco

New Member
Apr 22, 2020
27
1
3
Workstation: 1 NVMe 2TB drive, 10gbit NIC (Aquantia AQC107)
NAS: 8 SATA 2TB SSD's in RAID 5 2x10gbit NIC (Intel) EPYC 7282 with 32GB ram
Network: Unifi 16XG switch has the NAS connected directly on 2 bonded 10gbit ports (so 20gbit in theory), it's connected to a XS508M 10gbit switch, which is connected to the workstation with a single link

I'm trying to figure out why the transfer speed of our NAS to a workstation is limited to ~200MB/s. I've already ruled out local disk speed on the workstation, copying the same files from one dir to the other is at NVMe theoretic max speed, which sounds reasonable to me. So now the question is, can the NAS not read faster from its filesystem, or can the network not transfer faster.

So next step that should be easy but I couldn't find any decent tools is to figure out the filesystem performance. Anyone know of a tool that will read a bunch of files into memory or into /dev/null or whatever? All I find is tools that benchmark block devices, I'm trying to gauge real world performance.

Advice on a better solution than the dmraid RAID5 is welcome, but I'm trying to explore why it's not hitting the theoretical maximums before rearranging everything.
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
What is OS running on the workstation? Windows 10? Are you using SMB to connect to your (diy?) NAS?
 

tinco

New Member
Apr 22, 2020
27
1
3
Oh yeah sorry, the workstation is Windows 10 Pro, the server is Ubuntu Linux running smbd, those are some important details obviously.

I was incorrect about the NAS NIC, it's Broadcom not Intel, full specs are here: G221-Z30 (rev. 100) | GPU Servers - GIGABYTE Global we got a BIOS patch to have it run a 2nd gen EPYC CPU. It's got a graphics card as well but it wasn't doing any GPU tasks when I was benchmarking.
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
did you disable RSS on the windows side?
netsh int tcp set global rss=disabled
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
As Rand says, running some quick fio tests against your storage array will give you an idea of the maximum speeds they're capable of.

If you're using dm-raid with parity, have a think about increasing your stripe cache size, the typical default of 256 can be quite limiting even with platter-based drives let alone SSDs.

Assuming you're using windows explorer on the client side and samba on the server side though, I suspect you might be hitting the limits of a single CPU somewhere. My own old workstation wasn't capable of sustaining more than 200MB/s in explorer but could easily hit 450MB/s using multithreaded robocopy. Use perfmon or task manager or similar on the windows side and something like htop on the linux side to see if you have any cores pegged at ~100% during transfers.