Simple RAID 0 Performance issue - Writes faster than reads?

[Perry]

We had previously been running FreeNAS, but I wasn't getting the kind of performance I wanted from it, so I stripped it down and installed Linux. The plan is to get the RAID volumes working as fast as possible, then use iSCSI with SNS iSANmp to manage the iSCSI shares. All the hardware is set up and running, so I sat down yesterday to do some initial testing.

It's using an older Adaptec 52445 card that we previously had in a workstation where we were getting over 1.5GB/s for reads and writes on DAS. The SAN has a total of 128TB of drives (all 8TB WD Reds). (Other specs: Supermicro X10 motherboard, 64GB ECC RAM, Xeon 1620v3, Chelsio 40GbE NIC, IBM 40GbE switch)

Because we'll be accessing this storage from Windows and macs, I'm doing my initial I/O testing in the AJA Disk Speed application as it simulates our usage pretty nicely. And here's where it gets weird...

In my initial setup I have 4x 8TB drives in a RAID 0. This is just to start benchmarking the hardware in a best-case scenario, before we make RAID 5 pools, which I expect will be slower. In any case, I'm getting solid 800MB/s writes on this setup over SMB from Windows. Not bad for a 4-drive array. In the AJA tool, I get these speeds regardless of the size of the test set (so it's the same whether it's a 4GB test file or a 64GB test file).

My read speeds are in line with writes - basically the same numbers, UNTIL I make the test set 64GB. Then the read speed drops to about 70MB/second - about 11x slower than writes. Occasionally there are faster bursts up to about 200MB/s, but they don't last.

I've tried enabling and disabling read caching on the card, but get basically the same numbers either way. It's also worth noting that in the AJA tool I've tested both single-files and file sequences (that is, one 64GB file, vs a couple thousand smaller ones that amount to 64GB).

Any idea what might be happening here? I'd expect to see slightly faster reads than writes if anything, so something is clearly wrong.

 

[gea]

There are several aspects to consider.

Your AJA workload produces a more sequential type of workload. If you combine that with a hardware raid-0/5 you can expect a similar sequential read/write performance of up to n x disks (raid-0) or n-1 (raid-5). This is not so related to the type of filesystem but ZFS is slower than older filesystems due the additional data because of data checksums and some additional fragmentation due CopyOnWrite.

btw
If you use a more random workload you will detect that a raid-5/Z1 has only the iops of one disk as every disk must be positioned for every read/write. This is why mirror/raid-50 etc is faster as iops scale with number of raids/vdevs.

With ZFS you have a write cache that collects all writes (on Solaris up to 5s, on OpenZFS up to 10% of RAM/max 4GB per default). This means that any type of write workload on ZFS is collected in RAM and written sequentially to disk. Any write perforrmance is quite equal to the sequential write performace (asume enough RAM for ZFS writecaching)

Reads on ZFS are buffered by the rambased ARC that caches small random reads only based on datablocks in a read last/read most mannor. Sequential data is not cached unless you use a L2ARC with read ahead enabled.

This means that for sequential workloads ZFS os mostly faster on writes than reads. This is very different to a random read workload where these data as well as metadata is mostly delivered on ZFS from RAM with a read performance equal to RAM performance.

This differs when you use ZFS software raid instead of hardware raid as this gives more datasecurity, optionally more performance and better ZFS cache control. But basically ZFS remains faster even then on sequential loads on writes than on reads.

see my tests with AJA and other tools at
http://www.napp-it.org/doc/downloads/performance_smb2.pdf
http://napp-it.org/doc/downloads/optane_slog_pool_performane.pdf

btw.
Performance is extremely sensitive to OSX/Windows driver and settings

 

[Perry]

I'm starting to think my issue is an SMB problem, not a RAID issue. I ran some tests using dd on the server to try to simulate what the AJA tool is doing: making a 64GB file, then reading that file in. I'm getting writes near 700MB/S and reads just under 1GB/s, with just a 4-disc RAID0.

With FreeNAS, I was annoyed by the interface, and found that while it probably made sense for someone doing more general purpose work, it didn't seem like a good fit for us. Ultimately, iSCSI is going to be the way we go, with probably 4-6 physical RAIDs, each as its own iSCSI target. with iSANmp, there's no metadata server, it's peer-to-peer. Since we typically have one machine writing to any given share at a time, but occasionally need read access from another machine, iSCSI is a good fit and it eliminates all the overhead of SMB.

Hoping to have something set up and running by the end of the weekend, since I can do most of this from home.

Thanks!

 

[Perry]

I forced samba to use a minimum of SMB2 on the server, and that improved read performance a bit - now I'm getting about 250MB/s pretty consistently. But writes have dropped to 650MB/s since making that change.