ZFS NAS (what to expect, benchmarking,...)

[levak]

Hello!

I posted some other posts about which SLOG to pick for my NAS, but before that task, I got some test gear to perform some benchmarks to see where I stand and what can I expect from the system.

What do I have:
- IBM xServer 3550 M4
- 2x Intel Xeon E5-2640
- 164GB memory
- 2x SAS drives for system
- LSI SAS 9207-8e SAS HBA
- 2x Supermicro SC837E26-RJBOD1 JBOD case for drives
- 42x 4TB Seagate Constellation drives
- 4x Intel S3700 100GB SSDs gor SLOG
- OmniOS as OS

My final configuration will have 52 SAS drives, but for now I have to work with 42 SATA drives.

In this system I can try different configurations, but since I have 42 drives, I'm starting with 7x 6 drives RaidZ2 pool + 2x mirror SSD for SLOG.

My current workload looks like 50% seq / 50% random (according to iopattern dtrace script) and very little of it is sync write, at least according to zilstat (sync write every 4s-5s). Is there a way to see average I/O request size, so I can better define my test scenario?
Storage is mostly used for backups and a few clients use it for active data. Clients also mostly write and almost never read. Reads only happen when data is copied to another omnios server for backup (seq transfer I guess).

All clients connect through iSCSI, so I want to test iSCSI and local speeds.

The biggest problem that I have now is how to accurately measure my system performance, so I have a few questions for you guys:
- What tool do you use? I did some tests with 'fio' tool. Would you recommend it or do you use something else?
- What parameters do you use with 'fio'?
- How to see the actual transfer speed in OmniOS? 'iostat' gives weirdly high numbers, 'zpool iostat' looks a bit more realistic.
- According to the hardware, what speeds should or can I expect? What speeds are expected for seq read/write, what speeds for random read/write? Can I expect to saturate 10Gbps link when doing seq read/write?
- How would you test such configuration?

Matej

 

[Deci]

check out the performance figures i gave in this thread.

https://forums.servethehome.com/index.php?threads/a-not-so-high-density-storage-server.5253/

those figures are using various configurations up to 22 drives per sas card port and 2 sas cards, there will be a fair bit of overheads with all of the drives running from a single card (at 18 drives over the 3 ports/expanders im seeing 150mb/s per drive, at 66 drives im seeing 52mb/s ish per drive, on the plus side these drives are going to live an easy life if thats all they ever have to cope with) , if the host nodes have the PCI-E lanes to run multiple SAS cards and the drives split into a larger number of smaller enclosures/expanders you will see a performance increase, if that increase is neccecary to you is another matter entirely.

after installing and doing the basic setup of omnios you can install nappit ( napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux ownloads ) and that will give you a 30 day full featured trial, you can either purchase 1 or more extensions or just drop back to the basic free package by doing nothing after the 30 days, it includes bonnie++/dd/filebench etc to do some benchmarks on the pools and gives a fairly easy to use/monitor web management interface, (some functions will require shell access but for the most part the web UI has it covered).

 

[levak]

Interesting thread. It seems like I can't get near your results though.

I currently have 2 JBODs attached to a single controller and drives are added to vdev's randomly. I didn't check in which JBOD they are located when I was building vdev's. I have 7 RaidZ2 vdev's with 6x 4TB drives (Seagate Constellation enterprise drives). Sync on the pool is set to disabled, just to rule out ZIL's speed.

As a test tool, I use 'fio' since I think it's the most versatile tool and I can use the same parameters when testing from linux and solaris. I can also select seq/random and read/write tests, blocksize, I/O engine, it writes random data instead of 'zeros' like DD does resulting in a more real-world data...

Parameters I used for the current test:
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=seq-write --rw=write –sync=1
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=seq-read --rw=read –sync=1
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=seq-write --rw=write --bs=128k --sync=1
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=seq-read --rw=read --bs=128k --sync=1

Current results:
4k write: 840MB/s and 210000 iops
4k read: 1300MB/s and 352300 iops
128k write: 1265MB/s and 10200 iops
128k read: 1440MB/s and 11500 iops

I think I'm getting a bit slow speeds, specially for 128k block size. What do you think?

Matej

 

[levak]

I did some more testing with random write/read tests and results are really poor (S3700 100GB is used as SLOG device):
4K random write sync=disabled: 18MB/s - 4600 IOPS
4K random read sync=disabled: 40MB/s - 9000 IOPS
4K random write sync=standard: 18MB/s - 4400IOPS
4K random read sync=standard: 18MB/s - 4700 IOPS

Fio settings:
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=rand-write --rw=randwrite --bs=4k --sync=1
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=rand-read --rw=randread --bs=4k --sync=1
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=rand-write --rw=randwrite --bs=4k –sync=1
fio --runtime=900 --directory=/data/ --fallocate=none --size=20g --refill_buffers --ioengine=solarisaio --iodepth=32 --norandommap --numjobs=12 --group_reporting --name=rand-read --rw=randread --bs=4k –sync=1

 

[levak]

Currently I'm running iSCSI test from 4 clients (all with 10GBit) to OmniOS server. I'm using 128kb block size and doing seq-write with only 500mb/s combined from all hosts... I would expect it to be closer do 1000MB/s, like test on local machine.

Matej

 

[Deci]

is that 18mb/s each or total? what if you only run it from one machine? jumbo frames?

 

[levak]

18mb is when performing test on local machine, not over network...

 

[Deci]

I think you need go break the pool and start testing things in smaller groups.

Are those speeds just to the s3700 drives or to the pool as a whole? Best practice is to not have your slog/l2arc on the same controller/port as the data disks where possible as well. If fewer disks gives you better speeds you are probably running into overhead limits on the one card.

Test just the slog disks on their own, not in a slog sletup just a plain mirror.

Half the drives per jbod, and run tests again, so each one is half filled and remake arrays, don't add the slog, test it as is, add the slog and test again.

See where you are losing your performance.

 

[Entz]

Yeah that is a good approach. Those tests you did have odd results. sync should have no effect on read speeds yet yours dropped massively. I would say something isn't right in your pool or system.