Help me benchmark some MegaRAID controllers...

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
Hi folks,

Here's what I've currently got:
  • NEC 9267-8i controller with BBU
  • Six 3TB WD Red HDDs
  • One 500GB Samsung 850 EVO SSD
  • Ubuntu 17.04 machine, 16GB RAM, i7-4790k CPU (just consumer stuff)
I thought the 9267-8i would do CacheCade, but it turns out I was wrong.

This is for a home Plex/virtualbox/Samba box. I'd like to get a CacheCade setup because... well, I like playing with computer stuff. So practicality takes a bit of a back seat if the price is right.

What's an inexpensive way to get a CacheCade/RAID5/6 setup? It looks like I can get an LSI 9260-8i+LSI00292 hardware key for under $80. But...
  • Would spending an extra $60ish on an 9265-8i+LSI00290 hardware key yield noticeable improvement with these drives?
  • Are there other CacheCade-capable controllers (perhaps Dell/IBM/NEC-branded) that are good deals at this performance level?
Right now this box is free, so I'm happy to do some benchmarking. Maybe I can buy another controller or two and test them at various RAID0/5/6 with/without CacheCade configurations. If you give the guidance, I'm happy to do the repeated hardware and OS installs, etc., and share my findings.
 
Last edited:

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
Forgot to mention...

The reason for this is, I've asked several questions recently and this community has been incredibly helpful. I'd like to volunteer my time and resources to say thanks and give back.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
Would spending an extra $60ish on an 9265-8i+LSI00290 hardware key yield noticeable improvement with these drives?
I think you will get some improvements on small, random read iops. (For write caching you will need redundancy or parity).

I don't have experience with cachecade, but with maxcache (adaptecs ssd caching). In maxcache the raid controller constantly reads and writes to the ssds (=> sustained workload). Your 850 evo is not made for such a use case and the performance of your whole arrray will degrade once the cache in the ssd is filled up.
 

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
I think you will get some improvements on small, random read iops. (For write caching you will need redundancy or parity).

I don't have experience with cachecade, but with maxcache (adaptecs ssd caching). In maxcache the raid controller constantly reads and writes to the ssds (=> sustained workload). Your 850 evo is not made for such a use case and the performance of your whole arrray will degrade once the cache in the ssd is filled up.
Thanks i386, I appreciate your cautions. This is largely just an excuse to play with CacheCade using a large dataset (uhh, my own stuff). I might or might not keep that configuration. The contents will always be backed up. In the longer run, I might run the SSD directly as a system disk (as you previously recommended) or move to a NAS-based solution instead.

What's the STH-blessed way of testing disk performance in linux? I've seen STH posts discussing development of such a benchmark, but nothing about its completion or usage.
 

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
Are there "standard" fio job files for measuring things like sequential R/W, random R/W, and IOPS?

Edit: currently playing with the following commands:

Code:
# random read/write/readwrite
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randread --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randread -time_based -runtime=600
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randwrite --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randwrite -time_based -runtime=600
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randrw --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randrw --rwmixread=75 -time_based -runtime=600

# sequential read/write
fio --direct=1 --gtod_reduce=1 --name=read --filename=test --bs=64k --size=16G --readwrite=read
fio --direct=1 --gtod_reduce=1 --name=write --filename=test --bs=64k --size=16G --readwrite=write
 
Last edited:

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
Okay, I think I'm going to go with the following fio/iozone commands:


Code:
#!/bin/bash
rpath=/results/6x_3TB_RAID0_stripe64k_readahead

mkdir $rpath
pushd /test

# random read/write/readwrite
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randread --filename=test --bs=4k --iodepth=64 --size=8G --readwrite=randread -time_based -runtime=60 | tee $rpath/fio_randread.txt
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randwrite --filename=test --bs=4k --iodepth=64 --size=8G --readwrite=randwrite -time_based -runtime=60 | tee $rpath/fio_randwrite.txt
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randrw --filename=test --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 -time_based -runtime=60 | tee $rpath/fio_randrw.txt

# sequential read/write
fio --direct=1 --gtod_reduce=1 --name=read --filename=test --bs=64k --size=16G --readwrite=read -runtime=120 | tee $rpath/fio_seqread.txt
fio --direct=1 --gtod_reduce=1 --name=write --filename=test --bs=64k --size=16G --readwrite=write -runtime=120 | tee $rpath/seqwrite.txt

# iozone
iozone -a -I -Mce -n 1M -g 8G -Rb $rpath/iozone_all_try1.xls | tee $rpath/iozone_all_try1.txt
iozone -a -I -Mce -n 1M -g 8G -Rb $rpath/iozone_all_try2.xls | tee $rpath/iozone_all_try2.txt
iozone -a -I -Mce -n 1M -g 8G -Rb $rpath/iozone_all_try3.xls | tee $rpath/iozone_all_try3.txt


iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 1 -Rb $rpath/iozone_thr_t1.xls | tee $rpath/iozone_thr_t1.txt
iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 2 -Rb $rpath/iozone_thr_t2.xls  | tee $rpath/iozone_thr_t2.txt
iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 4 -Rb $rpath/iozone_thr_t4.xls  | tee $rpath/iozone_thr_t4.txt
iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 8 -Rb $rpath/iozone_thr_t8.xls  | tee $rpath/iozone_thr_t8.txt

popd
ls -ltr $rpath
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
Are there "standard" fio job files for measuring things like sequential R/W, random R/W, and IOPS?

Edit: currently playing with the following commands:

Code:
# random read/write/readwrite
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randread --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randread -time_based -runtime=600
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randwrite --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randwrite -time_based -runtime=600
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randrw --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randrw --rwmixread=75 -time_based -runtime=600

# sequential read/write
fio --direct=1 --gtod_reduce=1 --name=read --filename=test --bs=64k --size=16G --readwrite=read
fio --direct=1 --gtod_reduce=1 --name=write --filename=test --bs=64k --size=16G --readwrite=write
I think these numbers are a little bit unrealistic for a home plex/fileserver serving most of the time large files. Try it with qd <= 8 and 90(+)% read/10%write.
 

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
I think these numbers are a little bit unrealistic for a home plex/fileserver serving most of the time large files. Try it with qd <= 8 and 90(+)% read/10%write.
Thanks! Mostly I am looking to get benchmark data useful for you guys, not as much for me.

I've almost got it scripted up to create a bunch of RAID array configurations using storcli (successor to megacli), create a fresh filesystem each time, and run a bunch of benchmarks on it. The issue I'm fighting now is that after doing the array initialization via storcli, sometimes the array ends up on /dev/sda and sometimes it ends up on /dev/sdc, and I can't figure out a way to make it always land on /dev/sda.
 

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
Okay here's my main script, do.sh:

Code:
#!/bin/bash
if [ "$EUID" -ne 0 ]
  then echo "Please run as root"
  exit
fi


STRIPE_LIST='256'
IOPOLICY_LIST='direct cached'
PDCACHE_LIST='on off'

unset ARRAY_LIST
declare -A ARRAY_LIST
ARRAY_LIST[raid0_x6]='storcli64 /c0 add vd type=raid0 name=test drives=252:0-5 ra'
ARRAY_LIST[raid5_x6]='storcli64 /c0 add vd type=raid5 name=test drives=252:0-5 ra'
ARRAY_LIST[raid6_x6]='storcli64 /c0 add vd type=raid6 name=test drives=252:0-5 ra'
ARRAY_LIST[raid5_x5]='storcli64 /c0 add vd type=raid5 name=test drives=252:0-4 ra'
ARRAY_LIST[raid10_x6]='storcli64 /c0 add vd type=raid10 name=test drives=252:0-5 pdperarray=2 ra'
ARRAY_LIST[raid50_x6]='storcli64 /c0 add vd type=raid50 name=test drives=252:0-5 pdperarray=3 ra'

======

# make all array combinations
unset configs_to_test
declare -A configs_to_test
for stripe in $STRIPE_LIST
do
 for iopolicy in $IOPOLICY_LIST
 do
  for pdcache in $PDCACHE_LIST
  do
   for array_key in "${!ARRAY_LIST[@]}"
   do
    array_config=${ARRAY_LIST[$array_key]}" strip=$stripe $iopolicy pdcache=$pdcache"
    key=$array_key"_stripe${stripe}_iopolicy${iopolicy}_pdcache${pdcache}"
    configs_to_test[$key]=$array_config
   done
  done
 done
done



# report the configurations queued for test
echo "Configurations to test:"
for key in "${!configs_to_test[@]}"
do
 echo "  $key: ${configs_to_test[$key]}"
done
echo ""


# test all the configurations
for key in "${!configs_to_test[@]}"
do
 echo "NOW TESTING $key: ${configs_to_test[$key]}"

 storcli64 /c0/vall del force
 storcli64 /c0 flushcache

 eval ${configs_to_test[$key]}
 storcli64 /c0/v0 set wrcache=awb
 storcli64 /c0/v0 start init force  ;# force even if there's a mounted filesystem from last iteration

 while [[ $(storcli64 /c0/v0 show init) != *Not* ]]
 do
  echo "Waiting for array to initialize (`date`)"
  sleep 1
 done
 storcli64 /c0/v0 show init
 storcli64 /c0 flushcache
 parted -l  ;# verify the array landed on some device

 rm -rf ./$key
 mkdir -p ./$key
 pushd $key

 time ../bench.sh 2>&1 | tee all_tests.log

 popd
 echo "TESTING COMPLETE FOR $key: ${configs_to_test[$key]}"
done
You configure the four *_LIST variables at the top, then run the script using sudo. It assumes storcli64 is in your search path, and it uses it to create each virtual disk of interest. Then for each disk, it makes a results directory within the current directory, goes into it, and runs the bench.sh script.
 
Last edited:

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
Here's the bench.sh script:
Code:
#!/bin/bash

# this is the controller, as reported by 'parted -l'
# (used to find out on which device the array landed)
CONTROLLER='LSI MR9267-8i'

# this is the test filesystem mountpoint
TESTFS='/test'


################


if [ "$EUID" -ne 0 ]
  then echo "Please run as root"
  exit
fi


# get device that virtual disk landed on
while
 DEVICE=`parted -l -m 2>&1 | fgrep "$CONTROLLER" | cut -f 1 -d ':'`
 [[ -z $DEVICE ]]
do
 echo "Waiting for device (`date`)"
 sleep 1
done
echo "Got device $DEVICE"



umount $TESTFS || [ $? -eq 1 ]
sync
parted -s -a optimal $DEVICE mklabel gpt -- mkpart primary ext4 1 -1
parted -s $DEVICE print
mkfs.ext4 -F ${DEVICE}1
mount ${DEVICE}1 $TESTFS

rpath=`pwd`
pushd $TESTFS
sync

# random read/write/readwrite
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randread --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread --time_based --runtime=60s 2>&1 | tee $rpath/fio_randread.txt
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randwrite --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite --time_based --runtime=60s 2>&1 | tee $rpath/fio_randwrite.txt
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randrw --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 --time_based --runtime=60s 2>&1 | tee $rpath/fio_randrw.txt

# sequential read/write
fio --direct=1 --gtod_reduce=1 --name=read --filename=test --bs=64k --size=4G --readwrite=read --runtime=60s --time_based 2>&1 | tee $rpath/fio_seqread.txt
fio --direct=1 --gtod_reduce=1 --name=write --filename=test --bs=64k --size=4G --readwrite=write --runtime=60s --time_based 2>&1 | tee $rpath/fio_seqwrite.txt

# iozone
iozone -a -I -Mce -n 1M -g 4G -Rb $rpath/iozone_all.xls 2>&1 | tee $rpath/iozone_all.txt

iozone -I -c -e -r 256k -s 1G -i0 -i1 -i2 -i8 -t 1 -Rb $rpath/iozone_thr_t1.xls 2>&1 | tee $rpath/iozone_thr_t1.txt
iozone -I -c -e -r 256k -s 1G -i0 -i1 -i2 -i8 -t 4 -Rb $rpath/iozone_thr_t4.xls  2>&1 | tee $rpath/iozone_thr_t4.txt
iozone -I -c -e -r 256k -s 1G -i0 -i1 -i2 -i8 -t 16 -Rb $rpath/iozone_thr_t16.xls 2>&1 | tee $rpath/iozone_thr_t16.txt

popd
ls -ltr $rpath
This script partitions the array, creates an ext4 filesystem on it, mounts it at the /test mount point (which you must create before running any of this), runs a bunch of benchmark tests, and dumps the results file in the current directory (which is a unique directory created by do.sh for this configuration).
 
Last edited:

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
So I think the sum total of customization that would be needed is:
  • In bench.sh, edit CONTROLLER to match what you see in 'sudo parted -lm'.
  • In do.sh, edit the four *_LIST variables in do.sh to iterate over the array configurations of interest.
  • In do.sh, make sure the ARRAY_LIST commands use your controller number, disk numbers, etc.
  • Make the mount point directory (sudo mkdir /test).
  • Make a results directory on a filesystem on a non-array drive (mkdir /results).
  • Start 'er up (cd /results; sudo ./do.sh | tee do.out)
If you just want to get your configurations right, put an 'exit' after the "Configurations to test:" loop prints them out in do.sh.

Back up your data before doing this. It works for me, but you're doing lots of low-level things as root, so better safe than sorry.
 
Last edited:

chrispitude

Member
Dec 14, 2017
38
2
8
50
Allentown, PA
I got some runs done:

Code:
$ ls -1d raid*
raid0_x6_stripe256_iopolicycached_pdcacheon
raid0_x6_stripe64_iopolicycached_pdcacheon
raid10_x6_stripe256_iopolicycached_pdcacheon
raid10_x6_stripe64_iopolicycached_pdcacheon
raid50_x6_stripe256_iopolicycached_pdcacheon
raid50_x6_stripe64_iopolicycached_pdcacheon
raid5_x5_stripe256_iopolicycached_pdcacheon
raid5_x5_stripe64_iopolicycached_pdcacheon
raid5_x6_stripe256_iopolicycached_pdcacheon
raid5_x6_stripe64_iopolicycached_pdcacheon
raid6_x6_stripe256_iopolicycached_pdcacheon
raid6_x6_stripe64_iopolicycached_pdcacheon
Each directory has the generated benchmark files for that configuration:

Code:
$ ls raid0_x6_stripe256_iopolicycached_pdcacheon
all_tests.log     fio_randrw.txt     fio_seqread.txt   iozone_all.txt  iozone_thr_t16.txt  iozone_thr_t1.txt  iozone_thr_t4.txt
fio_randread.txt  fio_randwrite.txt  fio_seqwrite.txt  iozone_all.xls  iozone_thr_t16.xls  iozone_thr_t1.xls  iozone_thr_t4.xls
and now I can get creative with linux commands to mine data from the benchmarks:

Code:
$ fgrep READ *stripe256*/fio_seqread.txt
raid0_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=741MiB/s (777MB/s), 741MiB/s-741MiB/s (777MB/s-777MB/s), io=43.4GiB (46.6GB), run=60004-60004msec
raid10_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=387MiB/s (406MB/s), 387MiB/s-387MiB/s (406MB/s-406MB/s), io=22.7GiB (24.4GB), run=60009-60009msec
raid50_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=479MiB/s (503MB/s), 479MiB/s-479MiB/s (503MB/s-503MB/s), io=28.1GiB (30.2GB), run=60005-60005msec
raid5_x5_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=496MiB/s (520MB/s), 496MiB/s-496MiB/s (520MB/s-520MB/s), io=29.0GiB (31.2GB), run=60008-60008msec
raid5_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=601MiB/s (631MB/s), 601MiB/s-601MiB/s (631MB/s-631MB/s), io=35.2GiB (37.8GB), run=60002-60002msec
raid6_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=478MiB/s (502MB/s), 478MiB/s-478MiB/s (502MB/s-502MB/s), io=28.0GiB (30.1GB), run=60002-60002msec
$ fgrep WRITE *stripe256*/fio_seqwrite.txt
raid0_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=795MiB/s (834MB/s), 795MiB/s-795MiB/s (834MB/s-834MB/s), io=46.6GiB (50.0GB), run=60002-60002msec
raid10_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=406MiB/s (425MB/s), 406MiB/s-406MiB/s (425MB/s-425MB/s), io=23.8GiB (25.5GB), run=60009-60009msec
raid50_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=537MiB/s (563MB/s), 537MiB/s-537MiB/s (563MB/s-563MB/s), io=31.5GiB (33.8GB), run=60003-60003msec
raid5_x5_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=518MiB/s (543MB/s), 518MiB/s-518MiB/s (543MB/s-543MB/s), io=30.4GiB (32.6GB), run=60002-60002msec
raid5_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=623MiB/s (653MB/s), 623MiB/s-623MiB/s (653MB/s-653MB/s), io=36.5GiB (39.2GB), run=60018-60018msec
raid6_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=537MiB/s (563MB/s), 537MiB/s-537MiB/s (563MB/s-563MB/s), io=31.5GiB (33.8GB), run=60004-60004msec