Help me benchmark some MegaRAID controllers...

Discussion in 'RAID Controllers and Host Bus Adapters' started by chrispitude, Jan 12, 2018.

  1. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Hi folks,

    Here's what I've currently got:
    • NEC 9267-8i controller with BBU
    • Six 3TB WD Red HDDs
    • One 500GB Samsung 850 EVO SSD
    • Ubuntu 17.04 machine, 16GB RAM, i7-4790k CPU (just consumer stuff)
    I thought the 9267-8i would do CacheCade, but it turns out I was wrong.

    This is for a home Plex/virtualbox/Samba box. I'd like to get a CacheCade setup because... well, I like playing with computer stuff. So practicality takes a bit of a back seat if the price is right.

    What's an inexpensive way to get a CacheCade/RAID5/6 setup? It looks like I can get an LSI 9260-8i+LSI00292 hardware key for under $80. But...
    • Would spending an extra $60ish on an 9265-8i+LSI00290 hardware key yield noticeable improvement with these drives?
    • Are there other CacheCade-capable controllers (perhaps Dell/IBM/NEC-branded) that are good deals at this performance level?
    Right now this box is free, so I'm happy to do some benchmarking. Maybe I can buy another controller or two and test them at various RAID0/5/6 with/without CacheCade configurations. If you give the guidance, I'm happy to do the repeated hardware and OS installs, etc., and share my findings.
     
    #1
    Last edited: Jan 14, 2018
  2. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Forgot to mention...

    The reason for this is, I've asked several questions recently and this community has been incredibly helpful. I'd like to volunteer my time and resources to say thanks and give back.
     
    #2
  3. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Snagged a 9260-8i+LSI00292 key for $47 shipped!
     
    #3
  4. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,252
    Likes Received:
    283
    I think you will get some improvements on small, random read iops. (For write caching you will need redundancy or parity).

    I don't have experience with cachecade, but with maxcache (adaptecs ssd caching). In maxcache the raid controller constantly reads and writes to the ssds (=> sustained workload). Your 850 evo is not made for such a use case and the performance of your whole arrray will degrade once the cache in the ssd is filled up.
     
    #3
  5. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Thanks i386, I appreciate your cautions. This is largely just an excuse to play with CacheCade using a large dataset (uhh, my own stuff). I might or might not keep that configuration. The contents will always be backed up. In the longer run, I might run the SSD directly as a system disk (as you previously recommended) or move to a NAS-based solution instead.

    What's the STH-blessed way of testing disk performance in linux? I've seen STH posts discussing development of such a benchmark, but nothing about its completion or usage.
     
    #4
  6. aero

    aero Active Member

    Joined:
    Apr 27, 2016
    Messages:
    284
    Likes Received:
    46
    I prefer iozone for testing in Linux. That's just me though.
     
    #5
  7. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Thanks! I can try it. What's your recommended command line?
     
    #6
  8. aero

    aero Active Member

    Joined:
    Apr 27, 2016
    Messages:
    284
    Likes Received:
    46
    #7
  9. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Are there "standard" fio job files for measuring things like sequential R/W, random R/W, and IOPS?

    Edit: currently playing with the following commands:

    Code:
    # random read/write/readwrite
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randread --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randread -time_based -runtime=600
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randwrite --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randwrite -time_based -runtime=600
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randrw --filename=test --bs=4k --iodepth=64 --size=16G --readwrite=randrw --rwmixread=75 -time_based -runtime=600
    
    # sequential read/write
    fio --direct=1 --gtod_reduce=1 --name=read --filename=test --bs=64k --size=16G --readwrite=read
    fio --direct=1 --gtod_reduce=1 --name=write --filename=test --bs=64k --size=16G --readwrite=write
    
     
    #8
    Last edited: Jan 23, 2018
  10. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Okay, I think I'm going to go with the following fio/iozone commands:


    Code:
    #!/bin/bash
    rpath=/results/6x_3TB_RAID0_stripe64k_readahead
    
    mkdir $rpath
    pushd /test
    
    # random read/write/readwrite
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randread --filename=test --bs=4k --iodepth=64 --size=8G --readwrite=randread -time_based -runtime=60 | tee $rpath/fio_randread.txt
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randwrite --filename=test --bs=4k --iodepth=64 --size=8G --readwrite=randwrite -time_based -runtime=60 | tee $rpath/fio_randwrite.txt
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randrw --filename=test --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 -time_based -runtime=60 | tee $rpath/fio_randrw.txt
    
    # sequential read/write
    fio --direct=1 --gtod_reduce=1 --name=read --filename=test --bs=64k --size=16G --readwrite=read -runtime=120 | tee $rpath/fio_seqread.txt
    fio --direct=1 --gtod_reduce=1 --name=write --filename=test --bs=64k --size=16G --readwrite=write -runtime=120 | tee $rpath/seqwrite.txt
    
    # iozone
    iozone -a -I -Mce -n 1M -g 8G -Rb $rpath/iozone_all_try1.xls | tee $rpath/iozone_all_try1.txt
    iozone -a -I -Mce -n 1M -g 8G -Rb $rpath/iozone_all_try2.xls | tee $rpath/iozone_all_try2.txt
    iozone -a -I -Mce -n 1M -g 8G -Rb $rpath/iozone_all_try3.xls | tee $rpath/iozone_all_try3.txt
    
    
    iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 1 -Rb $rpath/iozone_thr_t1.xls | tee $rpath/iozone_thr_t1.txt
    iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 2 -Rb $rpath/iozone_thr_t2.xls  | tee $rpath/iozone_thr_t2.txt
    iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 4 -Rb $rpath/iozone_thr_t4.xls  | tee $rpath/iozone_thr_t4.txt
    iozone -I -c -e -r 256k -s 4G -i0 -i1 -i2 -i8 -t 8 -Rb $rpath/iozone_thr_t8.xls  | tee $rpath/iozone_thr_t8.txt
    
    popd
    ls -ltr $rpath
     
    #9
  11. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,252
    Likes Received:
    283
    I think these numbers are a little bit unrealistic for a home plex/fileserver serving most of the time large files. Try it with qd <= 8 and 90(+)% read/10%write.
     
    #10
  12. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Thanks! Mostly I am looking to get benchmark data useful for you guys, not as much for me.

    I've almost got it scripted up to create a bunch of RAID array configurations using storcli (successor to megacli), create a fresh filesystem each time, and run a bunch of benchmarks on it. The issue I'm fighting now is that after doing the array initialization via storcli, sometimes the array ends up on /dev/sda and sometimes it ends up on /dev/sdc, and I can't figure out a way to make it always land on /dev/sda.
     
    #11
  13. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Okay here's my main script, do.sh:

    Code:
    #!/bin/bash
    if [ "$EUID" -ne 0 ]
      then echo "Please run as root"
      exit
    fi
    
    
    STRIPE_LIST='256'
    IOPOLICY_LIST='direct cached'
    PDCACHE_LIST='on off'
    
    unset ARRAY_LIST
    declare -A ARRAY_LIST
    ARRAY_LIST[raid0_x6]='storcli64 /c0 add vd type=raid0 name=test drives=252:0-5 ra'
    ARRAY_LIST[raid5_x6]='storcli64 /c0 add vd type=raid5 name=test drives=252:0-5 ra'
    ARRAY_LIST[raid6_x6]='storcli64 /c0 add vd type=raid6 name=test drives=252:0-5 ra'
    ARRAY_LIST[raid5_x5]='storcli64 /c0 add vd type=raid5 name=test drives=252:0-4 ra'
    ARRAY_LIST[raid10_x6]='storcli64 /c0 add vd type=raid10 name=test drives=252:0-5 pdperarray=2 ra'
    ARRAY_LIST[raid50_x6]='storcli64 /c0 add vd type=raid50 name=test drives=252:0-5 pdperarray=3 ra'
    
    ======
    
    # make all array combinations
    unset configs_to_test
    declare -A configs_to_test
    for stripe in $STRIPE_LIST
    do
     for iopolicy in $IOPOLICY_LIST
     do
      for pdcache in $PDCACHE_LIST
      do
       for array_key in "${!ARRAY_LIST[@]}"
       do
        array_config=${ARRAY_LIST[$array_key]}" strip=$stripe $iopolicy pdcache=$pdcache"
        key=$array_key"_stripe${stripe}_iopolicy${iopolicy}_pdcache${pdcache}"
        configs_to_test[$key]=$array_config
       done
      done
     done
    done
    
    
    
    # report the configurations queued for test
    echo "Configurations to test:"
    for key in "${!configs_to_test[@]}"
    do
     echo "  $key: ${configs_to_test[$key]}"
    done
    echo ""
    
    
    # test all the configurations
    for key in "${!configs_to_test[@]}"
    do
     echo "NOW TESTING $key: ${configs_to_test[$key]}"
    
     storcli64 /c0/vall del force
     storcli64 /c0 flushcache
    
     eval ${configs_to_test[$key]}
     storcli64 /c0/v0 set wrcache=awb
     storcli64 /c0/v0 start init force  ;# force even if there's a mounted filesystem from last iteration
    
     while [[ $(storcli64 /c0/v0 show init) != *Not* ]]
     do
      echo "Waiting for array to initialize (`date`)"
      sleep 1
     done
     storcli64 /c0/v0 show init
     storcli64 /c0 flushcache
     parted -l  ;# verify the array landed on some device
    
     rm -rf ./$key
     mkdir -p ./$key
     pushd $key
    
     time ../bench.sh 2>&1 | tee all_tests.log
    
     popd
     echo "TESTING COMPLETE FOR $key: ${configs_to_test[$key]}"
    done
    You configure the four *_LIST variables at the top, then run the script using sudo. It assumes storcli64 is in your search path, and it uses it to create each virtual disk of interest. Then for each disk, it makes a results directory within the current directory, goes into it, and runs the bench.sh script.
     
    #12
    Last edited: Jan 26, 2018
  14. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Here's the bench.sh script:
    Code:
    #!/bin/bash
    
    # this is the controller, as reported by 'parted -l'
    # (used to find out on which device the array landed)
    CONTROLLER='LSI MR9267-8i'
    
    # this is the test filesystem mountpoint
    TESTFS='/test'
    
    
    ################
    
    
    if [ "$EUID" -ne 0 ]
      then echo "Please run as root"
      exit
    fi
    
    
    # get device that virtual disk landed on
    while
     DEVICE=`parted -l -m 2>&1 | fgrep "$CONTROLLER" | cut -f 1 -d ':'`
     [[ -z $DEVICE ]]
    do
     echo "Waiting for device (`date`)"
     sleep 1
    done
    echo "Got device $DEVICE"
    
    
    
    umount $TESTFS || [ $? -eq 1 ]
    sync
    parted -s -a optimal $DEVICE mklabel gpt -- mkpart primary ext4 1 -1
    parted -s $DEVICE print
    mkfs.ext4 -F ${DEVICE}1
    mount ${DEVICE}1 $TESTFS
    
    rpath=`pwd`
    pushd $TESTFS
    sync
    
    # random read/write/readwrite
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randread --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread --time_based --runtime=60s 2>&1 | tee $rpath/fio_randread.txt
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randwrite --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite --time_based --runtime=60s 2>&1 | tee $rpath/fio_randwrite.txt
    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=randrw --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 --time_based --runtime=60s 2>&1 | tee $rpath/fio_randrw.txt
    
    # sequential read/write
    fio --direct=1 --gtod_reduce=1 --name=read --filename=test --bs=64k --size=4G --readwrite=read --runtime=60s --time_based 2>&1 | tee $rpath/fio_seqread.txt
    fio --direct=1 --gtod_reduce=1 --name=write --filename=test --bs=64k --size=4G --readwrite=write --runtime=60s --time_based 2>&1 | tee $rpath/fio_seqwrite.txt
    
    # iozone
    iozone -a -I -Mce -n 1M -g 4G -Rb $rpath/iozone_all.xls 2>&1 | tee $rpath/iozone_all.txt
    
    iozone -I -c -e -r 256k -s 1G -i0 -i1 -i2 -i8 -t 1 -Rb $rpath/iozone_thr_t1.xls 2>&1 | tee $rpath/iozone_thr_t1.txt
    iozone -I -c -e -r 256k -s 1G -i0 -i1 -i2 -i8 -t 4 -Rb $rpath/iozone_thr_t4.xls  2>&1 | tee $rpath/iozone_thr_t4.txt
    iozone -I -c -e -r 256k -s 1G -i0 -i1 -i2 -i8 -t 16 -Rb $rpath/iozone_thr_t16.xls 2>&1 | tee $rpath/iozone_thr_t16.txt
    
    popd
    ls -ltr $rpath
    This script partitions the array, creates an ext4 filesystem on it, mounts it at the /test mount point (which you must create before running any of this), runs a bunch of benchmark tests, and dumps the results file in the current directory (which is a unique directory created by do.sh for this configuration).
     
    #13
    Last edited: Jan 25, 2018
  15. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    So I think the sum total of customization that would be needed is:
    • In bench.sh, edit CONTROLLER to match what you see in 'sudo parted -lm'.
    • In do.sh, edit the four *_LIST variables in do.sh to iterate over the array configurations of interest.
    • In do.sh, make sure the ARRAY_LIST commands use your controller number, disk numbers, etc.
    • Make the mount point directory (sudo mkdir /test).
    • Make a results directory on a filesystem on a non-array drive (mkdir /results).
    • Start 'er up (cd /results; sudo ./do.sh | tee do.out)
    If you just want to get your configurations right, put an 'exit' after the "Configurations to test:" loop prints them out in do.sh.

    Back up your data before doing this. It works for me, but you're doing lots of low-level things as root, so better safe than sorry.
     
    #14
    Last edited: Jan 25, 2018
  16. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    I got some runs done:

    Code:
    $ ls -1d raid*
    raid0_x6_stripe256_iopolicycached_pdcacheon
    raid0_x6_stripe64_iopolicycached_pdcacheon
    raid10_x6_stripe256_iopolicycached_pdcacheon
    raid10_x6_stripe64_iopolicycached_pdcacheon
    raid50_x6_stripe256_iopolicycached_pdcacheon
    raid50_x6_stripe64_iopolicycached_pdcacheon
    raid5_x5_stripe256_iopolicycached_pdcacheon
    raid5_x5_stripe64_iopolicycached_pdcacheon
    raid5_x6_stripe256_iopolicycached_pdcacheon
    raid5_x6_stripe64_iopolicycached_pdcacheon
    raid6_x6_stripe256_iopolicycached_pdcacheon
    raid6_x6_stripe64_iopolicycached_pdcacheon
    Each directory has the generated benchmark files for that configuration:

    Code:
    $ ls raid0_x6_stripe256_iopolicycached_pdcacheon
    all_tests.log     fio_randrw.txt     fio_seqread.txt   iozone_all.txt  iozone_thr_t16.txt  iozone_thr_t1.txt  iozone_thr_t4.txt
    fio_randread.txt  fio_randwrite.txt  fio_seqwrite.txt  iozone_all.xls  iozone_thr_t16.xls  iozone_thr_t1.xls  iozone_thr_t4.xls
    and now I can get creative with linux commands to mine data from the benchmarks:

    Code:
    $ fgrep READ *stripe256*/fio_seqread.txt
    raid0_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=741MiB/s (777MB/s), 741MiB/s-741MiB/s (777MB/s-777MB/s), io=43.4GiB (46.6GB), run=60004-60004msec
    raid10_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=387MiB/s (406MB/s), 387MiB/s-387MiB/s (406MB/s-406MB/s), io=22.7GiB (24.4GB), run=60009-60009msec
    raid50_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=479MiB/s (503MB/s), 479MiB/s-479MiB/s (503MB/s-503MB/s), io=28.1GiB (30.2GB), run=60005-60005msec
    raid5_x5_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=496MiB/s (520MB/s), 496MiB/s-496MiB/s (520MB/s-520MB/s), io=29.0GiB (31.2GB), run=60008-60008msec
    raid5_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=601MiB/s (631MB/s), 601MiB/s-601MiB/s (631MB/s-631MB/s), io=35.2GiB (37.8GB), run=60002-60002msec
    raid6_x6_stripe256_iopolicycached_pdcacheon/fio_seqread.txt:   READ: bw=478MiB/s (502MB/s), 478MiB/s-478MiB/s (502MB/s-502MB/s), io=28.0GiB (30.1GB), run=60002-60002msec
    $ fgrep WRITE *stripe256*/fio_seqwrite.txt
    raid0_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=795MiB/s (834MB/s), 795MiB/s-795MiB/s (834MB/s-834MB/s), io=46.6GiB (50.0GB), run=60002-60002msec
    raid10_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=406MiB/s (425MB/s), 406MiB/s-406MiB/s (425MB/s-425MB/s), io=23.8GiB (25.5GB), run=60009-60009msec
    raid50_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=537MiB/s (563MB/s), 537MiB/s-537MiB/s (563MB/s-563MB/s), io=31.5GiB (33.8GB), run=60003-60003msec
    raid5_x5_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=518MiB/s (543MB/s), 518MiB/s-518MiB/s (543MB/s-543MB/s), io=30.4GiB (32.6GB), run=60002-60002msec
    raid5_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=623MiB/s (653MB/s), 623MiB/s-623MiB/s (653MB/s-653MB/s), io=36.5GiB (39.2GB), run=60018-60018msec
    raid6_x6_stripe256_iopolicycached_pdcacheon/fio_seqwrite.txt:  WRITE: bw=537MiB/s (563MB/s), 537MiB/s-537MiB/s (563MB/s-563MB/s), io=31.5GiB (33.8GB), run=60004-60004msec
    
     
    #15
  17. chrispitude

    chrispitude Member

    Joined:
    Dec 14, 2017
    Messages:
    38
    Likes Received:
    1
    Hmm, I could use this setup to test various ext4 stride setups to see if any of them work synergistically with the controller's RAID access patterns...
     
    #16
Similar Threads: Help benchmark
Forum Title Date
RAID Controllers and Host Bus Adapters Help - Slower Than Expected SAS3 HBA Benchmarks Feb 5, 2017
RAID Controllers and Host Bus Adapters Help with LSI9206 (Not seeing drives in R620) Sunday at 7:02 PM
RAID Controllers and Host Bus Adapters X8DT6-F and second HBA MegaRaid 9240-4i ( CrossFlashing Help needed) Jul 7, 2018
RAID Controllers and Host Bus Adapters Need help Picking a HBA and Cables... Feb 4, 2018
RAID Controllers and Host Bus Adapters Urgent Help Required Jan 24, 2018

Share This Page