Napp-It not scaling well ...

Discussion in 'Solaris, Nexenta, OpenIndiana, and napp-it' started by Rand__, Oct 20, 2017.

  1. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    So I got myself a bunch of SSDs recently (total 12x S3700/400 GB) and wanted to evaluate the actual scaling factor of ZFS mirrors. So I set up Napp-It (OI2017.04, not updated) and started blasting away on the disk with bonnie.


    Code:
    NAME   SIZE  Mirrors  Date(y.m.d)   File   Seq-Wr-Chr   %CPU   Seq-Write   %CPU   Seq-Rewr   %CPU   Seq-Rd-Chr   %CPU   Seq-Read   %CPU   Rnd Seeks   %CPU   Files   Seq-Create   Rnd-Create
    testpool  372G   1  2017.10.18      32G    235 MB/s      72      251 MB/s  26     175 MB/s      20     280 MB/s  99      606 MB/s   26  13079,2/s   19 16  +++++/s   +++++/s
    testpool  744G   2  2017.10.18      32G    358 MB/s      98      480 MB/s  46     337 MB/s      42     271 MB/s  99      1046 MB/s  47  13830.2/s  21 16  +++++/s   +++++/s
    testpool  1.09T  3  2017.10.20      32G    356 MB/s      97      696 MB/s  69     492 MB/s      60     266 MB/s  99      1494 MB/s  65  13691.2/s  24 16  +++++/s   +++++/s
    testpool  1.45T  4  2017.10.20      32G    362 MB/s      97      999 MB/s  94     705 MB/s      88     279 MB/s  99      1872 MB/s  84  15700.1/s  26 16  +++++/s   +++++/s
    testpool  1.82T  5  2017.10.20      32G    361 MB/s      96      949 MB/s  92     701 MB/s      92     270 MB/s  99      2063 MB/s  93  +++++/s   +++  16  +++++/s   +++++/s 
    testpool  2.18T  6  2017.10.20      32G    347 MB/s      95      917 MB/s  89     716 MB/s      90     272 MB/s  99      2067 MB/s  97  15800.0/s  33 16  +++++/s   +++++/s
    

    The Box is an Intel Server with an 2630Lv4 [M0](10C,1.7 G base), 64 GB Ram total, 12xS3700 400GB on a M1215 with 12G expander (Res30).
    The actual tests are on a VM with passs through of the M1215, 4 Cores, 16GB

    The results are not really what I was expecting - scale up kind of stopped at 4 mirrors...
    Now there are a lot of potential reasons - cpu maxed, interface limitation or probably a lot more. Just wonder which one it actually is...
     
    #1
    Last edited: Oct 22, 2017
  2. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,412
    Likes Received:
    301
    Slightly better than the performance I got. I had shelved the project for a few weeks as I had to divert some of the money for SSDS towards something else.

    Definitely interested in seeing where you go from here.

    Scalable Xeon SAS3 Storage Server
     
    #2
  3. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,910
    Likes Received:
    1,518
    • You're using an expander backplane #1 problem here
    • You can't run 8x SSD at max performance on 1x (SAS2) HBA let alone 12 + only running 1 channel not 2 for 8 you're cutting perf 1/2 already.
    • Your data provided shows CPU maxing out, it takes CPU to do high IOPs, I don't know if a 2630L V4 can do that

      I'm not sure of your HBA but running 12 drives I would run 2x SAS3 or if you only have SAS2 2x SAS2 HBA obviously is better than 1 but you want to ditch the expander backplane to use all 4 lanes.
     
    #3
    gigatexal likes this.
  4. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    Thanks:)
    The interesting aspect is that Read speed went up to 2TB but write speed was limited to 1TB.
    But yes you point out the same issues that I assumed.
    But 12GB/s are 4800 MB/s raw, so getting half of that was irritating. But CPU is indeed a point - wonder whether bonnie is single threaded...
    Still makes me wonder why it's so hard on the CPU, would have thought that there were similar techs like TCP Checksum offload for storage too. But clearly I am no hardware person;)
     
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    So next run, added a second HBA (but still on same 2 uploads from expander) and exchanged the 2630L with an 2667v4.
    Better, but not really great.
    Will try directly attaching the drives next I guess.

    Code:
    NAME           SIZE     Mirrors     Date(y.m.d)      File      Seq-Wr-Chr      %CPU      Seq-Write      %CPU      Seq-Rewr      %CPU      Seq-Rd-Chr      %CPU      Seq-Read      %CPU      Rnd Seeks      %CPU      Files      Seq-Create      Rnd-Create 
    testpool      2.18T     6           2017.10.21      32G        472 MB/s         95     1333 MB/s        93     914 MB/s         86        411 MB/s         99     2081 MB/s        72     +++++/s      +++       16     +++++/s      +++++/s  
    
     
    #5
  6. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,910
    Likes Received:
    1,518
    Yeah, next up I would:

    - Direct attach and re-test
    - Keep HBAs on same CPU if possible
    - Keep drives mirrored to other drives on same HBA
    - Make sure HBA have same firmware/up to date known working
     
    #6
  7. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    874
    Likes Received:
    94
    Have you tried to see what is maxing out the cpu ?
    Could be bonnie but also zfs.
    I'd turn off zfs compression on the dataset for max performance.

    Is everything aligned ?
    I think this would be only setting ashift=12 in this case, but could be more.

    That would be cool if there was something like zfs_checksum_offload and lz4_compression_offload to the HBA :D
     
    #7
  8. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    Looks like its indeed bonnie
    Code:
    PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
    5753 root        1   0    0 7488K 5948K cpu/1    2:08 24.97% bonnie++
    
    And it looks like its switching CPUs (seeing it on /0,/1,/3) after all, but never going over 25% (1 core).

    Else I have not deviated from the default, so need to check if compression is on per default on Napp-It.

    And I was not even looking for zfs based offlaod, but would have thought that the HBA would handle most of the heavy lifting - if there is any. Would have thought mirroring to be relatively light on the HW.
     
    #8
  9. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    874
    Likes Received:
    94
    Why not benchmark with fio + numjobs ?
    Should scale better across cores.

    Think mirroring is done by zfs and the HBA doesn't know there is a mirror.

    Not sure about compression, could be throughput is higher when its on - at least for data that can be compressed.
     
    #9
    gigatexal likes this.
  10. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    Not looking for largest possible numbers, just wanted to see the impact of different drive configurations and bonnie was readily available (i.e. 1 click no work).
    Just wondered why it didn't scale any more ;)
     
    #10
  11. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    Next set of results - 2 hbas, direct attached. have not checked HBA firmware yet tbh, but both are m1215's.

    Code:
    NAME      SIZE     Mirrors     Date(y.m.d)      File      Seq-Wr-Chr      %CPU      Seq-Write      %CPU      Seq-Rewr      %CPU      Seq-Rd-Chr      %CPU      Seq-Read      %CPU      Rnd Seeks      %CPU      Files      Seq-Create      Rnd-Create
    testpool      1.45T     4     2017.10.22      32G          491 MB/s        97        1117 MB/s      78      748 MB/s        69      403 MB/s        99      2111 MB/s        71      +++++/s      +++        16      +++++/s      +++++/s
    testpool      1.82T     5     2017.10.22      32G          486 MB/s        97        1321 MB/s      93      932 MB/s        90      400 MB/s        99      2523 MB/s        88      +++++/s      +++        16      +++++/s      +++++/s 
    testpool      2.18T     6     2017.10.22      32G          476 MB/s        95        1269 MB/s      91      916 MB/s        90      405 MB/s        99      2682 MB/s        93      +++++/s      +++        16      +++++/s      +++++/s 
    
     
    #11
  12. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,910
    Likes Received:
    1,518
    IF you google ZFS compression benchmarks there's someone who did a bunch of tests and compression usually increases performance, worthwhile to read it and compare with his #s as in some cases it may hurt but in most it helped, more than I ever thought possible.
     
    #12
  13. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,910
    Likes Received:
    1,518
    So writes have nearly doubled already it looks like, right?
     
    #13
  14. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    874
    Likes Received:
    94
    no doubt compression increases performance with real world data.
    but to get just raw, comparable numbers for different disks i would disable it.
     
    #14
  15. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    Well 917 to 1321 is not really 'nearly doubled' ;) Improved by 50%, yes.
    Makes me wonder why 10 drives are faster then 12 on the other hand...

    I also see that CPU load is up again - not sure how much faster than a 2667 it can get in a server though.
     
    #15
  16. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,910
    Likes Received:
    1,518
    @Rand__ I was looking at the initial #s you posted... since the 1st test to the last it looks to have doubled, if not more... ???
     
    #16
  17. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,888
    Likes Received:
    657
    Depends on what you were looking at ;)
    Rows in the datasheet are increasing number of mirror pairs (1->6).
    Later tests had 1 test only (6 pairs) at post #5 and 3 tests (4-6 mirrors) at post #11.
     
    #17
Similar Threads: Napp-It scaling
Forum Title Date
Solaris, Nexenta, OpenIndiana, and napp-it Napp-It not scaling well ... - revisited ;) Nov 20, 2019
Solaris, Nexenta, OpenIndiana, and napp-it napp-it Windows Sharing Permissions Feb 7, 2020
Solaris, Nexenta, OpenIndiana, and napp-it Is there plans to integrate a VM manager into Napp-IT? Jan 28, 2020
Solaris, Nexenta, OpenIndiana, and napp-it napp-it 14b AIO upgrade path? Jan 12, 2020
Solaris, Nexenta, OpenIndiana, and napp-it ESXi 6.7u3, Napp-It AIO, & Tuning Jan 8, 2020

Share This Page