Napp-It not scaling well ...

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
So I got myself a bunch of SSDs recently (total 12x S3700/400 GB) and wanted to evaluate the actual scaling factor of ZFS mirrors. So I set up Napp-It (OI2017.04, not updated) and started blasting away on the disk with bonnie.


Code:
NAME   SIZE  Mirrors  Date(y.m.d)   File   Seq-Wr-Chr   %CPU   Seq-Write   %CPU   Seq-Rewr   %CPU   Seq-Rd-Chr   %CPU   Seq-Read   %CPU   Rnd Seeks   %CPU   Files   Seq-Create   Rnd-Create
testpool  372G   1  2017.10.18      32G    235 MB/s      72      251 MB/s  26     175 MB/s      20     280 MB/s  99      606 MB/s   26  13079,2/s   19 16  +++++/s   +++++/s
testpool  744G   2  2017.10.18      32G    358 MB/s      98      480 MB/s  46     337 MB/s      42     271 MB/s  99      1046 MB/s  47  13830.2/s  21 16  +++++/s   +++++/s
testpool  1.09T  3  2017.10.20      32G    356 MB/s      97      696 MB/s  69     492 MB/s      60     266 MB/s  99      1494 MB/s  65  13691.2/s  24 16  +++++/s   +++++/s
testpool  1.45T  4  2017.10.20      32G    362 MB/s      97      999 MB/s  94     705 MB/s      88     279 MB/s  99      1872 MB/s  84  15700.1/s  26 16  +++++/s   +++++/s
testpool  1.82T  5  2017.10.20      32G    361 MB/s      96      949 MB/s  92     701 MB/s      92     270 MB/s  99      2063 MB/s  93  +++++/s   +++  16  +++++/s   +++++/s 
testpool  2.18T  6  2017.10.20      32G    347 MB/s      95      917 MB/s  89     716 MB/s      90     272 MB/s  99      2067 MB/s  97  15800.0/s  33 16  +++++/s   +++++/s

The Box is an Intel Server with an 2630Lv4 [M0](10C,1.7 G base), 64 GB Ram total, 12xS3700 400GB on a M1215 with 12G expander (Res30).
The actual tests are on a VM with passs through of the M1215, 4 Cores, 16GB

The results are not really what I was expecting - scale up kind of stopped at 4 mirrors...
Now there are a lot of potential reasons - cpu maxed, interface limitation or probably a lot more. Just wonder which one it actually is...
 
Last edited:

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
Slightly better than the performance I got. I had shelved the project for a few weeks as I had to divert some of the money for SSDS towards something else.

Definitely interested in seeing where you go from here.

Scalable Xeon SAS3 Storage Server
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
  • You're using an expander backplane #1 problem here
  • You can't run 8x SSD at max performance on 1x (SAS2) HBA let alone 12 + only running 1 channel not 2 for 8 you're cutting perf 1/2 already.
  • Your data provided shows CPU maxing out, it takes CPU to do high IOPs, I don't know if a 2630L V4 can do that

    I'm not sure of your HBA but running 12 drives I would run 2x SAS3 or if you only have SAS2 2x SAS2 HBA obviously is better than 1 but you want to ditch the expander backplane to use all 4 lanes.
 
  • Like
Reactions: gigatexal

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Thanks:)
The interesting aspect is that Read speed went up to 2TB but write speed was limited to 1TB.
But yes you point out the same issues that I assumed.
But 12GB/s are 4800 MB/s raw, so getting half of that was irritating. But CPU is indeed a point - wonder whether bonnie is single threaded...
Still makes me wonder why it's so hard on the CPU, would have thought that there were similar techs like TCP Checksum offload for storage too. But clearly I am no hardware person;)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
So next run, added a second HBA (but still on same 2 uploads from expander) and exchanged the 2630L with an 2667v4.
Better, but not really great.
Will try directly attaching the drives next I guess.

Code:
NAME           SIZE     Mirrors     Date(y.m.d)      File      Seq-Wr-Chr      %CPU      Seq-Write      %CPU      Seq-Rewr      %CPU      Seq-Rd-Chr      %CPU      Seq-Read      %CPU      Rnd Seeks      %CPU      Files      Seq-Create      Rnd-Create 
testpool      2.18T     6           2017.10.21      32G        472 MB/s         95     1333 MB/s        93     914 MB/s         86        411 MB/s         99     2081 MB/s        72     +++++/s      +++       16     +++++/s      +++++/s
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Yeah, next up I would:

- Direct attach and re-test
- Keep HBAs on same CPU if possible
- Keep drives mirrored to other drives on same HBA
- Make sure HBA have same firmware/up to date known working
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
Have you tried to see what is maxing out the cpu ?
Could be bonnie but also zfs.
I'd turn off zfs compression on the dataset for max performance.

Is everything aligned ?
I think this would be only setting ashift=12 in this case, but could be more.

would have thought that there were similar techs like TCP Checksum offload for storage too.
That would be cool if there was something like zfs_checksum_offload and lz4_compression_offload to the HBA :D
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Looks like its indeed bonnie
Code:
PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
5753 root        1   0    0 7488K 5948K cpu/1    2:08 24.97% bonnie++
And it looks like its switching CPUs (seeing it on /0,/1,/3) after all, but never going over 25% (1 core).

Else I have not deviated from the default, so need to check if compression is on per default on Napp-It.

And I was not even looking for zfs based offlaod, but would have thought that the HBA would handle most of the heavy lifting - if there is any. Would have thought mirroring to be relatively light on the HW.
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
Why not benchmark with fio + numjobs ?
Should scale better across cores.

Think mirroring is done by zfs and the HBA doesn't know there is a mirror.

Not sure about compression, could be throughput is higher when its on - at least for data that can be compressed.
 
  • Like
Reactions: gigatexal

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Not looking for largest possible numbers, just wanted to see the impact of different drive configurations and bonnie was readily available (i.e. 1 click no work).
Just wondered why it didn't scale any more ;)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Next set of results - 2 hbas, direct attached. have not checked HBA firmware yet tbh, but both are m1215's.

Code:
NAME      SIZE     Mirrors     Date(y.m.d)      File      Seq-Wr-Chr      %CPU      Seq-Write      %CPU      Seq-Rewr      %CPU      Seq-Rd-Chr      %CPU      Seq-Read      %CPU      Rnd Seeks      %CPU      Files      Seq-Create      Rnd-Create
testpool      1.45T     4     2017.10.22      32G          491 MB/s        97        1117 MB/s      78      748 MB/s        69      403 MB/s        99      2111 MB/s        71      +++++/s      +++        16      +++++/s      +++++/s
testpool      1.82T     5     2017.10.22      32G          486 MB/s        97        1321 MB/s      93      932 MB/s        90      400 MB/s        99      2523 MB/s        88      +++++/s      +++        16      +++++/s      +++++/s 
testpool      2.18T     6     2017.10.22      32G          476 MB/s        95        1269 MB/s      91      916 MB/s        90      405 MB/s        99      2682 MB/s        93      +++++/s      +++        16      +++++/s      +++++/s
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
IF you google ZFS compression benchmarks there's someone who did a bunch of tests and compression usually increases performance, worthwhile to read it and compare with his #s as in some cases it may hurt but in most it helped, more than I ever thought possible.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Next set of results - 2 hbas, direct attached. have not checked HBA firmware yet tbh, but both are m1215's.

Code:
NAME      SIZE     Mirrors     Date(y.m.d)      File      Seq-Wr-Chr      %CPU      Seq-Write      %CPU      Seq-Rewr      %CPU      Seq-Rd-Chr      %CPU      Seq-Read      %CPU      Rnd Seeks      %CPU      Files      Seq-Create      Rnd-Create
testpool      1.45T     4     2017.10.22      32G          491 MB/s        97        1117 MB/s      78      748 MB/s        69      403 MB/s        99      2111 MB/s        71      +++++/s      +++        16      +++++/s      +++++/s
testpool      1.82T     5     2017.10.22      32G          486 MB/s        97        1321 MB/s      93      932 MB/s        90      400 MB/s        99      2523 MB/s        88      +++++/s      +++        16      +++++/s      +++++/s
testpool      2.18T     6     2017.10.22      32G          476 MB/s        95        1269 MB/s      91      916 MB/s        90      405 MB/s        99      2682 MB/s        93      +++++/s      +++        16      +++++/s      +++++/s
So writes have nearly doubled already it looks like, right?
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
no doubt compression increases performance with real world data.
but to get just raw, comparable numbers for different disks i would disable it.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
So writes have nearly doubled already it looks like, right?
Well 917 to 1321 is not really 'nearly doubled' ;) Improved by 50%, yes.
Makes me wonder why 10 drives are faster then 12 on the other hand...

I also see that CPU load is up again - not sure how much faster than a 2667 it can get in a server though.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
@Rand__ I was looking at the initial #s you posted... since the 1st test to the last it looks to have doubled, if not more... ???
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Depends on what you were looking at ;)
Rows in the datasheet are increasing number of mirror pairs (1->6).
Later tests had 1 test only (6 pairs) at post #5 and 3 tests (4-6 mirrors) at post #11.