Linux-Bench: Storage

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Patrick

Administrator
Staff member
Dec 21, 2010
12,517
5,832
113
OK - Time to start the official planning thread! STH did start over five years ago with NAS/ storage system reviews and benchmarks. It is time to get the storage side going again after a successful Linux-Bench CPU project.

It is time for an official Linux-Bench Storage benchmark thread.

Goals of the Linux-Bench Storage bench:
  • Linux based
  • "Free" to run (no proprietary software)
  • Easy to run, even from a LiveCD
  • Output to a standard log file
  • Drive pre-conditioning
  • Look at transfer but also transfer over time
  • Make results easy to compare
  • Minimal CPU impact
  • Test NAS, SAN and DAS
Potential benchmarks:
  • fio
  • iozone
  • iometer
Any others we should consider? In this thread please feel free to give feedback on benchmarks/ benchmark parameters, test procedures, goals of the project.

This is ultra exciting to kick off. The next step will be the Github project launch.

Github: STH-Dev/linux-bench-storage · GitHub
 
Last edited:
  • Like
Reactions: Chuntzu and Patriot

Patriot

Moderator
Apr 18, 2011
1,453
793
113
OK - Time to start the official planning thread! STH did start over five years ago with NAS/ storage system reviews and benchmarks. It is time to get the storage side going again after a successful Linux-Bench CPU project.

It is time for an official Linux-Bench Storage benchmark thread.

Goals of the Linux-Bench Storage bench:
  • Linux based
  • "Free" to run (no proprietary software)
  • Easy to run, even from a LiveCD
  • Output to a standard log file
  • Drive pre-conditioning
  • Look at transfer but also transfer over time
  • Make results easy to compare
  • Minimal CPU impact
Potential benchmarks:
  • fio
  • iozone
  • iometer
Any others we should consider? In this thread please feel free to give feedback on benchmarks/ benchmark parameters, test procedures, goals of the project.

This is ultra exciting to kick off. The next step will be the Github project launch.

Not used Iozone in years, I don't know how good it is for high qdepth.
Iometer is good in windows broken in linux.
Phoronix uses Bonnie++ not sure how good that is but something to look at.

FIO is the winner here I have quite a few sweep test I can lay down.
You do need to use the proper libraries otherwise you will be stuck in sync mode ala qdepth=1


Excited to see STH moving away from kiddie benchmarks like crystal disk and atto and the like.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,517
5,832
113
@Patriot. Great points. Also will make an edit. I received a google chat reminding me that I should include the facility to test NAS/ SAN and DAS.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,517
5,832
113
Got a great bit of feedback via PM. One way we could go with this is VMs and a test procedure.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
IOzone has really interesting graphs. I'd definitely like to see what we can do for the visualization of data. Bonnie++ is relatively simple and fast.

Wouldn't the NAS/SAN stuff be pretty dependent on the network as well? Somehow need to standardize that in order to get community results. Even multi-stream, etc.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,517
5,832
113
If anyone has ideas on this, please let me know. For the next few days I am going to have around 20 different drives around for testing.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
I'll try to play with some of the standard tools and start getting some scripts this weekend. I assume normalize test hardware to m1015 or something to remove cache effects and make repeatable?

First round would be single disk tests and whatnot as well?
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,517
5,832
113
I'll try to play with some of the standard tools and start getting some scripts this weekend. I assume normalize test hardware to m1015 or something to remove cache effects and make repeatable?

First round would be single disk tests and whatnot as well?
Likely, although I need to send some hardware out so there is a somewhat short window to test all of these drives in single/ multi-drive modes.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
So I'm going to start to work on this a bit. I have a Bonnie++ script working, etc. Testing out Iozone. Here are a few questions:

Not a lot of good tools in Linux and most are outdated or unmaintained. How important are the concepts of compressed data, uncompressable, initing the stream, etc?

For example, I was playing with Iozone (need to rerun the tests). Here's some results with surface graphs (download to view the tabs). I have a friend who can help invest time to build a cool parser tool, are the iozone tests still worthwhile? I've got a great wishlist of how I'd love to see it...
 
  • Like
Reactions: Patrick

Jeggs101

Well-Known Member
Dec 29, 2010
1,528
241
63
Sounds like the CPU project all over again.

fio is the key though.

I find compressible still a nice data point since like logs are totally compressible. I also think it is less relevant now. SSD makers are filling up 6g buses with uncompressible data and pipes are only so big. If compression is done at the drive controller level, then you're still stuck on a 6g pipe to the drive. Compressing at the host could get you more effective than at the drive level, but then what you care about is uncompressible speeds anyway.

When Sandforce came out, drives could not saturate because NAND and controllers were slower than bus. Now bus is slower than controllers.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Yep, fio is the next one on my list. I wanted to squash the ones that I knew first. It looks really interesting.

Playing with both raw disk tests if I can find a way as well as filesystem based tests. Filesystem ones can be affected by driver, FS choice, etc. But you can then do the NAS tests.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,528
241
63
Yep, fio is the next one on my list. I wanted to squash the ones that I knew first. It looks really interesting.

Playing with both raw disk tests if I can find a way as well as filesystem based tests. Filesystem ones can be affected by driver, FS choice, etc. But you can then do the NAS tests.
I like file system over raw not just because of NAS, but also because that's how drives are used mostly. How many disks do you use raw at home or work?
 

Patriot

Moderator
Apr 18, 2011
1,453
793
113
I like file system over raw not just because of NAS, but also because that's how drives are used mostly. How many disks do you use raw at home or work?
While they are how it is used... raw allows a consistent measure rather than have a kernel update completely change the XFS performance attributes.
I would rather raw and the occasional FS comparisons. Play with one variable at a time.