Introducing the STHbench.sh benchmark script

Discussion in 'Processors and Motherboards' started by Patrick, Oct 6, 2013.

  1. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,463
    Likes Received:
    214
    C2750 w/ 4x 4GB 1600MHz DIMMs:

    Function Best Rate MB/s
    Copy: 15215.0
    Scale: 9728.2
    Add: 10564.3
    Triad: 10498.7

    Aren't specs 2x that?
     
    #101
  2. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,463
    Likes Received:
    214
    And Haswell E3-1220v3 4x 1600MHz DIMMs

    Function Best Rate MB/s
    Copy: 18561.4
    Scale: 12533.4
    Add: 13933.4
    Triad: 13952.7

    Your script works really well. Super easy to run.
     
    #102
  3. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,463
    Likes Received:
    214
    Hey Patrick/ everyone,

    Here's an idea:
    What if there was a results table at the end. So you didn't have to hunt for the results? Right now Dev008 is still giving many thousands of lines in the log. Not bad, but would be easier if it just had either benchmark output OR benchmark output + config. Only reason for config is so that I can archive for later.
     
    #103
  4. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,498
    Likes Received:
    4,441
    Updated Dev009.sh with the NPB and STREAM scripts.
    Chuckleb added to credits.

    This is really moving along this week.

    Just so everyone knows, once this gets stable, the plan is to get it parsed (will likely pay someone to do it.) Jeggs101 that should help you quite a bit. Another thing I guess we could do is to stop adding to the log during the wget and make portions. That would make it much easier to see benchmark results. The current output is many thousands of lines.

    Going to start running these on CentOS 6.5 and Ubuntu 13.10.
     
    #104
  5. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    I adapted the Stream tests from TangoWhiskey9 originally so feel free to credit him as well, otherwise I wouldn't have bothered to test. :)
     
    #105
  6. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    Had a chance to test scripts on my AMD systems and the 6100/6200 CPUs reports cores differently than Intel. The Opterons put two physical chips on a die, each with its own memory controller so the method of counting cores doesn't work. What I'm lokoing for is to isolate the NUMA nodes anyway, to lock jobs within their nodes.

    Here's a revised section that uses lscpu to get NUMA Node count and if that doesn't exist, drops back to the first model. This has been tested on 1P, 2P and 4P AMD 6100 and AMD 6200 systems, as well as Intel E5 series 1P and 2P systems. This can replace the existing sections in the npb and stream scripts. I used the stream tests to verify. It's still not ideal, but lscpu seems to exist on Debian, Ubuntu, and Centos/RH. I don't have a Deb or Ubuntu box so can't confirm that it parses the same. Feel free to tell me if it doesn't.

    Code:
    # Determine number of physical cores (not hyperthread) and set OMP to cores value
    
    
    if [ -e /usr/bin/lscpu ]
    then
    	numanodes=$(/usr/bin/lscpu | grep "NUMA node(s):" |cut -d":" -f2)
    	coresocket=$(/usr/bin/lscpu | grep "Core(s) per socket:" |cut -d":" -f2)
    	cores=$((numanodes*coresocket))						   
    else
    	procs=$(grep "physical id" /proc/cpuinfo | sort -u | wc -l)
    	pcores=$(grep "cpu cores" /proc/cpuinfo |sort -u |cut -d":" -f2)
    	cores=$((procs*pcores))
    fi
    
    
    export OMP_NUM_THREADS=$cores
    export GOMP_CPU_AFFINITY=0-$((cores-1))
    echo $GOMP_CPU_AFFINITY
    
    
    
    Jeggs, I've been running stream and the results seem consistent. I'll keep testing.

    Thanks
     
    #106
  7. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    Got bored. Here's another test that is commonly run. NAMD. This is highly threaded so I am using all threads so whatever the OS reports is fine. I even made the output look pretty. For reference, my results:

    Using 32 threads
    Running benchmark... (will take a while)
    Time per step 0.0825135

    Using 16 threads
    Running benchmark... (will take a while)
    Time per step 0.0971788

    More results are here: http://www.ks.uiuc.edu/Research/namd/performance.html


    Code:
    #!/bin/bash
    
    
    cores=$(grep "processor" /proc/cpuinfo | wc -l)
    
    
    wget https://dl.dropboxusercontent.com/u/124184/NAMD_2.9_Linux-x86_64-multicore.tar.gz
    wget https://dl.dropboxusercontent.com/u/124184/apoa1.tar.gz
    
    
    tar xvfz NAMD_2.9_Linux-x86_64-multicore.tar.gz 
    tar xvfz apoa1.tar.gz
    
    
    echo "Using" $cores "threads"
    echo "Running benchmark... (will take a while)"
    
    
    cd NAMD_2.9_Linux-x86_64-multicore
    timeperstep=$(./namd2 +p$cores +setcpuaffinity ../apoa1/apoa1.namd | grep "Benchmark time" | tail -1 | cut -d" " -f6)
    
    
    
    
    echo "Time per step" $timeperstep
    
    
     
    #107
    Last edited: Jan 5, 2014
  8. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    I wanted to get baseline numbers for others to use as they run tests so I populated a spreadsheet and made it open to the world. Yes, smart me, can't wait for the spam.

    Anyway, feel free to put in results if you want, or use mine as comparisons. I'm going to run it on a bunch of my systems as I can get time on them. Also feel free to add more columns, I only added for the ones that I've been working on, I can go back and run more tests on those systems as well.

    http://goo.gl/t96G4f

    As usual, tell me if you have problems.

    Thanks,
     
    #108
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,498
    Likes Received:
    4,441
    Chuckleb, seen this one? Added two to your results excel

    apoa1/
    apoa1/apoa1.pdb
    apoa1/apoa1.psf
    apoa1/par_all22_prot_lipid.xplor
    apoa1/par_all22_popc.xplor
    apoa1/apoa1.namd
    Using 4 threads
    Running benchmark... (will take a while)
    ------------- Processor 0 Exiting: Called CmiAbort ------------
    Reason: FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


    Charm++ fatal error:
    FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


    Time per step 0.830613
    root@Avoton2550:~#
     
    #109
  10. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    Odd, it couldn't read the file and you're root, so it should have been created. I'd try running from a folder in your homedir perhaps? Just to be sure permissions are good and everything is contained. That's how I ran all my tests since my homedir is globally mounted so I didn't have to copy things around. But yes, seems to be a permission issue that it couldn't write out the output file so it couldn't read it in Charm. Mind you that you got results of 0.83 or so. ;)
     
    #110
  11. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,498
    Likes Received:
    4,441
    Seems like that was it. I'm running as user "patrick" now :)
     
    #111
  12. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,498
    Likes Received:
    4,441
    Hmmm... what is writing to /usr/tmp?

    Running benchmark... (will take a while)
    ------------- Processor 0 Exiting: Called CmiAbort ------------
    Reason: FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


    Charm++ fatal error:
    FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


    Time per step 0.451986
    patrick@rangeley:~$

    and

    Running benchmark... (will take a while)
    ------------- Processor 0 Exiting: Called CmiAbort ------------
    Reason: FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


    Charm++ fatal error:
    FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


    Time per step 0.831208
    patrick@Avoton2550:~$
     
    #112
  13. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    /usr/tmp/ does not exists on Ubuntu systems by default.
     
    #113
  14. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    Ah, there's the genius for ya! This was a precompiled binary vs source so I couldn't troubleshoot much. I need to fire up more VMs with different distros I guess.

    Guess easy solution, not preferred, is test for /usr/tmp but regular user account can't create the folder. Might have to modify source route.
     
    #114
  15. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,498
    Likes Received:
    4,441
    Ha! Good point. Just was going to report back that one too.

    Chuckleb - if we stick in the main script, it gets run as su so that should help. Compiling may also be good though.
     
    #115
  16. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Chuckle...not to take the wind out of your sails, but I think before we start modifying source, we need to take a step back and think a little more big picture.

    What are we really trying to test and communicate in the reviews?
    How do our benchmarks either compared to other review sites?
    What makes our benchmarks better/more useful/realworld compared to other sites?

    Based on those answers decide what we really want to bench for. I think we can all find all kinds of benchmarks out there that test various things, but what does it mean to the casual reader? How can they take this and compare to another product that was only reviewed on a different site?

    I think we should do some analysis of what other sites are doing and add some of those same benches where it makes sense to have comparability. I think we need to focus on integrating those value-add benchmarks that we have all talked about. ngnix, apache, memcached, etc.

    What do you guys think?
     
    #116
  17. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Honestly....no need to even modify the source, just a add a mkdir -p /usr/tmp/ to the Debian install function.
     
    #117
    Last edited: Jan 6, 2014
  18. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    Heh, I generally try to not run things as root and force my student workers to do the same. We generally don't have root permissions on other systems that we log onto so it's a good habit for them to figure out how to get things to work. I didn't realize that things were being run as root.

    Regarding the other point of what the overall goal is, yes, I think the group needs to figure out what it wants to do. I don't generally run benchmarks and when I do, it's pretty specific to the project at hand so the generic systemwide numbers don't really help. On top of that, we get pretty limited in choice for what to buy. For home, it's usually money that is the first factor. That said, this has been interesting to run since I've finally bothered to compare my systems and see what they can do on more standardized tests.

     
    #118
  19. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,498
    Likes Received:
    4,441
    Ok will add to Dev010. I ran Dev009 on CentOS and (not) surprisingly it ran all the way through without crashing. Good sign.
     
    #119
  20. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,498
    Likes Received:
    4,441
    OK so here are the changes:
    Added the mkdir to the debian install on Dev010
    Hosting the binaries of NAMD
    Added NAMD

    Going to run before I update the first post. Until then http://forums.servethehome.com/pjk/STHbench-Dev010.sh

    And so we do not use too much dropbox bandwidth:
    Code:
    #!/bin/bash
    # NAMD Benchmark http://www.ks.uiuc.edu/Research/namd/performance.html
    cores=$(grep "processor" /proc/cpuinfo | wc -l)
    
    
    wget http://forums.servethehome.com/pjk/NAMD_2.9_Linux-x86_64-multicore.tar.gz
    wget http://forums.servethehome.com/pjk/124184/apoa1.tar.gz
    
    
    tar xvfz NAMD_2.9_Linux-x86_64-multicore.tar.gz 
    tar xvfz apoa1.tar.gz
    
    
    echo "Using" $cores "threads"
    echo "Running benchmark... (will take a while)"
    
    
    cd NAMD_2.9_Linux-x86_64-multicore
    timeperstep=$(./namd2 +p$cores +setcpuaffinity ../apoa1/apoa1.namd | grep "Benchmark time" | tail -1 | cut -d" " -f6)
    
    
    echo "Time per step" $timeperstep
    
     
    #120

Share This Page