Introducing the STHbench.sh benchmark script

Jeggs101

Well-Known Member
Dec 29, 2010
1,489
228
63
C2750 w/ 4x 4GB 1600MHz DIMMs:

Function Best Rate MB/s
Copy: 15215.0
Scale: 9728.2
Add: 10564.3
Triad: 10498.7

Aren't specs 2x that?
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,489
228
63
And Haswell E3-1220v3 4x 1600MHz DIMMs

Function Best Rate MB/s
Copy: 18561.4
Scale: 12533.4
Add: 13933.4
Triad: 13952.7

Your script works really well. Super easy to run.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,489
228
63
Hey Patrick/ everyone,

Here's an idea:
What if there was a results table at the end. So you didn't have to hunt for the results? Right now Dev008 is still giving many thousands of lines in the log. Not bad, but would be easier if it just had either benchmark output OR benchmark output + config. Only reason for config is so that I can archive for later.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,965
4,924
113
Updated Dev009.sh with the NPB and STREAM scripts.
Chuckleb added to credits.

This is really moving along this week.

Just so everyone knows, once this gets stable, the plan is to get it parsed (will likely pay someone to do it.) Jeggs101 that should help you quite a bit. Another thing I guess we could do is to stop adding to the log during the wget and make portions. That would make it much easier to see benchmark results. The current output is many thousands of lines.

Going to start running these on CentOS 6.5 and Ubuntu 13.10.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
I adapted the Stream tests from TangoWhiskey9 originally so feel free to credit him as well, otherwise I wouldn't have bothered to test. :)
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Had a chance to test scripts on my AMD systems and the 6100/6200 CPUs reports cores differently than Intel. The Opterons put two physical chips on a die, each with its own memory controller so the method of counting cores doesn't work. What I'm lokoing for is to isolate the NUMA nodes anyway, to lock jobs within their nodes.

Here's a revised section that uses lscpu to get NUMA Node count and if that doesn't exist, drops back to the first model. This has been tested on 1P, 2P and 4P AMD 6100 and AMD 6200 systems, as well as Intel E5 series 1P and 2P systems. This can replace the existing sections in the npb and stream scripts. I used the stream tests to verify. It's still not ideal, but lscpu seems to exist on Debian, Ubuntu, and Centos/RH. I don't have a Deb or Ubuntu box so can't confirm that it parses the same. Feel free to tell me if it doesn't.

Code:
# Determine number of physical cores (not hyperthread) and set OMP to cores value


if [ -e /usr/bin/lscpu ]
then
	numanodes=$(/usr/bin/lscpu | grep "NUMA node(s):" |cut -d":" -f2)
	coresocket=$(/usr/bin/lscpu | grep "Core(s) per socket:" |cut -d":" -f2)
	cores=$((numanodes*coresocket))						   
else
	procs=$(grep "physical id" /proc/cpuinfo | sort -u | wc -l)
	pcores=$(grep "cpu cores" /proc/cpuinfo |sort -u |cut -d":" -f2)
	cores=$((procs*pcores))
fi


export OMP_NUM_THREADS=$cores
export GOMP_CPU_AFFINITY=0-$((cores-1))
echo $GOMP_CPU_AFFINITY
Jeggs, I've been running stream and the results seem consistent. I'll keep testing.

Thanks
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Got bored. Here's another test that is commonly run. NAMD. This is highly threaded so I am using all threads so whatever the OS reports is fine. I even made the output look pretty. For reference, my results:

Using 32 threads
Running benchmark... (will take a while)
Time per step 0.0825135

Using 16 threads
Running benchmark... (will take a while)
Time per step 0.0971788

More results are here: http://www.ks.uiuc.edu/Research/namd/performance.html


Code:
#!/bin/bash


cores=$(grep "processor" /proc/cpuinfo | wc -l)


wget https://dl.dropboxusercontent.com/u/124184/NAMD_2.9_Linux-x86_64-multicore.tar.gz
wget https://dl.dropboxusercontent.com/u/124184/apoa1.tar.gz


tar xvfz NAMD_2.9_Linux-x86_64-multicore.tar.gz 
tar xvfz apoa1.tar.gz


echo "Using" $cores "threads"
echo "Running benchmark... (will take a while)"


cd NAMD_2.9_Linux-x86_64-multicore
timeperstep=$(./namd2 +p$cores +setcpuaffinity ../apoa1/apoa1.namd | grep "Benchmark time" | tail -1 | cut -d" " -f6)




echo "Time per step" $timeperstep
 
Last edited:

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
I wanted to get baseline numbers for others to use as they run tests so I populated a spreadsheet and made it open to the world. Yes, smart me, can't wait for the spam.

Anyway, feel free to put in results if you want, or use mine as comparisons. I'm going to run it on a bunch of my systems as I can get time on them. Also feel free to add more columns, I only added for the ones that I've been working on, I can go back and run more tests on those systems as well.

http://goo.gl/t96G4f

As usual, tell me if you have problems.

Thanks,
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,965
4,924
113
Chuckleb, seen this one? Added two to your results excel

apoa1/
apoa1/apoa1.pdb
apoa1/apoa1.psf
apoa1/par_all22_prot_lipid.xplor
apoa1/par_all22_popc.xplor
apoa1/apoa1.namd
Using 4 threads
Running benchmark... (will take a while)
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


Charm++ fatal error:
FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


Time per step 0.830613
root@Avoton2550:~#
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Odd, it couldn't read the file and you're root, so it should have been created. I'd try running from a folder in your homedir perhaps? Just to be sure permissions are good and everything is contained. That's how I ran all my tests since my homedir is globally mounted so I didn't have to copy things around. But yes, seems to be a permission issue that it couldn't write out the output file so it couldn't read it in Charm. Mind you that you got results of 0.83 or so. ;)
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,965
4,924
113
Seems like that was it. I'm running as user "patrick" now :)
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,965
4,924
113
Hmmm... what is writing to /usr/tmp?

Running benchmark... (will take a while)
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


Charm++ fatal error:
FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


Time per step 0.451986
patrick@rangeley:~$

and

Running benchmark... (will take a while)
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


Charm++ fatal error:
FATAL ERROR: Error opening XSC output file /usr/tmp/apoa1-out.xsc: No such file or directory


Time per step 0.831208
patrick@Avoton2550:~$
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Ah, there's the genius for ya! This was a precompiled binary vs source so I couldn't troubleshoot much. I need to fire up more VMs with different distros I guess.

Guess easy solution, not preferred, is test for /usr/tmp but regular user account can't create the folder. Might have to modify source route.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,965
4,924
113
/usr/tmp/ does not exists on Ubuntu systems by default.
Ha! Good point. Just was going to report back that one too.

Chuckleb - if we stick in the main script, it gets run as su so that should help. Compiling may also be good though.
 

nitrobass24

Moderator
Dec 26, 2010
1,083
127
63
TX
Chuckle...not to take the wind out of your sails, but I think before we start modifying source, we need to take a step back and think a little more big picture.

What are we really trying to test and communicate in the reviews?
How do our benchmarks either compared to other review sites?
What makes our benchmarks better/more useful/realworld compared to other sites?

Based on those answers decide what we really want to bench for. I think we can all find all kinds of benchmarks out there that test various things, but what does it mean to the casual reader? How can they take this and compare to another product that was only reviewed on a different site?

I think we should do some analysis of what other sites are doing and add some of those same benches where it makes sense to have comparability. I think we need to focus on integrating those value-add benchmarks that we have all talked about. ngnix, apache, memcached, etc.

What do you guys think?
 

nitrobass24

Moderator
Dec 26, 2010
1,083
127
63
TX
Ha! Good point. Just was going to report back that one too.

Chuckleb - if we stick in the main script, it gets run as su so that should help. Compiling may also be good though.
Honestly....no need to even modify the source, just a add a mkdir -p /usr/tmp/ to the Debian install function.
 
Last edited:

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Heh, I generally try to not run things as root and force my student workers to do the same. We generally don't have root permissions on other systems that we log onto so it's a good habit for them to figure out how to get things to work. I didn't realize that things were being run as root.

Regarding the other point of what the overall goal is, yes, I think the group needs to figure out what it wants to do. I don't generally run benchmarks and when I do, it's pretty specific to the project at hand so the generic systemwide numbers don't really help. On top of that, we get pretty limited in choice for what to buy. For home, it's usually money that is the first factor. That said, this has been interesting to run since I've finally bothered to compare my systems and see what they can do on more standardized tests.

Honestly....no need to event modify the source, just a add a mkdir -p /usr/tmp/ to the Debian install function.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,965
4,924
113
Honestly....no need to even modify the source, just a add a mkdir -p /usr/tmp/ to the Debian install function.
Ok will add to Dev010. I ran Dev009 on CentOS and (not) surprisingly it ran all the way through without crashing. Good sign.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,965
4,924
113
OK so here are the changes:
Added the mkdir to the debian install on Dev010
Hosting the binaries of NAMD
Added NAMD

Going to run before I update the first post. Until then http://forums.servethehome.com/pjk/STHbench-Dev010.sh

And so we do not use too much dropbox bandwidth:
Code:
#!/bin/bash
# NAMD Benchmark http://www.ks.uiuc.edu/Research/namd/performance.html
cores=$(grep "processor" /proc/cpuinfo | wc -l)


wget http://forums.servethehome.com/pjk/NAMD_2.9_Linux-x86_64-multicore.tar.gz
wget http://forums.servethehome.com/pjk/124184/apoa1.tar.gz


tar xvfz NAMD_2.9_Linux-x86_64-multicore.tar.gz 
tar xvfz apoa1.tar.gz


echo "Using" $cores "threads"
echo "Running benchmark... (will take a while)"


cd NAMD_2.9_Linux-x86_64-multicore
timeperstep=$(./namd2 +p$cores +setcpuaffinity ../apoa1/apoa1.namd | grep "Benchmark time" | tail -1 | cut -d" " -f6)


echo "Time per step" $timeperstep