Introducing the STHbench.sh benchmark script

Jeggs101

Well-Known Member
Dec 29, 2010
1,482
222
63
Here is my redis output where I truncated a bunch of the errors:
Code:
Starting Redis server...
Writing to socket: Connection refused
Writing to socket: Connection refused
.....
Writing to socket: Connection refused
Writing to socket: Connection refused
SET: 290782.22 requests per second
GET: 327118.09 requests per second

1 18793
5 23121
10 24143
20 24266
30 24963
40 24894
50 25006
60 25069
70 25000
80 25157
90 25063
100 25272
 

nitrobass24

Moderator
Dec 26, 2010
1,082
126
63
TX
Jegss - A few questions so i can try to replicate this.

What OS/Version?
User Account Type - Standard or Root?
Are you running the script as Sudo?
 

nitrobass24

Moderator
Dec 26, 2010
1,082
126
63
TX
You are right about build-essential. What is wrong, however, is that the apt-get has a reference to phoronix-test-suite which is not available on the debian repositories in which case the entire apt-get install after phoronix-test-suite is not applied. To fix this you should add this instead to the Update_Install_Debian() function:
wget -N http://phoronix-test-suite.com/releases/repo/pts.debian/files/phoronix-test-suite_4.8.6_all.deb && sudo dpkg -i phoronix-test-suite_4.8.6_all.deb
So this code is in the Dev05 build, but its after the apt-get cmd, so its basically useless. I am moving it up in 006.

So when I added this before Update_Install_Debian(), it ended up erroring out on my Ubuntu VM.

I think the real answer is moving away from PTS.
Might be, but im unsure what other options are better suited.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,814
4,767
113
So this code is in the Dev05 build, but its after the apt-get cmd, so its basically useless. I am moving it up in 006.



Might be, but im unsure what other options are better suited.
Ok not going to mess with this if you are nitrobass. Maybe the answer is to compile and run stream, pybench, openssl bench and 7-Zip benchmarks instead of using PTS. Will look into it.
 

nitrobass24

Moderator
Dec 26, 2010
1,082
126
63
TX
Pat - Looking at your memcached comments in the file. What do you think about memtier_bench instead? memtier_benchmark: A High-Throughput Benchmarking Tool for Redis & Memcached | Redis & Memcached

Also Dev006 is in your inbox. I made the following changes:
  • Removed all "sudo" commands - Running the whole script as sudo/root prevents the script from requiring re-auth/stalling out on slower systems as the whole script will run under a single parent process as root
  • Added check for root privileges and user alerting
  • Updated Debian Install process based on feedback from Mir
  • Added 'apt-get install -f' based on errors i saw on Ubuntu 12.04.03
  • Added Multi-threading to sysbench
  • Removed verbose flag from Tar commands
  • Fixed Crafty not found bug
  • Cleaned up some previously commented out code
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,814
4,767
113
Ha! Wow thanks nitrobass24! I saw your e-mail and was going to ask what is in this version. Will post it in two hours or so when I am at the main workstation.

Memtier_bench could be really cool! I have a feeling that is going to require a lot to get setup though.
 

mir

New Member
Dec 21, 2013
9
0
0
Since some of the software is installed outside the package repository using make install would it not be an idea to remove that software before the script terminates?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,814
4,767
113
Since some of the software is installed outside the package repository using make install would it not be an idea to remove that software before the script terminates?
It would be awesome to be able to do that. Right now the focus is more on getting a set of benchmarks completed, then really uninstalling them. Certainly see the wisdom in removal though.

Will post Dev006 as soon as I can if you wanted to take a look.
 

mir

New Member
Dec 21, 2013
9
0
0
It would be awesome to be able to do that. Right now the focus is more on getting a set of benchmarks completed, then really uninstalling them. Certainly see the wisdom in removal though.
Since Dev06 is running as user root through sudo the software will get installed on the host and stay there. Running a benchmark should not leave executable files unknown to the user running the script.
Will post Dev006 as soon as I can if you wanted to take a look.
I will take a look. Shouldn't be to difficult to make uninstall - use ./configure --prefix=show_odd_folder && make install && study what gets installed && add rm -f files_found_discovered_in_odd_folder
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,814
4,767
113
Thanks nitrobass24 and mir:

Dev006 posted: http://forums.servethehome.com/pjk/STHbench-Dev006.sh
 

mir

New Member
Dec 21, 2013
9
0
0
Is this the crafty the scripts wants to install: crafty - state-of-the-art chess engine, compatible with xboard ?
If so what is the purpose of installing this?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,814
4,767
113
mir - great question. Crafty is the chess engine. I think it is also compatible with xboard (Crafty - XBoard). We started this as Ubuntu only so we were using the repo version rather than download/ make.
 

MiniKnight

Well-Known Member
Mar 30, 2012
2,981
884
113
NYC
IMO here is what you guys should focus on:

Benchmark updates
- Get rid of crafty and PTS installs - just compile these manually
- Do the memtier that nitrobass was talking about

I still think STHbench needs some sort of normal web server nginx or apache benchmark with php. Maybe a MySQL/ MariaDB one also. I know these are hard to size if you are doing Atom and ARM up to multi processor

Script updates
- Uninstall lines
- Option to run individual tests or sets of tests
- Option to run script multiple times

One that is done, you need to start parsing and getting to an online results viewer.
 

mir

New Member
Dec 21, 2013
9
0
0
Here is a diff containing changes which includes removing what gets installed as well as some fixes for minor bugs.
Code:
--- STHbench-Dev006.sh	2013-12-24 14:23:29.982863453 +0100
+++ STHbench-Dev006.sh.new	2013-12-25 13:25:55.917188997 +0100
@@ -11,13 +11,19 @@
 
 #############Set Functions################
 
+NEED_PTS=1
+
 # Update and install required packages (Debian)
 Update_Install_Debian()
 {
 apt-get -y update && apt-get -y upgrade && apt-get install -f
-apt-get -y install build-essential libx11-dev libglu-dev hardinfo crafty sysbench expect make gcc php5-curl php5-common php5-cli php5-gd libfpdi-php
+apt-get -y install build-essential libx11-dev libglu-dev hardinfo sysbench expect php5-curl php5-common php5-cli php5-gd libfpdi-php
 apt-get install -f
-wget -N http://phoronix-test-suite.com/releases/repo/pts.debian/files/phoronix-test-suite_4.8.6_all.deb && dpkg -i phoronix-test-suite_4.8.6_all.deb
+dpkg -s phoronix-test-suite 2>&1 > /dev/null 2>&1
+NEED_PTS=$(echo $?)
+if [ $NEED_PTS > 0 ]; then
+    wget -N http://phoronix-test-suite.com/releases/repo/pts.debian/files/phoronix-test-suite_4.8.6_all.deb && dpkg -i phoronix-test-suite_4.8.6_all.deb
+fi
 }
 # Update and install required packages (CentOS/RHEL)
 Update_Install_RHEL()
@@ -96,11 +102,6 @@
 # C-Ray 1.1
 wget -N http://www.futuretech.blinkenlights.nl/depot/c-ray-1.1.tar.gz && tar -zxf c-ray-1.1.tar.gz && cd c-ray-1.1 && make && cat scene | ./c-ray-mt -t 32 -s 7500x3500 > foo.ppm | tee c-ray1.txt && cat sphfract | ./c-ray-mt -t 32 -s 1920x1200 -r 8 > foo.ppm && cd ..
 
-# Crafty
-if [[ $DIST = "Debian" ]]; then
-/usr/games/crafty 'log off' bench end
-fi
-
 # Phoroncrix Test Suite
 expect <<EOD
 set timeout -1
@@ -133,9 +134,6 @@
 cp utils/redis_init_script /etc/init.d/redis_6379
 [[ -d /var/redis ]] || ( mkdir /var/redis && mkdir /var/redis/6379 )
 
-cd src
-cp redis-server redis-cli redis-benchmark /usr/local/bin/
-
 service redis_6379 start
 
 # Original redis benchmark set/ get test
@@ -165,4 +163,30 @@
 
 redis-cli shutdown
 
+cd ..
+
+# Remove Redis
+rm -rf /etc/redis
+rm -f /etc/init.d/redis_6379
+rm -rf /var/redis
+rm -f /usr/local/bin/redis-benchmark
+rm -f /usr/local/bin/redis-check-aof
+rm -f /usr/local/bin/redis-check-dump
+rm -f /usr/local/bin/redis-cli
+rm -f /usr/local/bin/redis-server
+rm -f redis-stable.tar.gz
+rm -rf redis-stable
+
+# Remove c-ray
+rm -f c-ray-1.1.tar.gz
+rm -rf c-ray-1.1
+
+# Remove UnixBench5.1.3.tgz
+rm -f UnixBench5.1.3.tgz
+rm -rf UnixBench
+
+# Remove Phoronix test suite
+[ "$NEED_PTS" > "0" -a "$DIST" = "Debian" ] && apt-get -y --purge remove phoronix-test-suite && rm -f phoronix-test-suite_4.8.6_all.deb
+
 cd ~/
+
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
11,814
4,767
113
Thanks Mir! May be a day or two before I can update due to the holidays but thank you so much for your help on this.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,482
222
63
mir can you post your new version? I have some time this weekend so I can give it a go.
 

mir

New Member
Dec 21, 2013
9
0
0
mir can you post your new version? I have some time this weekend so I can give it a go.
Here it is
Code:
#!/bin/bash

# (c) 2013 ServeTheHome.com and ServeThe.biz

# Verify if the script is executed with Root Privileges #
ROOT=`/usr/bin/id -u`
if [[ $ROOT -ne "0" ]]; then
echo "Please execute the script with root privileges"
exit 1
fi

#############Set Functions################

NEED_PTS=1

# Update and install required packages (Debian)
Update_Install_Debian()
{
apt-get -y update && apt-get -y upgrade && apt-get install -f
apt-get -y install build-essential libx11-dev libglu-dev hardinfo sysbench expect php5-curl php5-common php5-cli php5-gd libfpdi-php
apt-get install -f
dpkg -s phoronix-test-suite 2>&1 > /dev/null 2>&1
NEED_PTS=$(echo $?)
if [ $NEED_PTS > 0 ]; then
    wget -N http://phoronix-test-suite.com/releases/repo/pts.debian/files/phoronix-test-suite_4.8.6_all.deb && dpkg -i phoronix-test-suite_4.8.6_all.deb
fi
}
# Update and install required packages (CentOS/RHEL)
Update_Install_RHEL()
{
rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -Uhv http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
yum -y update && yum -y upgrade
yum -y groupinstall "Development Tools" && yum -y install wget sysbench libX11 perl-Time-HiRes mesa-libGLU hardinfo phoronix-test-suite expect php-common glibc.i686
}

#Determine OS
# Detects which OS and if it is Linux then it will detect which Linux Distribution.
OS=`uname -s`
REV=`uname -r`
MACH=`uname -m`

if [ "${OS}" = "SunOS" ] ; then
	OS=Solaris
	DIST=Solaris
	ARCH=`uname -p`	
	OSSTR="${OS} ${REV}(${ARCH} `uname -v`)"
elif [ "${OS}" = "AIX" ] ; then
	DIST=AIX
	OSSTR="${OS} `oslevel` (`oslevel -r`)"
elif [ "${OS}" = "Linux" ] ; then
	KERNEL=`uname -r`
	if [ -f /etc/redhat-release ] ; then
		DIST='RedHat'
		PSUEDONAME=`cat /etc/redhat-release | sed s/.*\(// | sed s/\)//`
		REV=`cat /etc/redhat-release | sed s/.*release\ // | sed s/\ .*//`
	elif [ -f /etc/centos-release ] ; then
		DIST='CentOS'
		PSUEDONAME=`cat /etc/centos-release | sed s/.*\(// | sed s/\)//`
		REV=`cat /etc/centos-release | sed s/.*release\ // | sed s/\ .*//`
	elif [ -f /etc/SUSE-release ] ; then
		DIST=`cat /etc/SUSE-release | tr "\n" ' '| sed s/VERSION.*//`
		REV=`cat /etc/SUSE-release | tr "\n" ' ' | sed s/.*=\ //`
	elif [ -f /etc/mandrake-release ] ; then
		DIST='Mandrake'
		PSUEDONAME=`cat /etc/mandrake-release | sed s/.*\(// | sed s/\)//`
		REV=`cat /etc/mandrake-release | sed s/.*release\ // | sed s/\ .*//`
	elif [ -f /etc/debian_version ] ; then
		DIST="Debian"
		PSUEDONAME=`cat /etc/debian_version`
		REV=""
	fi
	if [ -f /etc/UnitedLinux-release ] ; then
		DIST="${DIST}[`cat /etc/UnitedLinux-release | tr "\n" ' ' | sed s/VERSION.*//`]"
	fi
	OSSTR="${OS} ${DIST} ${REV}(${PSUEDONAME} ${KERNEL} ${MACH})"
fi

##########Update and install required packages###########

if [ "${DIST}" = "CentOS" ] ; then
Update_Install_RHEL
elif [ "${DIST}" = "RedHat" ] ; then
Update_Install_RHEL
elif [ "${DIST}" = "Debian" ] ; then
Update_Install_Debian
fi

# Display script output and append to log
exec > >(tee --append "$(basename $0)".log)
exec 2>&1
echo ${OSSTR}

#############Run Benchmarks###############

# HardInfo
hardinfo --generate-report --report-format text

# UnixBench 5.1.3
wget -N https://byte-unixbench.googlecode.com/files/UnixBench5.1.3.tgz && tar -zxf UnixBench5.1.3.tgz && cd UnixBench && time make && wget -N http://forums.servethehome.com/pjk/fix-limitation.patch && patch Run fix-limitation.patch && ./Run dhry2reg whetstone-double syscall pipe context1 spawn execl shell1 shell8 shell16 && cd ..

# C-Ray 1.1
wget -N http://www.futuretech.blinkenlights.nl/depot/c-ray-1.1.tar.gz && tar -zxf c-ray-1.1.tar.gz && cd c-ray-1.1 && make && cat scene | ./c-ray-mt -t 32 -s 7500x3500 > foo.ppm | tee c-ray1.txt && cat sphfract | ./c-ray-mt -t 32 -s 1920x1200 -r 8 > foo.ppm && cd ..

# Phoroncrix Test Suite
expect <<EOD
set timeout -1
spawn -noecho phoronix-test-suite batch-setup
expect {
   "Do you agree to these terms and wish to proceed (Y/n):" { send "y\n"; exp_continue }
   "Enable anonymous usage / statistics reporting (Y/n):" { send "n\n"; exp_continue }
   "Enable anonymous statistical reporting of installed software / hardware (Y/n):" { send "n\n"; exp_continue }

   "Run all test options (Y/n):" { send "y\n"; exp_continue }
   "Save test results when in batch mode (Y/n):" { send "n\n"; exp_continue }
}
EOD
phoronix-test-suite batch-benchmark pts/stream pts/compress-7zip pts/openssl pts/pybench

# sysbench CPU test prime
echo "Running sysbench CPU Single Thread"
sysbench --test=cpu --cpu-max-prime=1000000 run
echo "Running sysbench CPU Multi-Threaded"
nproc=`nproc`
sysbench --num-threads=${nproc} --test=cpu --cpu-max-prime=1000000 run

# redis Benchmark based on feedback. Next step is to add memchached as seen here: http://oldblog.antirez.com/post/redis-memcached-benchmark.html
wget http://download.redis.io/redis-stable.tar.gz
tar xzf redis-stable.tar.gz && cd redis-stable && make install

wget http://forums.servethehome.com/pjk/6379.conf
[[ -d /etc/redis ]] || ( mkdir /etc/redis && cp 6379.conf /etc/redis/6379.conf )

cp utils/redis_init_script /etc/init.d/redis_6379
[[ -d /var/redis ]] || ( mkdir /var/redis && mkdir /var/redis/6379 )

service redis_6379 start

# Original redis benchmark set/ get test

redis-benchmark -n 1000000 -t set,get -P 32 -q -c 200

BIN=redis-benchmark

payload=32
iterations=100000
keyspace=100000

for clients in 1 5 10 20 30 40 50 60 70 80 90 100
do
    SPEED=0
    for dummy in 0 1 2
    do
        S=$($BIN -n $iterations -r $keyspace -d $payload -c $clients | grep 'per second' | tail -1 | awk '{print $1}')
	VALUE=$(echo $S | awk '{printf "%.0f",$1}')
        if [ $(($VALUE > $SPEED)) != 0 ]
        then
            SPEED=$VALUE
        fi
    done
    echo "$clients $SPEED"
done

redis-cli shutdown

cd ..

# Remove Redis
rm -rf /etc/redis
rm -f /etc/init.d/redis_6379
rm -rf /var/redis
rm -f /usr/local/bin/redis-benchmark
rm -f /usr/local/bin/redis-check-aof
rm -f /usr/local/bin/redis-check-dump
rm -f /usr/local/bin/redis-cli
rm -f /usr/local/bin/redis-server
rm -f redis-stable.tar.gz
rm -rf redis-stable

# Remove c-ray
rm -f c-ray-1.1.tar.gz
rm -rf c-ray-1.1

# Remove UnixBench5.1.3.tgz
rm -f UnixBench5.1.3.tgz
rm -rf UnixBench

# Remove Phoronix test suite
[ "$NEED_PTS" > "0" -a "$DIST" = "Debian" ] && apt-get -y --purge remove phoronix-test-suite && rm -f phoronix-test-suite_4.8.6_all.deb

cd ~/
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,814
4,767
113
Uploaded and added Dev007 based on mir's updates above. Testing now.

Super cool!
 

TangoWhiskey9

Active Member
Jun 28, 2013
402
59
28
Dev007 runs really slowly specifically on the prime single-threaded calculation. Took almost 2 hours one one of my lower clocked machines.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,482
222
63
Patrick sent 13.10 server output.

Bugs noted:
- Crafty seems to be gone in that version.
- pts is not working