Railsbench

Railsbench is a small collection of ruby and shell scripts which make measuring raw performance of rails apps a snap. All tests are run from the command prompt making performance regression testing easy.

Railsbench measures the raw performance of Rails request processing, ignoring the time spent passing the request from the web server to the Rails application.

In addition, a patch for the ruby garbage collector is provided, which can be used to reduce the amount of time spent doing garbage collection, trading memory for speed, as usual (see file GCPATCH for details).

This software was written and conceived by Stefan Kaes. The author can be reached via email: <skaes at gmx dot net>. Please send comments, bug reports, patches or feature requests to this address.

DOWNLOAD

The latest version is available here.

INSTALLATION

See file INSTALL contained in the distribution.

FILES

site-ruby/benchmark.rb
replacement for the one that comes with the ruby distribution
install into the distributions site-ruby directory
lib/railsbench.rb
defines classes RailsBenchmark and RailsBenchmarkWithActiveRecordStore
switches:
-gcXXX  perform gc after XXX requests
-log  turn on rails logging
-nocache  turn off rails caching
-path  print $: after loading rails and exit
config/bechmarks.yml
specification of urls to benchmark
install into $RAILS_ROOT/config
config/benchmarks.rb
defines constant RAILSBENCH which is used by script perf_bench
modify this to add custom argument processing
install to $RAILS_ROOT/config
script/perf_bench n options
main ruby script to run a given benchmark
switches (in addition to railsbench switches):
-mix  alternates urls in given benchmark
script/run_urls n options
run a given benchmark without benchmarking
switches as for perf_bench
script/perf_run n [ option-string [ config-name ] ]
run a given benchmark, store performance data in a file in directory $RAILS_PERF_DATA and print results
script/perf_diff n common-options option-string1 option-string2 [ config-name1 [ config-name2 ] ]
run a given benchmark with two different sets of arguments
store data into directory $RAILS_PERF_DATA and print comparison data
script/perf_loop n options
used by perf_run and perf_diff
calls perf_bench $RAILS_PERF_RUNS times
script/perf_times file
analyse and print performance data
script/perf_comp file
compare two performance data sets and print results

ENVIRONMENT

RAILS_ROOT
must be set to point to your rails app
RAILS_PERF_DATA
performance data sets will be stored into this directory
if not set, $HOME will be used
RAILS_PERF_RUNS
the number of times perf_loop will run perf_bench on a single invocation
if not set, 3 runs will be performed
RAILS_BENCHMARK_FILE
perf_bench send it's output to this file

USAGE

The two main scripts are named perf_run and perf_diff.

perf_run 100
runs the list of urls named "default" specified in benchmkarks.yml (see below), requesting each url $RAILS_PERF_RUNS * 100 times.
perf_run 100 "-bm=list -aws"
runs the benchmark named 'list' and passes the expanded second argument to the rails app. By processing arguments inside your environment.rb file, you can set performance affecting options. For example, you could load action web service only if -aws is passed and measure the performance effect of omitting it.

Benchmark data is stored in directory $RAILS_PERF_DATA, which should be set in your environment. If not set, $HOME is used. By default, data is stored in file $RAILS_PERF_DATA/perf_run.$BENCHMARK.txt, where $BENCHMARK will be set according to the -bm option.

perf_run 100 "-bm=index -mail" mail
will store benchmark data in file $RAILS_PERF_DATA/`current-date´.index.mail.txt.

You can get nicely formatted output of benchmark data by running

perf_times file
The ouput of perf_times will look similar to 
perf data file: /d/perfdata/09-15.all.native_routing.txt
    requests=1000, options=-p3 -bm=all -mysql_session -fast_routes

loading environment                0.40159

page request                          total  stddev%     r/s    ms/r
/empty/index                        1.20234   0.4098   831.7    1.20
/welcome/index                      1.33902   0.1074   746.8    1.34
/rezept/index                       1.43960   0.7505   694.6    1.44
/rezept/myknzlpzl                   1.42853   0.2709   700.0    1.43
/rezept/show/713                    3.85798   0.0605   259.2    3.86
/rezept/cat/Hauptspeise             4.43199   0.0389   225.6    4.43
/rezept/cat/Hauptspeise?page=5      4.55130   0.2091   219.7    4.55
/rezept/letter/G                    4.49349   0.0627   222.5    4.49
For each page specified, perf_times n will print the mean elapsed time for running it n times, accompanied be the standard deviation in % and the mean number of request per second as well as milliseconds per request. Script perf_run will automatically print the obtained data after finishing the run using perf_times.

Script perf_diff runs a list of urls with two different option lists. So

perf_diff 100 "-bm=blogs -mysql_session" "-mail=0" "-mail=1" cf1 cf2
would run benchmark 'blogs' twice, first as
perf_run 100 "-bm=blogs -mysql_session -mail=0" cf1
and then
perf_run 100 "-bm=blogs -mysql_session -mail=1" cf2
printing a comparison of the bechmark data obtained after finishing the second run. cf1 and cf2 can be omitted, in which case data is stored in $RAILS_PERF_DATA/perf_run1.$BENCHMARK.txt and $RAILS_PERF_DATA/perf_run2.$BENCHMARK.txt.

The output of perf_comp will look like this: 
perf data file 1: 09-05.all.trunk
  requests=1000, options=-trunk -bm=all -mysql_session -routing_cache -gc100

perf data file 2: 09-05.all.trunk.ar_patched
  requests=1000, options=-trunk -bm=all -mysql_session -routing_cache -gc100

page                               c1 real   c2 real    r/s    r/s   ms/r   ms/r  c1/c2
/empty/index                       1.71842   1.72319  581.9  580.3   1.72   1.72   1.00
/welcome/index                     1.88373   1.87635  530.9  532.9   1.88   1.88   1.00
/rezept/index                      2.01678   2.08301  495.8  480.1   2.02   2.08   0.97
/rezept/myknzlpzl                  2.02136   2.07894  494.7  481.0   2.02   2.08   0.97
/rezept/show/713                   5.93261   4.67877  168.6  213.7   5.93   4.68   1.27
/rezept/cat/Hauptspeise            6.72034   4.94570  148.8  202.2   6.72   4.95   1.36
/rezept/cat/Hauptspeise?page=5     6.85539   5.07756  145.9  196.9   6.86   5.08   1.35
/rezept/letter/G                   6.63041   5.03189  150.8  198.7   6.63   5.03   1.32

Script perf_bench can also be invoked manually to run a given benchmark like so:

perf_bench 100 -bm=blogs -mysql_session -mail=1 >/dev/null
Performance data is sent to $RAILS_BENCHMARK_FILE, HTML output ends up on stdout. If $RAILS_BENCHMARK_FILE is not set, performance data is sent to stderr.

CONFIGURATION

Benchmarks are configured through file benchmarks.yml. Example: 

default:
  index, other

index:
  uri: /test/index
  new_session: true

other:
  -
    uri: /test/list
  -
    uri: /test/alphabetic
    query_params: page=7
defines 3 benchmarks:
-bm=other consists of 2 urls (/test/list and /test/alphabetic)
-bm=index will run (/test/index)
-bm=default will run all urls in "index" and "other"
The key uri: is mandatory, query_params: and new_session: are optional.

new_session: true/false specifies whether a session cookie is sent with the request (default). To make this work correctly, you need to set session_data on the instance. See file benchmarks.rb for details.

Instead of 

uri: /test/alphabetic
query_params: page=7
one could have written 
uri: /test/alphabetic?page=7

CAVEATS