The MySQL Benchmark Suite

This section should contain a technical description of the MySQL benchmark suite (and crash-me), but that description is not written yet. Currently, you can get a good idea of the benchmark by looking at the code and results in the sql-bench directory in any MySQL source distribution.

This benchmark suite is meant to be a benchmark that will tell any user what operations a given SQL implementation performs well or poorly.

Note that this benchmark is single-threaded, so it measures the minimum time for the operations performed. We plan to add a lot of multi-threaded tests to the benchmark suite in the future.

The following tables show some comparative benchmark results for several database servers when accessed through ODBC on a Windows NT 4.0 machine.

Reading 2000000 rows by indexSecondsSeconds
mysql 367249
mysql_odbc 464 
db2_odbc 1206 
informix_odbc 121126 
ms-sql_odbc 1634 
oracle_odbc 20800 
solid_odbc 877 
sybase_odbc 17614 
Inserting 350768 rowsSecondsSeconds
mysql 381206
mysql_odbc 619 
db2_odbc 3460 
informix_odbc 2692 
ms-sql_odbc 4012 
oracle_odbc 11291 
solid_odbc 1801 
sybase_odbc 4802 

For the preceding tests, MySQL was run with an index cache size of 8M.

We have gathered some more benchmark results at http://www.mysql.com/information/benchmarks.html.

Note that Oracle is not included because they asked to be removed. All Oracle benchmarks have to be passed by Oracle! We believe that makes Oracle benchmarks very biased because the above benchmarks are supposed to show what a standard installation can do for a single client.

To use the benchmark suite, the following requirements must be satisified:

The benchmark suite is located in the sql-bench directory of MySQL source distributions. To run the benchmark tests, change location into that directory and execute the run-all-tests script:

shell> cd sql-bench
shell> perl run-all-tests --server=server_name

server_name is one of supported servers. You can get a list of all options and supported servers by invoking run-all-tests --help.

crash-me tries to determine what features a database supports and what its capabilities and limitations are by actually running queries. For example, it determines:

We can find the result from crash-me on a lot of different databases at http://www.mysql.com/information/crash-me.php.