This section should contain a technical description of the MySQL benchmark suite (and crash-me), but that description is not written yet. Currently, you can get a good idea of the benchmark by looking at the code and results in the sql-bench directory in any MySQL source distribution.
This benchmark suite is meant to be a benchmark that will tell any user what operations a given SQL implementation performs well or poorly.
Note that this benchmark is single-threaded, so it measures the minimum time for the operations performed. We plan to add a lot of multi-threaded tests to the benchmark suite in the future.
The following tables show some comparative benchmark results for several database servers when accessed through ODBC on a Windows NT 4.0 machine.
Reading 2000000 rows by index | Seconds | Seconds |
mysql | 367 | 249 |
mysql_odbc | 464 | |
db2_odbc | 1206 | |
informix_odbc | 121126 | |
ms-sql_odbc | 1634 | |
oracle_odbc | 20800 | |
solid_odbc | 877 | |
sybase_odbc | 17614 |
Inserting 350768 rows | Seconds | Seconds |
mysql | 381 | 206 |
mysql_odbc | 619 | |
db2_odbc | 3460 | |
informix_odbc | 2692 | |
ms-sql_odbc | 4012 | |
oracle_odbc | 11291 | |
solid_odbc | 1801 | |
sybase_odbc | 4802 |
For the preceding tests, MySQL was run with an index cache size of 8M.
We have gathered some more benchmark results at http://www.mysql.com/information/benchmarks.html.
Note that Oracle is not included because they asked to be removed. All Oracle benchmarks have to be passed by Oracle! We believe that makes Oracle benchmarks very biased because the above benchmarks are supposed to show what a standard installation can do for a single client.
To use the benchmark suite, the following requirements must be satisified:
The benchmark suite is provided with MySQL source distributions, so you must have a source distribution. You can either download a released distribution from http://www.mysql.com/downloads/, or use the current development source tree (see Installing source tree).
The benchmark scripts are written in Perl and use the Perl DBI module to access database servers, so DBI must be installed. You will also need the server-specific DBD drivers for each of the servers you want to test. For example, to test MySQL, PostgreSQL, and DB2, the DBD::mysql, DBD::Pg, and DBD::DB2 modules must be installed.
The benchmark suite is located in the sql-bench directory of MySQL source distributions. To run the benchmark tests, change location into that directory and execute the run-all-tests script:
shell> cd sql-bench shell> perl run-all-tests --server=server_name
server_name is one of supported servers. You can get a list of all options and supported servers by invoking run-all-tests --help.
crash-me tries to determine what features a database supports and what its capabilities and limitations are by actually running queries. For example, it determines:
What column types are supported
How many indexes are supported
What functions are supported
How big a query can be
How big a VARCHAR column can be
We can find the result from crash-me on a lot of different databases at http://www.mysql.com/information/crash-me.php.