MySQL Test Suite

Until recently, our main full-coverage test suite was based on proprietary customer data and for that reason has not been publicly available. The only publicly available part of our testing process consisted of the crash-me test, a Perl DBI/DBD benchmark found in the sql-bench directory, and miscellaneous tests located in tests directory. The lack of a standardized publicly available test suite has made it difficult for our users, as well developers, to do regression tests on the MySQL code. To address this problem, we have created a new test system that is included in Unix source distributions and binary distributions starting with Version 3.23.29. The tests can be run under Unix, or on Windows in the Cygwin environment if the server has been compiled under Cygwin. They cannot be run in a native Windows environment currently.

The current set of test cases doesn't test everything in MySQL, but it should catch most obvious bugs in the SQL processing code, OS/library issues, and is quite thorough in testing replication. Our eventual goal is to have the tests cover 100% of the code. We welcome contributions to our test suite. You may especially want to contribute tests that examine the functionality critical to your system, as this will ensure that all future MySQL releases will work well with your applications.

Running the MySQL Test Suite

The test system consist of a test language interpreter (mysqltest), a shell script to run all tests(mysql-test-run), the actual test cases written in a special test language, and their expected results. To run the test suite on your system after a build, type make test or mysql-test/mysql-test-run from the source root. If you have installed a binary distribution, cd to the install root (eg. /usr/local/mysql), and do scripts/mysql-test-run. All tests should succeed. If not, you should try to find out why and report the problem if this is a bug in MySQL. See Reporting mysqltest bugs.

If you have a copy of mysqld running on the machine where you want to run the test suite you do not have to stop it, as long as it is not using ports 9306 and 9307. If one of those ports is taken, you should edit mysql-test-run and change the values of the master and/or slave port to one that is available.

You can run one individual test case with mysql-test/mysql-test-run test_name.

If one test fails, you should test running mysql-test-run with the --force option to check if any other tests fails.

Extending the MySQL Test Suite

You can use the mysqltest language to write your own test cases. Unfortunately, we have not yet written full documentation for it. You can, however, look at our current test cases and use them as an example. The following points should help you get started:

  • The tests are located in mysql-test/t/*.test

  • A test case consists of ; terminated statements and is similar to the input of mysql command-line client. A statement by default is a query to be sent to MySQL server, unless it is recognized as internal command (eg. sleep).

  • All queries that produce results---for example, SELECT, SHOW, EXPLAIN, etc., must be preceded with @/path/to/result/file. The file must contain the expected results. An easy way to generate the result file is to run mysqltest -r < t/test-case-name.test from mysql-test directory, and then edit the generated result files, if needed, to adjust them to the expected output. In that case, be very careful about not adding or deleting any invisible characters - make sure to only change the text and/or delete lines. If you have to insert a line, make sure the fields are separated with a hard tab, and there is a hard tab at the end. You may want to use od -c to make sure your text editor has not messed anything up during edit. We hope that you will never have to edit the output of mysqltest -r as you only have to do it when you find a bug.

  • To be consistent with our setup, you should put your result files in mysql-test/r directory and name them test_name.result. If the test produces more than one result, you should use test_name.a.result, test_name.b.result, etc.

  • If a statement returns an error, you should on the line before the statement specify with the --error error-number. The error number can be a list of possible error numbers separated with ','.

  • If you are writing a replication test case, you should on the first line of the test file, put source include/master-slave.inc;. To switch between master and slave, use connection master; and connection slave;. If you need to do something on an alternate connection, you can do connection master1; for the master, and connection slave1; for the slave.

  • If you need to do something in a loop, you can use something like this:

       let $1=1000;
       while ($1)
       {
        # do your queries here
        dec $1;
       }
       

  • To sleep between queries, use the sleep command. It supports fractions of a second, so you can do sleep 1.3;, for example, to sleep 1.3 seconds.

  • To run the slave with additional options for your test case, put them in the command-line format in mysql-test/t/test_name-slave.opt. For the master, put them in mysql-test/t/test_name-master.opt.

  • If you have a question about the test suite, or have a test case to contribute, send an email message to the MySQL internals mailing list. See Mailing-list. As this list does not accept attachments, you should ftp all the relevant files to: ftp://support.mysql.com/pub/mysql/Incoming/

Reporting Bugs in the MySQL Test Suite

If your MySQL version doesn't pass the test suite you should do the following:

  • Don't send a bug report before you have found out as much as possible of what when wrong! When you do it, please use the mysqlbug script so that we can get information about your system and MySQL version. See Bug reports.

  • Make sure to include the output of mysql-test-run, as well as contents of all .reject files in mysql-test/r directory.

  • If a test in the test suite fails, check if the test fails also when run by its own:

      cd mysql-test
      mysql-test-run --local test-name
      

    If this fails, then you should configure MySQL with --with-debug and run mysql-test-run with the --debug option. If this also fails send the trace file var/tmp/master.trace to ftp://support.mysql.com/pub/mysql/secret so that we can examine it. Please remember to also include a full description of your system, the version of the mysqld binary and how you compiled it.

  • Try also to run mysql-test-run with the --force option to see if there is any other test that fails.

  • If you have compiled MySQL yourself, check our manual for how to compile MySQL on your platform or, preferable, use one of the binaries we have compiled for you at http://www.mysql.com/downloads/. All our standard binaries should pass the test suite !

  • If you get an error, like Result length mismatch or Result content mismatch it means that the output of the test didn't match exactly the expected output. This could be a bug in MySQL or that your mysqld version produces slight different results under some circumstances.

    Failed test results are put in a file with the same base name as the result file with the .reject extension. If your test case is failing, you should do a diff on the two files. If you cannot see how they are different, examine both with od -c and also check their lengths.

  • If a test fails totally, you should check the logs file in the mysql-test/var/log directory for hints of what went wrong.

  • If you have compiled MySQL with debugging you can try to debug this by running mysql-test-run with the --gdb and/or --debug options. See Making trace files.

    If you have not compiled MySQL for debugging you should probably do that. Just specify the --with-debug options to configure! See Installing source.