Java Software Unit Testing
By James Jianbo Huang February 2002
printer-friendly versionAbstract
Unit testing is critical to the quality of object-oriented software,
because it solidifies the foundation -- the objects. This article
explains the uses of a unit testing package for Java called
JUnit, discusses the pros and cons of packages like such, and
introduces how to do unit testing with JudoScript and explain why it is the
way to go.
nit testing ensures the quality of independent logical units of the
software. In the case of Java, such logical units are classes and
sometimes the collaboration of a number of classes. Automated unit
testing is essential to regression tests.
In this article, I will explain a Java unit testing package called
JUnit, what it is and what it does. I will discuss issues
regarding its approach, draw some clear-cut conclusions, and present
unit testing in JudoScript to show that this is the way to go.
JUnit is a Java development package for writing Java code to test
other Java software. In essence, it formalizes test cases and test
suites in the form of Java classes, and provides a few assertion and
reporting facilities, so that Java classes for test cases are developed
consistently.
Since it is a testing tool, it can potentially be used by three
types of engineers:
- Experienced Java developers
- Java-savvy QA engineers
- QA engineers with no Java programming experience
For experienced Java programmers, it probably takes less than an
hour to start writing useful testing code with JUnit, because JUnit is
just another Java package, and programmers use various Java packages
day-in-day-out. If you do JUnit programming already, simply skip to
the next section.
If you never program in Java, you are not up to using JUnit. Period.
You can go to section 3 to learn how to use JudoScript for unit testing;
otherwise, you will have to learn Java first.
This section is for Java-savvy QA engineers. First off, it is a good
idea to print out the following image, the class diagram of JUnit's key
package, junit.framework
.
A class diagram depicts the relationships among classes. For instance,
in the middle of this diagram lies the most important classes,
junit.framework.TestCase
and junit.framework.TestSuite
and interface junit.framework.Test
; both these classes
implements interface Test
; TestCase
also extends class Assert
(so that it can directly
use all the methods in Assert
). Since TestSuite
only cares about Test
instances, you can include other test
suites in a (main) test suite.
To write a test case, you write a Java class that extends TestCase
,
in a couple of ways. Let me use a real test case to demonstrate. I need
to test the following class, which contains a static method that runs a
secure copy command to tranfer files between local and a remote machine
running the SSH server:
public class SSHFactory
{
public static void scp(String host,
String user,
String password,
String ciper,
String[] src,
String dest,
boolean recursive,
boolean toremote) throws Exception
{
... ... ...
}
}
The following is a run of a test case:
Listing 1. SCP Test Case 1 |
import junit.framework.*;
import com.judoscript.ext.SSHFactory;
public class MyTestCase1 extends TestCase
{
protected void runTest() {
try {
SSHFactory.scp("ssh.judoscript.com", // host
"myuser", // user name
"itspassword", // password
null,
new String[]{ "temp/file1.txt", "temp/file2.bin" },
"temp/", // destination on the remote machine
false, // not recursive
true); // to remote machine
} catch(Exception e) {
fail(e.getMessage());
}
}
public static void main(String[] args) {
try {
MyTestCase1 tcase = new MyTestCase1();
TestResult tres = tcase.run();
System.out.println("# of failures: " + tres.failureCount());
} catch(Exception e) {
e.printStackTrace();
}
}
} // end of class MyTestCase1.
|
For the test case, all you have to do is override the runTest()
method for your testing code. Method runTest()
is invoked by the
JUnit framework to conduct the testing. This test copies two files under "temp/"
in the current directory to the remote server. If it does not throw exceptions,
it is considered successful. This criterion is rather loose, because even if
it did not throw exception, we still don't know whether the files are copied
correctly or not. One way to remedy this is to issue another scp()
that copies these files back from the remote machine; if the identical files
are retrieved, then we are confident that the file copies were indeed good.
As I mentioned earlier, there are two ways to write test cases, so let us use
the second approach.
Listing 2. SCP Test Case 2 |
import junit.framework.*;
import com.judoscript.ext.SSHFactory;
import java.io.*;
public class MyTestCase2 extends TestCase
{
public MyTestCase2(String method) { super(method); }
public void testSCP_NonRecursive() {
try {
// copy to remote machine
SSHFactory.scp("ssh.judoscript.com", // host
"myuser", // user name
"itspassword", // password
null,
new String[]{ "temp/file1.txt", "temp/file2.bin" },
"temp/", // dest on the remote machine
false, // not recursive
true); // to remote machine
// copy from remote machine
SSHFactory.scp("ssh.judoscript.com", // host
"myuser", // user name
"itspassword", // password
null,
new String[] { "temp/back/" }, // local dest
"temp/file1.txt", // source on the remote machine
false, // not recursive
false); // from remote machine
SSHFactory.scp("ssh.judoscript.com", // host
"myuser", // user name
"itspassword", // password
null,
new String[] { "temp/back/" }, // local dest
"temp/file2.bin", // source on the remote machine
false, // not recursive
false); // from remote machine
// compare the files -- just compare their sizes
long f1_len = new File("temp/file1.txt").length();
long f1b_len = new File("temp/back/file1.txt").length();
long f2_len = new File("temp/file2.bin").length();
long f2b_len = new File("temp/back/file2.bin").length();
if ((f1_len != f1b_len) || (f2_len != f2b_len))
fail("SCP file copy failed.");
} catch(Exception e) {
fail(e.getMessage());
}
}
public static void main(String[] args) {
try {
MyTestCase2 tcase = new MyTestCase2("testSCP_NonRecursive");
TestResult tres = tcase.run();
System.out.println("# of failures: " + tres.failureCount());
} catch(Exception e) {
e.printStackTrace();
}
}
} // end of class MyTestCase2.
|
In this case, we implemented a method called "testSCP_NonRecursive";
then, in main()
, we passed this name to the constructor when
creating a test case object tcase
. This method has to be public.
Why we need this approach?
In real world testing, many test cases require certain settings. For
instance, if the remote host is not running SSH when we run the test case,
it will fail but that is not a bug; it is a user error. Or the test is run
on a different PC that does not contain "temp/file1.txt" -- another user
error. In JUnit, you can set up and clean up these "test fixture" by
overriding methods setUp()
and tearDown()
, which
are called before and after the test by JUnit framework. Many times, these
two methods actually take a lot more code than the test cases themselves.
For example, in order to make the test case self-contained, we should prepare
the files to be copied in setUp()
, and remove them in
tearDown()
. In order to reuse setUp() and tearDown(), JUnit
adopts the name-based test case methods as we saw in the second exmaple.
This way, we can implement another test case called "testSCP_Recursive" in
the same class to use the same set-up/tear-down methods.
To automate testing, JUnit provides class TestSuite
. You
can add multiple test cases and even other test suites into a test suite,
and then run them altogether.
import junit.framework.*;
import com.judoscript.ext.SSHFactory;
import java.io.*;
public class MyTestCase3 extends TestCase
{
public MyTestCase3(String method) { super(method); }
public void testSCP_NonRecursive() {
... ... ...
}
public void testSCP_Recursive() {
... ... ...
}
protected void setUp() {
// set up the test directories and files on the local machine
... ... ...
}
protected void tearDown() {
// remove the test directories and files on the local machine
... ... ...
}
public static void main(String[] args) {
try {
TestSuite suite = new TestSuite();
suite.addText(new MyTestCase3("testSCP_NonRecursive"));
suite.addText(new MyTestCase3("testSCP_Recursive"));
TestResult tres = suite.run();
System.out.println("# of failures: " + tres.failureCount());
} catch(Exception e) {
e.printStackTrace();
}
}
} // end of class MyTestCase3.
This code is straightforward. The class itself implements two test
cases. In main()
, we created a test suite object and two
test case objects, and added both to the suite. A single run of the
suite does the job.
JUnit is a simple idea and simple framework. It tries to standardize
the way people write test cases and suites for Java, in Java. In fact,
in the article JUnit A Cook's Tour, it is clearly stated,
"the number one goal is to write a framework within which we have
some glimmer of hope that developers will actually write tests."
Assuming developers are only
willing to write Java code, JUnit cuts their excuses not writing any
tests. The question is, is this a blessing for the QA team?
»»» Top «««
In a team environment, tests are run and maintained by QA engineers,
not developers. Writing tests in Java has at least the following three
problems. The root of the problems is, Java is a system language,
designed for building robust software, not for doing things
directly and quickly.
- Test cases in Java will take a lot of coding. Java has these
characteristics:
- It requires a lot of house-keeping code -- code for class
definition, method definition, exception handling ... that is not
directly useful for the tasks at hand.
- Java APIs are detailed and low level, providing maximum flexibility.
To do anything useful, however, you need many lines of code. Think
of the SCP test case above, and try to create a directory tree with
a few files in it for the recursive copy test.
The consequences are:
- Setting up necessary test environment may take a lot of coding and
dependencies.
- The more code, the more bugs. When the size of test case code
approaches or even surpasses that of the testees, the test cases
need be tested first before running them.
- Developers will write a lot of code that will not be used by any
customers. How many developers will whole-heartedly take this job?
You can expect a lot more Q&D (quick-and-dirty) code than in
production software.
- QA engineers are probably not up to the task of writing or even
maintaining test cases. Even if they are capable, they will not be
able to modify the test cases as fast, wasting precious testing time.
- Java code must be compiled into class files before run. For a production
software or application, this is ok because the class files will be
packaged and deployed. In case of testing, the test environment may
change all the time (due to limited test machines and other resources),
modification to parameters is commonplace, the compilation adds an
unnecessary step and hurts productivity.
- Java software are built with classes, interfaces and inheritance;
for a well designed software, it can have elegant architecture. But
for busy QA personel, how many people really have the time and
capacity to handle the complexity? Who maintain the test case code?
Therefore, Java is a wrong language for general testing; designing
a formal framework does not help. It will
- cost developers' time,
- add complexity to test case maintenance,
- slow down parameter modifications, and
- add more time to analyze failed test cases.
If some developer decides to use JUnit for his/her own testing, it is
perfectly fine. For a team with developers and QA testers, even if all the
QA testers are fully Java potent, this approach still slows down the pace
and productivity and should never be adopted.
What is the solution then? Scripting! Scripting languages are
designed to do many things easily, quickly and effectively; they are
perfect for testing. Since they are also general-purpose programming
langauges, they are up to almost anything. In fact, conceptualizing
tests as just test cases and test suites may not be enough, because
it limits the ways test cases be organized. For instance, I may want
to connect to a test database, create a few test tables with data,
then test a series of test cases before I drop all the test tables
and disconnect from the database server. Such tasks are better left
to the QA testers who do the testing.
Testing is a step-by-step process of execution and analysis. There
is no shortcut, complicated architectures bring in little value.
Powerful scripting languages are the tool for testing.
»»» Top «««
For Java software testing, JudoScript is a perfect match for these reasons:
- JudoScript is a complete, general-purpose scripting language. It
emphazises on doing things quickly and intuitively. At the same
time, it is capable of anything that Java is capable of.
- JudoScript is designed ground-up for Java platforms. You can use any Java
objects seamlessly in the scripts. For this reason, you can extend
JudoScript by simply creating regular Java classes.
- JudoScript programs are run without compiling, they are easy to modify
and maintain.
- JudoScript's rich, ready-to-use features can be used for testing itself,
and are invaluable for test environment setting up and cleaning up.
- JudoScript's feature list is virtually unlimited because it can use any
Java software packages. You can use free Java software such as
jCVS, java DNS, ... or commercial Java software, or Java software
developed in house.
- JudoScript is great not only for automated unit testing, but also for
functional testing, web load testing, database application testing,
XML document verification, and more.
- JudoScript can be used in integrated build environment such as
Ant, because it supports
BSF (Bean-Scripting-Framework), a standard interface for Java
software to interact with scripting langauges.
- JudoScript has complete documentation, sample code and topical articles,
on-line or hard copy. Documentation is such an important part
of this language, more and more information is coming.
- Because JudoScript is a general-purpose programming language, it can
be used beyond testing, so knowledge about JudoScript is reusable in
other situations. Many propriatary testing languages only work in
their own test environment. JudoScript is also an easier way to learn
Java, thanks to its smooth interaction between the two.
- Last but not least, JudoScript is totally free. How much are you paying now?
As we mentioned earilier, there are many different set-ups for testing.
The following following scheme have multiple test cases share a same pair
of set-up/clean-up routines:
test environment verify and setup; // make sure the test database server
// is live and make a connection.
run test case 1;
run test case 2;
run test case 3;
test environment cleanup; // disconnect from the database
This is best modeled in JudoScript using functions:
function test_Foo_setup() {
connect 'jdbc:oracle:thin@localhost:1521:mydb', 'orauser', 'orapass';
}
function test_Foo_cleanup() {
disconnect();
}
function test_Foo_1() { ... }
function test_Foo_2() { ... }
function test_Foo_3() { ... }
// run the test:
test_Foo_setup();
test_Foo_1();
test_Foo_2();
test_Foo_3();
test_Foo_cleanup();
This is another scenario:
test environment verify and setup;
test set up A;
run test case 1;
test clean up A;
test set up A;
run test case 2;
test clean up A;
test set up B;
run test case 3;
test clean up A;
run test case 4;
test environment cleanup;
In addition to the global setup/cleanup, many test cases have their own
environment; some of the test cases share a same pair of setup/cleanup
routines. In this case, the inheritance feature of object-oriented
programming helps to orginize code nice and clean:
function test_Foo_setup() { ... }
function test_Foo_cleanup() { ... }
class Test_Base
{
function setup() {} // empty
function cleanup() {} // empty
function doTest() {} // empty -- should be overridden
function runTest() {
setup();
doTest();
cleanup();
}
}
class Test_Foo_Group_A extends Test_Base
{
function setup() { /* group A setup */ }
function cleanup() { /* group A cleanup */ }
function doTest() {} // empty -- should be overridden
}
class Test_Foo_Group_B extends Test_Base
{
function setup() { /* group B setup */ }
function cleanup() { /* group B cleanup */ }
function doTest() {} // empty -- should be overridden
class Test_Foo_1 extends Test_Foo_Group_A
{
construtor { desc = "Project Foo's test case 1."; }
function doTest { /* test case 1 code */ }
}
class Test_Foo_2 extends Test_Foo_Group_A
{
construtor { desc = "Project Foo's test case 2."; }
function doTest { /* test case 2 code */ }
}
class Test_Foo_3 extends Test_Foo_Group_B
{
construtor { desc = "Project Foo's test case 3."; }
function doTest { /* test case 3 code */ }
}
class Test_Foo_4 extends Test_Base
{
construtor { desc = "Project Foo's test case 4."; }
function doTest { /* test case 4 code */ }
}
// Run the tests
test_Foo_setup();
(new Test_Foo_1).runTest();
(new Test_Foo_2).runTest();
(new Test_Foo_3).runTest();
(new Test_Foo_4).runTest();
test_Foo_cleanup();
The class hierarchy is like this:
Test_Base
Test_Foo_Group_A
Test_Foo_1
Test_Foo_2
Test_Foo_Group_B
Test_Foo_3
Test_Foo_4
In effect, the test cases are grouped; each group shares a pair of
setup()
and cleanup()
methods. You get a
nicely laid out paradigm. The cost is house-keeping code and complexity.
A by-product is, you can bundle more information (such as descriptions)
with the test cases.
The second scenario can be implement with functions as well. The
point is, JudoScript is flexible enough for you to lay out tests in the most
appropriate way for the situation. JudoScript language has built-in assertion
and exception handling facility. It also has standard, error and log
output streams. If you have some special reporting needs, you can
easily write your own report functions, perhaps using the the flexible
print
and println
statements in JudoScript. JudoScript
scripts can include other scripts; and external programs can be run
from another script. There are many ways you can organize your test
suites.
To truly understand JudoScript language and feel its power, there is no other
way than reading featured articles such as Introduction
to JudoScript, and try it out by yourself. JudoScript is an easy-to-use and
easy-to-learn language; even if you have never programmed in Java, you
won't have any problem start writing useful JudoScript code.
»»» Top «««
- SCP Test Case 1
- SCP Test Case 2