On Testing - Scheduling Your Tests

Running Test Cases

Organizing and scheduling test cases can be one of the hardest things to do in running unit or integration tests. Even using a test harness like TestNG requires care and attention to get scheduling and other substantial benefits.

Precedence or What Comes Before

When you are unit testing, you often don't have to worry about what tests come before which. Using JUnit, or other test harnesses modeled after JUnit, you don't have a way to worry about ordering. TestNG allows you to run multiple test cases with much lower overhead because you can take advantage of what came before. Even the simplest of integration tests likely requires separate stages or phases to occur.

Test Suite Design

TestNG can be run against a test directory or have a file that details the tests that are to be run. When TestNG is run against a directory, it first looks for all the Java files that start or end with "Test". From the list of files, it further inspects them to find TestNG annotations to come up with the list of methods. The test classes are run in lexical order, and the test methods within each class are run in lexical order.

The second approach uses a testng.xml file (or whatever you have decided to name it) that describes the test suite. It contains one or more test elements, within which are listed a set of test classes. The tests described in each test are run in the order the tests are entered in the test suite.

Example Test Suite File

<suite name="Full Test Suite">
  <test name="Basic Unit Tests">
    <class name="com.example.BasicUnitTestOne"/>
    <class name="com.example.BasicUnitTestTwo"/>
  </test>
  <test name="Combined Unit Tests">
    <class name="com.example.ACombinedUnitTest"/>
  </test>
  <test name="Basic Integration Tests">
    ...
  </test>
  ...
</suite>

The tests will be run in the given order. The methods within each class will be run in lexical order.

Obviously, using a test suite file gives you much more control over the order in which the tests are run. There are several reasons you may want to control the order of tests in this manner:

• Basic unit tests should be completed before more comprehensive unit tests. If you imagine a call graph, basic unit tests will test all of the methods at the very edge of the call graph, the ones that don't call other methods. More comprehensive unit tests will test out the interior nodes, the methods that call other methods that, presumably, have already been tested.
• Cheap tests should be run before expensive ones. Some tests may run for minutes or hours. You want those to run after the quick tests. If the quick tests fail, you don't want to run the expensive ones.
• TestNG has the option to run tests in parallel. You can add the attribute parallel="tests" to the suite element, giving permission to run each test within a thread.

Test Class Design

TestNG has a powerful facility that allows you to classify your test cases using a group. You can use more than one group, and you can apply groups at the class level or the method level. Groups used at the class level apply to every method in the class. You can use that annotation in the following manner:

Test Group Contrived Example

    @Test(groups="mathml")
    public class MathMLBasicTest {
        @Test(groups={"basic","continuity"}
        public void testBasicMathMLContinuity() throws Exception {
            // stuff
        }
    
        @Test(groups={"basic","regular"}, 
            dependsOnGroups="continuity")
        public void testAbdelianMath() throws Exception {
            // more stuff
        }
    
        @Test(groups="advanced", 
            dependsOnGroups={"continuity","regular"})
        public void testAberrantGroups() throws Exception {
            // yet more stuff
        }
    }

These groups are collected for the entire test suite. This means you can make tests conditional on the successful running of tests for a particular feature or resource. If the tests in a group fail, any tests that depend on that group will be skipped and should show up as SKIPPED in the test report.

A less powerful sequencing control is also available that allows you to add dependsOnMethod to the Test annotation. It gives you more fine-grained control at the expense of increased maintenance of the test dependencies.

Repeating Or Ordering Test Methods

At some point you may be faced with having to run test methods in a particular order, and you may have to run test methods a number of times.

If you need to run methods in a class in a particular order, given that the lexical ordering used by default is not very edifying, the first approach to look at is to go back into the test suite file.

Sequenced Methods

<suite name="Full Test Suite">
  <test name="Sequenced Unit Tests">
    <class name="com.example.BasicUnitTestOne">
      <methods>
        <include name="zzzTestMethod"/>
        <include name="bbbTestMethod"/>
        <include name="aaaTestMethod"/>
      </methods>
    </class>
  </test>
</suite>

The methods/include elements allow you to provide fine-grained control over the ordering.

You can also provide fine-grained sequencing by introducing a proxy class. The original testing class is refactored so that the @Test and other annotations are removed. The proxy class is set up by copying the annotations onto its own methods, which are then named in a lexical ordering that reflects the sequencing that you want.

In fact, if you need to repeat a test method you will have to use a proxy class, as the test suite will only run a particular test method once. One example of a situation that requires repeated testing is when you need to know the interaction effect between methods -- however, it is a good bet that the tests need to be redesigned to separate the test methods from the underlying interacting processes.