Saturday 4 May 2013

On Testing - Scheduling Your Tests


Running Test Cases

Organizing and scheduling test cases can be one of the hardest things to do in running unit or integration tests. Even using a test harness like TestNG requires care and attention to get scheduling and other substantial benefits.

Precedence or What Comes Before

When you are unit testing, you often don't have to worry about what tests come before which. Using JUnit, or other test harnesses modeled after JUnit, you don't have a way to worry about ordering. TestNG allows you to run multiple test cases with much lower overhead because you can take advantage of what came before. Even the simplest of integration tests likely requires separate stages or phases to occur.

Test Suite Design

TestNG can be run against a test directory or have a file that details the tests that are to be run. When TestNG is run against a directory, it first looks for all the Java files that start or end with "Test". From the list of files, it further inspects them to find TestNG annotations to come up with the list of methods. The test classes are run in lexical order, and the test methods within each class are run in lexical order.

The second approach uses a testng.xml file (or whatever you have decided to name it) that describes the test suite. It contains one or more test elements, within which are listed a set of test classes. The tests described in each test are run in the order the tests are entered in the test suite.

Example Test Suite File
<suite name="Full Test Suite">
  <test name="Basic Unit Tests">
    <class name="com.example.BasicUnitTestOne"/>
    <class name="com.example.BasicUnitTestTwo"/>
  </test>
  <test name="Combined Unit Tests">
    <class name="com.example.ACombinedUnitTest"/>
  </test>
  <test name="Basic Integration Tests">
    ...
  </test>
  ...
</suite>
The tests will be run in the given order. The methods within each class will be run in lexical order.

Obviously, using a test suite file gives you much more control over the order in which the tests are run. There are several reasons you may want to control the order of tests in this manner:
  • Basic unit tests should be completed before more comprehensive unit tests. If you imagine a call graph, basic unit tests will test all of the methods at the very edge of the call graph, the ones that don't call other methods. More comprehensive unit tests will test out the interior nodes, the methods that call other methods that, presumably, have already been tested.
  • Cheap tests should be run before expensive ones. Some tests may run for minutes or hours. You want those to run after the quick tests. If the quick tests fail, you don't want to run the expensive ones.
  • TestNG has the option to run tests in parallel. You can add the attribute parallel="tests" to the suite element, giving permission to run each test within a thread.

Test Class Design

TestNG has a powerful facility that allows you to classify your test cases using a group. You can use more than one group, and you can apply groups at the class level or the method level. Groups used at the class level apply to every method in the class. You can use that annotation in the following manner:
Test Group Contrived Example
    @Test(groups="mathml")
    public class MathMLBasicTest {
        @Test(groups={"basic","continuity"}
        public void testBasicMathMLContinuity() throws Exception {
            // stuff
        }
    
        @Test(groups={"basic","regular"}, 
            dependsOnGroups="continuity")
        public void testAbdelianMath() throws Exception {
            // more stuff
        }
    
        @Test(groups="advanced", 
            dependsOnGroups={"continuity","regular"})
        public void testAberrantGroups() throws Exception {
            // yet more stuff
        }
    }
These groups are collected for the entire test suite. This means you can make tests conditional on the successful running of tests for a particular feature or resource. If the tests in a group fail, any tests that depend on that group will be skipped and should show up as SKIPPED in the test report.

A less powerful sequencing control is also available that allows you to add dependsOnMethod to the Test annotation. It gives you more fine-grained control at the expense of increased maintenance of the test dependencies.

Repeating Or Ordering Test Methods

At some point you may be faced with having to run test methods in a particular order, and you may have to run test methods a number of times.

If you need to run methods in a class in a particular order, given that the lexical ordering used by default is not very edifying, the first approach to look at is to go back into the test suite file.

Sequenced Methods
<suite name="Full Test Suite">
  <test name="Sequenced Unit Tests">
    <class name="com.example.BasicUnitTestOne">
      <methods>
        <include name="zzzTestMethod"/>
        <include name="bbbTestMethod"/>
        <include name="aaaTestMethod"/>
      </methods>
    </class>
  </test>
</suite>

The methods/include elements allow you to provide fine-grained control over the ordering.

You can also provide fine-grained sequencing by introducing a proxy class. The original testing class is refactored so that the @Test and other annotations are removed. The proxy class is set up by copying the annotations onto its own methods, which are then named in a lexical ordering that reflects the sequencing that you want.

In fact, if you need to repeat a test method you will have to use a proxy class, as the test suite will only run a particular test method once. One example of a situation that requires repeated testing is when you need to know the interaction effect between methods -- however, it is a good bet that the tests need to be redesigned to separate the test methods from the underlying interacting processes.

Saturday 26 November 2011

Message Sequence Charts


I love me some message sequence charts. I have been using them off and on since I first learned about them in the early '80's. Whenever I have a timing problem, a race condition to explore, a set of interlocked state machines, or some protocols that have to work with each other, I fire up my MSC machine. In my case, the MSC machine is usually a pad of paper, or, when I need to share my thoughts with others, a white board.

Less often, I will use a graphics package, even a CASE package, to put my thoughts into the computer for more permanent documentation. Somehow, the formality of the medium changes the process. I am no longer capturing my thoughts; I am doing "documentation." Putting pen to paper is more exploratory, in my experience, than is using a text editor, word processor, or CASE system.

Either way you choose to do it is fine. The important point is that message sequence charts will bolster your communications, and help you identify problems in the system you are working with. As you explain to your colleagues where you think the problem lies, the MSC will highlight the vulnerabilities; as they explain to you where they think the problem lies, they will refer to the MSC over and over again. It its a sure fire conversation lubricant.

How Do The Sequence Charts Work?

First, you identify the players and then you identify how they communicate and when. We are often faced with a bunch of stock elements, with a familiar cast of characters. We want to see how we might treat those characters.

Let's look at a "typical", blog-style example: a web application. It will be a class sign-up sheet, where you select which class you want to sign up for, the registration is recorded, and a seat is held for you at the time of the class. Some of the characters are coming into focus: the learner signing up for the class, the web application and a database. Undoubtedly, the use cases have already popped into your head; the stick figures and boxes are also a good start for a cast of characters.



In Figure 1, I have the cast of characters, represented by the vertical lines, sending arrows at each other; each arrow represents a message. The underlying protocols don't need to be modelled, at least at this level; we may come back to them later. For our purposes the medium that carries the arrows will make its best effort to deliver the message, and will often allow for a reply. We have three characters in our cast, represented by 3 vertical lines. We have two messages plus two reply messages, represented by the angling arrows.

The Angle of the Arrow ...

The angling arrows are pretty important in the representation. Anybody familiar with MSCs knows that synchronous operations have horizontal arrows, while asynchronous operations, ones off the clock, so to speak, have angling arrows. They angle because time passes differently between the actors sending and receiving the arrows; think Einsteinian relativity: from one observer's viewpoint a pair of events on another timeline might have occurred in a different order than another observer might see.

Whenever arrows angle like that, there is always the chance of arrows crossing; when arrows cross, conflicts occur, and when arrows cross at critical points, race conditions can occur. Our example in Figure 1 shows us 4 places where the arrows are angled. Each of those places is vulnerable to conflicts and race conditions. To do a good job in designing the application, you have to identify those conflicts and engineer solutions that prevent race conditions from occurring and handle conflicts in such a way that bad things don't happen.

Looking at Figure 1 some more, we see arrow 1, the request from the Learner to the Application. How can conflicts occur here? The most basic conflict is where the Learner conflicts with himself: pressing the Submit button twice will generate two reservations. In fact, the browser will close the first connection (making arrow 4 into an orphan) and, in fact, making the entire first transaction into an orphan. We can model this situation readily, as in Figure 2.


Simple web applications, where you have a web server that invokes applications, effectively as subroutines, will see each request from the browser and direct them to different instances of the web application. Each web application that is invoked, AppA and AppB, see an individual exchange with the Learner and try to satisfy it. AppA, not knowing that it has lost its connection with the Learner, will send message 2A to the Database where the class registration is made, returning via message 3A. When AppA tries to send the registration confirmation, message 4A, to the Learner, it finally discovers the closed connection and the confirmation is dropped on the floor.

Meanwhile, AppB has gone ahead, made a reservation, prepared the confirmation and has returned it to the Learner. We now have two copies of the reservation in the database, confusing student and teacher.

In this particular case, we didn't catch the conflict at all, until the teacher reviewing the reservations noticed the duplication. The teacher's annoyance with the web application continues to grow.

From Figure 2, we can see 3 places where the conflict could have been caught and dealt with, in various shades of courtesy. In fact, it is probably best if we could deal with the issue in all three places, but any one place will do for now.

Starting at the right side, with message 2 and response 3, we can see that if we have a proper database that does database-y things it can be set up to properly reject a duplicate reservation by the same person for the same class. It will be up to AppB to recognize that a duplicate was entered, and to provide a duplicate confirmation page to the Learner.

From the left side, we can presume we require a modern browser, and then we can "require" that Javascript be enabled on the browser in order to filter the redundant Submit button operation. This is often a weak and onerous requirement: some people won't know how to enable Javascript readily and in a safe manner; other people are strict about trusting sites. If the rest of your application does not require Javascript, it may not be practical to require it for this one operation.

In the middle, we would like the application itself to recognize when redundant requests occur. This is where the application server, if you have one, comes in, by supplying credentials on the Learner's browser that determine the web application instance and makes sure to route the submission to the correct application instance. Alternatively, supplying a transaction ID as a hidden field in the browser that is submitted with the button click might be enough to recognize the duplicated transaction. AppA and AppB from Figure 2 would merge but messages 1A, 1B and 2A would still be initiated. The application is then responsible for recognizing that it got 2 submissions, perhaps with identical content, and suppressing message 2B and somehow getting response 3A to present reservation confirmation 4B.

Conclusion

There are several other places where race conditions occur in our very simple application but I will leave them as an exercise for the reader. Here are the things I would like you to take away with you:

* Race conditions are a fact of life. Good software deals with all of the race conditions, at least to the point of recognizing when they occur. Some times all you can do is make sure to fail gracefully.

* The 3 points where you can deal with the example collision are the browser (suppressing multiple clicks), the web application (in an application server environment) and the database (presuming an ACID database is available). All of these solutions have to be included for comprehensive protection: in the browser to prevent accidental multiple-click submissions, in the application to at least report when a collision has occurred, and in the database to, once and for all, catch race conditions.

Friday 25 November 2011

Five F’s of Release Management



Releasing an update to a software system requires balancing a number of factors: budget, time window, people and, of course, the expectations of your clients and community. How do you balance all of the different demands? How do you move forward?

I am assuming that you have client involvement and they help you set priorities on the things that matter to them. They have requested features and capabilities and understand where the users are coming from. The not-so-easy part is the backlog of items that are sitting there, staring at you, not blinking ...


My approach has been to select items from several different aspects of the system, much as you might plan a meal. I presume that all of your work has been itemized into tasks or issues; it doesn’t matter if you are using a trouble tracking system or a project scheduler. You have a backlog of items and need to get the mound of items reduced in size.

Foundation - The Bricks and Mortar

The foundation of your system is the pieces that none of your clients really see, the operating system, the database management system, middleware, etc. You might want to bring in a new library or use a new set of table indexes. They may be part of a planned upgrade cycle or necessary for a function to be delivered down the road. These kinds of nuts and bolts are the things that your clients will say, “Why are you bothering me about that?”

Functions and Features

Functions are the operations and mechanisms your software carries out behind the scenes. Features are the benefits the system provides, the things that customers refer to when they think about your system. Functions and features blur together, but I tend to think of features as consisting of several functions wired together to provide a specific benefit to the customer. A function can be used by several different features. A search function is the mechanism for using search criteria to return a set of results. The search feature is looking up a customer to schedule a sales call.

Facade - The Face You Put On Things

The facade is primarily the user interface of the system. It includes the color schemes, screen work flows, form design and report layouts. You can also include help text here. This aspect is often described as the cosmetics of the system.

It can also include using radio buttons versus pop-up menu, making a label have the same case on every screen that uses that label, tool tip consistency and punctuation.

Fixes - The Bugs

Bugs and issues in software systems have to be dealt with. During maintenance mode, it can be tempting to only include bug fixes. Except for high profile or urgent fixes, such releases won’t engage the interest of people using the system. If you have clients testing the system, it always helps to dangle some enhancements in front of them.

Some Other F's
There are some other F's that you can add in, but they are more variations on a theme. 

Faster: We always want the system to go faster but first we want it to go right. Once you gain a bit of breathing space, speeding up the system brings lots of benefits. I tend to roll Faster aspects into Foundation, Function or Facade, the places were perceived slowness appear most.

Free: Everybody likes free. Some issues can be dealt with "for free" because they come along as part of the package. Highlighting free stuff for your clients provides them with the sense that you are looking out for value.

Mixing the Release Items

I advocate taking a mix of items from each aspect. If you have a number of components that are visible to the client, it is also useful to include issues for those components.

I have been in development projects where there was a lot of pressure from on high to do "bug fix" only releases, and then "feature releases", or just focus on features leaving the bugs outstanding. They never seemed to feel right; someone always got left out and some of the piles kept getting bigger and bigger. 

A balanced approach seems to provide something for everyone. Everybody wins.