JMRI: Unit testing with JUnit

JUnit is a system for building "unit tests" of software. Unit tests are small tests that make sure that individual parts of the software do what they're supposed to do. In a distributed project like JMRI, where there are lots of developers in only loose communication with each other, unit tests are a good way to make sure that the code hasn't been broken by a change.

For more information on JUnit, see the JUnit home page. A very interesting example of test-based development is available from Robert Martin's book.

Some of the classes have JUnit tests available. It's good to add JUnit tests as you make changes (test your new functionality to make sure that it is working, and keeps working), when you have to figure out what somebodys code does (the test documents exactly what should happen!), and when you track down a bug (make sure it doesn't come back).

Running the Tests

To run the existing tests, say


   ant alltest

This will compile the test code, which lives in the "test" subdirectory of the "java" directory in our usual code distributions, and then run the tests under a GUI. If you know the name of your test class, or the test class for your package, you can run that directly with the "runtest" script:


   ant tests
   ./runtest.csh jmri.jmrit.powerpanel.PowerPanelTest

The first line compiles all the test code, and the second runs a specific test or test suite.

Nightly Builds

Every night about 1:45AM Pacific time, the "nightlybuild.csh" script from a normal checkout is run. This

updates from the developer CVS
cleans out any existing compilations, etc
rebuilds from scratch with the debug target
runs the jmri.HeadLessTest app, which in turn runs the JUnit tests.

If any of these steps fail, including logging any warning or error messages, the log is emailed to the "jmri-builds" list at SourceForge. You can subscribe to this list to get the bad news as quickly as possible, or view the archives to see past logs.

(When the build succeeds, nothing is mailed, to cut down on traffic)

Writing Tests

By convention, we have a "test" class shadowing (almost) every real class. The "test" directory contains a tree of package directories parallel to the "src" tree. Each test class has the same name as the class to be tested, except with "Test" appended, and will appear in the "test" source tree. For example, the "jmri.Version" class's source code is in "src/jmri/Version.java", and it's test class is "jmri.VersionTest" found in "test/jmri/VersionTest.java".

There are additional classes which are used to group the test classes for a particular package into JUnit test suites.

Writing Additional Tests

To write additional tests for a class with existing tests, first locate the test class. (If one doesn't exist, see the section below about writing tests for a new class)

To that test class, add one or more test methods using the JUnit conventions. Basically, each method needs a name that starts with "test", e.g. "testFirst", and has to have a "public void" signature. JUnit will handle everything after that.

In general, test methods should be small, testing just one piece of the classes operation. That's why they're called "unit" tests.

Writing Tests for a New Class

(Needs info here: basically, copy some other package, and don't forget to put an entry in the enclosing package's test suite)

Writing Tests for a New Package

(Needs info here: basically, copy some other package, and don't forget to put an entry in the enclosing package's test suite)

Handling Log4J Output From Tests

JMRI uses Log4j to handle logging of various conditions, including error messages and debugging information. Tests are intended to run without error or warning output, so that it's immediately apparent from an empty standard log that they ran cleanly.

Log4j usage in the test classes themselves has two aspects:

It's perfectly OK to use log.debug(...) statements to make it easy to debug problems in test statements. log.info(...) can be used sparingly to indicate normal progress, because it's normally turned off when running the tests.
In general, log.warn or log.error should only be used when the test then goes on to trigger a JUnit assertion or exception, because the fact that an error is being logged does not show up directly in the JUnit summary of results.

On the other hand, you might want to deliberately provoke errors in the code being tested to make sure that the conditions are being handled properly. This will often produce log.error(...) or log.warn(...) messages, which must be intercepted and checked.

To allow this, JMRI runs it's using tests with a special log4j appender, which stores messages so that the JUnit tests can look at them before they are forwarded to the log. There are two aspects to making this work:

All the test classes should include common code in their setup() and teardown() code to ensure that log4j is properly initiated, and that the custom appender is told when a test is beginning and ending. // The minimal setup for log4J protected void setUp() { apps.tests.Log4JFixture.setUp(); } protected void tearDown() { apps.tests.Log4JFixture.tearDown(); }
When a test is deliberately invoking a message, it should then use the check to see that the message was created. For example, if the class under test is expected to do
log.warn("Provoked message"); the invoking test case should follow that with the line:
JUnitAppender.assertWarnMessage("Provoked message");
It will be a JUnit error if a log.warn(...) or log.error(...) message is produced that isn't matched to a JUnitAppender.assertWarnMessage(...) call.

Key Metaphors

Resetting the InstanceManager

The InstanceManager is persistant from test to test. In many cases, this is not an issue, so we don't insist that it be cleared after every test. But if you are going to reference the InstanceManager, you should clear and reset it to ensure you get reproducible results.

    jmri.InstanceManager i = new jmri.InstanceManager(){
        protected void init() {
            super.init();
            root = this;
        }
    };

This can go in the test class setup() method for convenience.