Skip to main content

Info on JMRI:
Development tools
Structure
Techniques and Standards
How To
Functional Info
Background Info

JMRI: Unit testing with JUnit

JUnit is a system for building "unit tests" of software. Unit tests are small tests that make sure that individual parts of the software do what they're supposed to do. In a distributed project like JMRI, where there are lots of developers in only loose communication with each other, unit tests are a good way to make sure that the code hasn't been broken by a change.

For more information on JUnit, see the JUnit home page. A very interesting example of test-based development is available from Robert Martin's book.

Some of the classes have JUnit tests available. It's good to add JUnit tests as you make changes (test your new functionality to make sure that it is working, and keeps working), when you have to figure out what somebodys code does (the test documents exactly what should happen!), and when you track down a bug (make sure it doesn't come back).

Running the Tests

To run the existing tests, say 
   ant alltest
This will compile the test code, which lives in the "test" subdirectory of the "java" directory in our usual code distributions, and then run the tests under a GUI. (To make sure you've recompiled everything, you may want to do ant clean first) If you know the name of your test class, or the test class for your package, you can run that directly with the "runtest" script: 
   ant tests
   ./runtest.csh jmri.jmrit.powerpanel.PowerPanelTest
The first line compiles all the test code, and the second runs a specific test or test suite.

Nightly Builds

 Every night about 1:45AM Pacific time (and sometimes more often), the "nightlybuild.csh" script from a normal checkout is run. This
  1. updates from the developer CVS
  2. cleans out any existing compilations, etc
  3. rebuilds from scratch with the debug target
  4. runs the jmri.HeadLessTest app, which in turn runs the JUnit tests.

If any of these steps fail, including logging any warning or error messages, the log is emailed to the "jmri-builds" list at SourceForge. You can subscribe to this list to get the bad news as quickly as possible, or view the archives to see past logs.

(When the build succeeds, nothing is mailed, to cut down on traffic)

Writing Tests

By convention, we have a "test" class shadowing (almost) every real class. The "test" directory contains a tree of package directories parallel to the "src" tree. Each test class has the same name as the class to be tested, except with "Test" appended, and will appear in the "test" source tree. For example, the "jmri.Version" class's source code is in "src/jmri/Version.java", and it's test class is "jmri.VersionTest" found in "test/jmri/VersionTest.java".

There are additional classes which are used to group the test classes for a particular package into JUnit test suites.

Writing Additional Tests for an Existing Class

To write additional tests for a class with existing tests, first locate the test class. (If one doesn't exist, see the section below about writing tests for a new class)

To that test class, add one or more test methods using the JUnit conventions. Basically, each method needs a name that starts with "test", e.g. "testFirst", and has to have a "public void" signature. JUnit will handle everything after that.

In general, test methods should be small, testing just one piece of the classes operation. That's why they're called "unit" tests.

Writing Tests for a New Class

(Needs info here: basically, copy some other package, and don't forget to put an entry in the enclosing package's test suite)

Writing Tests for a New Package

(Needs info here: basically, copy some other package, and don't forget to put an entry in the enclosing package's test suite)

Key Metaphors

Handling Log4J Output From Tests

JMRI uses Log4j to handle logging of various conditions, including error messages and debugging information. Tests are intended to run without error or warning output, so that it's immediately apparent from an empty standard log that they ran cleanly.

Log4j usage in the test classes themselves has two aspects:

  1. It's perfectly OK to use log.debug(...) statements to make it easy to debug problems in test statements. log.info(...) can be used sparingly to indicate normal progress, because it's normally turned off when running the tests.
  2. In general, log.warn or log.error should only be used when the test then goes on to trigger a JUnit assertion or exception, because the fact that an error is being logged does not show up directly in the JUnit summary of results.

On the other hand, you might want to deliberately provoke errors in the code being tested to make sure that the conditions are being handled properly. This will often produce log.error(...) or log.warn(...) messages, which must be intercepted and checked.

To allow this, JMRI runs it's using tests with a special log4j appender, which stores messages so that the JUnit tests can look at them before they are forwarded to the log. There are two aspects to making this work:

  1. All the test classes should include common code in their setup() and teardown() code to ensure that log4j is properly initiated, and that the custom appender is told when a test is beginning and ending. 
        // The minimal setup for log4J
    protected void setUp() throws Exception { 
        apps.tests.Log4JFixture.setUp(); 
        super.setUp();
    }
    protected void tearDown() throws Exception { 
        super.tearDown();
        apps.tests.Log4JFixture.tearDown(); 
    }
  2. When a test is deliberately invoking a message, it should then use the check to see that the message was created. For example, if the class under test is expected to do 
        log.warn("Provoked message");
    the invoking test case should follow that with the line: 
        JUnitAppender.assertWarnMessage("Provoked message");

    It will be a JUnit error if a log.warn(...) or log.error(...) message is produced that isn't matched to a JUnitAppender.assertWarnMessage(...) call.

Resetting the InstanceManager

If you are testing code that is going to reference the InstanceManager, you should clear and reset it to ensure you get reproducible results.

Your setUp() implementation should start with: 

    super.setUp();
    jmri.util.JUnitUtil.resetInstanceManager();
    jmri.util.JUnitUtil.initInternalTurnoutManager();
    jmri.util.JUnitUtil.initInternalLightManager();
    jmri.util.JUnitUtil.initInternalSensorManager();
(You can omit the initialization managers you don't need)

Your tearDown() should end with: 

    jmri.util.JUnitUtil.resetInstanceManager();
    super.tearDown();

Running Listeners

JMRI is a multi-threaded application. Listeners for JMRI objects are notified on various threads. Sometimes you have to wait for that to take place. To do that, after you invoke a something that will notifylisteners, but before you check the results, do 
    JUnitUtil.releaseThread(this);
This uses a nominal delay, suitable for almost all uses. If you're doing a lot of notifications, or some complex calculation in the listeners, you can instead do 
    JUnitUtil.releaseThread(this, 200);
where the argument is the number of extra milliseconds to let other threads work. Don't make this longer than really needed, though, because you colleagues will have to wait that long every time they run the tests.

Note that this should not be used to synchronize with Swing threads. See the next section for that.

Testing Swing Code

AWT and Swing code runs on a separate thread from JUnit tests. Once a Swing or AWT object has been displayed (via show() or setVisible(true)), it cannot be reliably accessed from the JUnit thread. Even using the listener delay technique described above isn't reliable.

For the simplest possible test, displaying a window for manual interaction, it's OK to create and invoke a Swing object from a JUnit test. Just don't try to interact with it once it's been displayed!

Because we run tests in "headless" mode during the continuous integration builds, it's important that Swing (and AWT) access in tests be enclosed within a mode check: 


        if (!System.getProperty("jmri.headlesstest","false").equals("true")) {
            suite.addTest(myTest.suite());
        }
This will run the myTest suite of tests only when a display is available.

For many tests, you'll both make testing reliable and improve the structure of your code by separating the GUI (Swing) code from the JMRI logic and communications. This lets you check the logic code separately, but invoking those methods and checking the state them update.

For more complicated GUI testing, we're starting to use JFCUnit.

For a very simple example of the use of JFCUnit, see the test/jmri/util/SwingTestCaseTest.java file.

To use JFCUnit, you first inherit your class From SwingTestCase instead of TestCase. This is enough to get basic operation of Swing tests; the base class pauses the test thread until Swing (actually, the AWT event mechanism) has completed all processing after every Swing call in the test. (For this reason, the tests will run much slower if you're e.g. moving the mouse cursor around while they're running)

For more complex GUI testing, you can invoke various aspects of the interface and check internal state using test code.

Issues

JUnit uses a custom classloader, which can cause problems finding singletons and starting Swing. If you get the error about not being able to find or load a class, suspect that adding the missing class to the test/junit/runner/excluded.properties file would fix it.

As a test you can try setting the "-noloading" option in the main of whichever test class you're having trouble with: 

static public void main(String[] args) {
    String[] testCaseName = {"-noloading", LogixTableActionTest.class.getName()};
    junit.swingui.TestRunner.main(testCaseName);
}

The right long-term fix is to have all classes with loader issues included in the test/junit/runner/excluded.properties file. JUnit uses those properties to decide how to handle loading and reloading of classes.