A custom JUnit Runner

Sometimes, JUnit does not exactly fit the bill. In a lot of cases, I can work around it by extending JUnit using the rules feature or creatively implementing a parameterized unit test. However, in the last months I had several occasions, where the result of this approach was clumsy or the problem not solvable at all. This resulted in some things not being as well tested as I would have liked them to be. So I set out to implement custom JUnit runners. And it turned out to be simple!

Extending or customizing JUnit with a custom(ized) runner is relatively straight forward. JUnit will choose a custom runner whenever your test is annotated with the @RunWith annotation. The argument of the annotation specifies the desired runner. This runner can be any class extending Runner and providing a constructor that takes Class<?> descriptor  of the annotated test as an argument.
Consider that you whish to execute your tests for a specific set of different locales. Your system allows manipulating the locale to be used via a global class LocaleHolder:

class LocaleHolder {
    private static Locale locale;

    public static Locale set(Locale locale) {
        Locale old = LocaleHolder.locale;
        LocaleHolder.locale = locale;
        return old;
    }
    public static Locale get() {
        return locale;
    }
}

Now you want to implement your tests just once and have them executed for each possible Locale:

import com.example.junit.Locales;
@RunWith(Locales.class)
public class LocalesTest{
    @Test
    public void xyzIsCool(){
        System.out.println("Locale in use: " + LocaleHolder.get());
    }
}

Please note, that in the real word, you can achieve the same by simply using a parameterized JUnit test, but the task has a perfect scope for an example.
In the first step, we have to implement the custom “Locales” runner. As this runner will execute the same tests multiple times and therefore will be a node in the test runners gui, it should inherit from ParentRunner. Suite, which extends ParentRunner, is even better suited, as it implements a complete runner that can be used out of the box:

public class Locales extends Suite {
    private static final Iterable<Locale> localesToUse =
            asList(Locale.FRENCH, Locale.GERMAN, Locale.ENGLISH);
   
     public Locales(Class<?> klass) throws InitializationError {
        super(klass, extracAndCreateRunners(klass));
    }

    private static List<Runner>
            extracAndCreateRunners(Class<?> klass) throws InitializationError {
        List<Runner> runners = new ArrayList<Runner>();
        for(Locale locale : localesToUse){
            runners.add(new LocalesRunner(locale, klass));
        }
        return runners;
    }
}

Basically, the LocalesRunner itself should set the correct locale before each test and execute the test in the same fashion as JUnit would do. The simplest way to get new functionality of the ground, is to derive from BlockJUnit4ClassRunner, which is the default test runner in JUnit . The best way to extend or augment standard JUnit behavior programmatically, is to inject your functionality in the Statement chain that the base class runner builds. For more information about extending statements, refer to the various JUnit rules examples, as they use this mechanism, too.

class LocalesRunner extends BlockJUnit4ClassRunner { 
    private final Locale locale;
    LocalesRunner(Locale locale, Class<?> klass) throws InitializationError { 
        super(klass); 
        this.locale = locale; 
    }
    @Override 
    protected Statement methodBlock(final FrameworkMethod method) { 
        return new Statement() { 
            @Override 
            public void evaluate() throws Throwable { 
                Locale oldLocale = LocaleHolder.set(locale); 
                try { 
                    LocalesRunner.super.methodBlock(method).evaluate(); 
                } finally { 
                    LocaleHolder.set(oldLocale); 
                } 
            }         }; 
    } 
    @Override// The name of the test class 
    protected String getName() { 
        return String.format("%s [%s]", super.getName(), locale); 
    }
    @Override// The name of the test method 
    protected String testName(final FrameworkMethod method) { 
        return String.format("%s [%s]", method.getName(), locale); 
    } 
}

Now we are almost done (heard that one before?). The example should compile and the Test at the beginning of this post should get executed for each locale in the list. We also extended the test name output, so that one can see what test was executed for which locale.

A last interesting challenge was to think about how to write unit test for a custom test runner. In a lot of cases, it was sufficient to provide a simple test case like in this example. In other cases I needed to develop the runners incrementally using TDD, so I wanted smaller, specific unit tests. It turned out, that it is pretty easy to run JUnit for my test runner test cases programmatically:

private Result runTest(Class<?> testClass) {
    JUnitCore core = new JUnitCore();
    return core.run(testClass);
}

Writing such custom runners is obviously not difficult and can be done for specific problems in any project. It is done fast an helps you to provide more convenient ways to write test according to your projects needs.

Reuse and composition of Unit Tests

From time to time I want to be able to reuse already finished unit test. e.g. when I do a new implementation of an interface. In most cases, I did write an abstract base fixture, that I extend as needed:

public abstract class IterableFixture<T> {
    protected  abstract Iterable<T> create(); 
    @Test
    public void iterationDoesSomething(){
        assertThat(…);
    }
}

Whenever I need to write an Iterable<T> now, I can reuse the generic test like this:

public class MyIterableTest extends IterableFixture<Integer> {
    @Override
    protected Iterable<Integer> create(){
        return new MyIterable<Integer>(…);
    } 
    @Test
    public void otherTest(){
        assertThat(…);
    }
}

Over the time, multiple base fixtures emerged, e.g. for equals/hashcode implementations, iteration, collections, serializing  and so on. When new classes needed multiple of these tests, I was forced to implement multiple test classes, e.g. MyFooIsSerializableTest, MyFooImplementsIterableTest, MyFooEqualsTest,… Although this was simple and practical, I never really  liked that I have too check for multiple green bubbles inside my IDE’s test runner for one class under test. I also did not like that I could not see in one spot what features my classes are implementing. This was extremely true for collection implementations, as they can be un/modifiable, im/mutable and fixed/variable-size. Not to speak about allowing null or similar variants.

What I really wanted, was to run my single test for my class and have each feature nicely listed, in a similar manner as JUnit Suites do:

Just using a Suite did not help much, because I still had to extend my base fixtures. Additionally I had to add them to the Suites parameter list.

One of the reason that it was so clumsy to work with this approach are the plethora of derived classes - one for each feature that was implemented. So I decided, that using A factory that gets injected into the test classes constructor will simplify the approach. Then I looked at JUnits Parameterized runner, as it has the similar task to instantiate test fixture with constructor argument. The JUnit extension mechanism for custom runners showed to be very straight forward, so I decided to build my own Features Runner.

While thinking about how the using this runner I tried different variant on where to place which annotation. My requirements were:

Information shall  be defined statically using annotations

The user shall only have to implement the factories that create the systems under test

The factories shall be as  simple as possible

The factory implementations shall not impose any requirements on its constructors. This implies, that the user will new the factories, which give him the advantage that he can debug problems on more complex factories.

The feature shall look and feel like using JUnits built in runners. Parameterized is considered a nice template

Each generic feature fixture will have its on associated factory

I have spiked several versions. Here is an example of the final one. A class testing a collection for the features of being iterable and unmodifiable.

@RunWith(Features.class)
public class ArrayAdapterTestSuite {
    @Feature(Unmodifiable.class)
    public static Unmodifiable.Factory<Integer> unmodifiableFeature() {
        return new Unmodifiable.Factory<Integer>() {
            @Override
            public Collection<Integer> createCollection() {
                return asList(0, 1, 2);
            }

            @Override
            public Integer createUniqueItem(int id) {
                return id;
            }
        };
    }
    @Feature(Iterable.class)
    public static Iterable.Factory<Integer> iterableFeature() {
        return new Iterable.Factory<Integer>() {
            @Override
            public java.lang.Iterable<Integer> createIterable() {
                return asList(0, 1, 2);
            }
        };
    }
}

The feature fixture is implemented like this:

public class Unmodifiable<T> implements FeatureFixture {
    private final Factory<T> factory;

    public Unmodifiable(Factory<T> factory) {
        this.factory = factory;
    }

    public interface Factory<T> {
        Collection<T> createCollection();
        T createUniqueItem(int id);
    }

    @Test(expected = UnsupportedOperationException.class)
    public void addIsUnsupported() {
        Collection<T> unmodifiable = factory.createCollection();
        unmodifiable.add(factory.createUniqueItem(42));
    }
}

The feature test is self contained. Only a factory needs to be supplied. The features are assembled by tagging a public static method creating a factory or a public static field with a @Feature annotations. The runner will ensure that the types of the factory and the annotated feature do match. This is a runtime check, as there is no way to achieve this at compile time because of java’s type erasure.

The Features Suite runner will get executed because the @RunsWith annotation on top of your test class. JUnit will instantiate it and pass it the Class<?> descriptor of your test. It will get inspected in the same way JUnit does it by using reflection. The gained information will be used to construct explicit FeatureRunner with the feature test Class<?> descriptor and the extracted factory.

Because the FeatureRunner would execute a group of test cases, I decided to use Suite as a base class for the implementation:

public class Features extends Suite {
    public Features(Class<?> klass) throws InitializationError {
        super(klass, extractAndCreateRunners(klass));
    }

    private static List<Runner> extractAndCreateRunners(Class<?> klass)
                                                  throws InitializationError {
        List<Runner> runners = new ArrayList<Runner>();
        for (FeatureAccessor field : extractFieldsWithTest(klass)) {
            Class<? extends FeatureFixture> test = field.getFeature();
            runners.add(new FeatureRunner(test, field.getFactory()));
        }
        addSuiteIfItContainsTests(klass, runners);
        return runners;
    }

    private static void addSuiteIfItContainsTests(Class<?> klass,
                                                  List<Runner> runners) {
        try {
            runners.add(new BlockJUnit4ClassRunner(klass));
        } catch (InitializationError e) {// do nothing, no tests
        }
    }

    private static abstract class FeatureAccessor<TField …> {
        private final TField field;
        static <TField extends …> boolean isValid(TField field) {
            return Modifier.isPublic(field.getModifiers())
                    && Modifier.isStatic(field.getModifiers())
                    && field.isAnnotationPresent(Feature.class);
        }

        static <TField …> FeatureAccessor<?> createFrom(final TField field) {
            …
        }

        Class<? extends FeatureFixture> getFeature() {
            return field.getAnnotation(Feature.class).value();
        }
       
        Object getFactory() { 
            …
            return this.field.invoke(null);
        }
    }

    private static List<FeatureAccessor> extractFieldsWithTest(Class<?> klass) {
        List<FeatureAccessor> factoryFieldsWithProperty =
                                 new ArrayList<FeatureAccessor>();
        for (Field field : klass.getFields()) {
            if (!FeatureAccessor.isValid(field)) {
                continue;
            }
            factoryFieldsWithProperty.add(FeatureAccessor.createFrom(field));
        }
        …
        return factoryFieldsWithProperty;
    }
}

The FeatureRunner simply extends JUnits default runner in passing the factory instance into the test constructor:

class FeatureRunner extends BlockJUnit4ClassRunner {
    private final Object factory;

    FeatureRunner(Class<?> klass, Object factory)
                                          throws InitializationError {
        super(klass);
        if(factory == null){
            throw new InitializationError(…);
        }
        this.factory = factory;
    }

    protected Object createTest() throws Exception {
        return getTestClass().getOnlyConstructor().newInstance(factory);
    }
    …
}

Changing my base class fixtures to FeatureFixtures was extremely simple and composing standard unit test bits really is fun now! If you are interested in this solution, the source code is on google-code. I hope it will help you as much as it does help me!