Easier Android test reporting with Maven and Hudson

Despite being considered immature and/or a waste of time by many, automated testing of Android applicationis possible. Arguably it should be in the repertoire of any serious Android developer, but today I am not going to delve into this discussion and talk about Robotium, Robolectric and so on. I am just going to show you how it just got a lot easier .

Android testing is mostly done with the testing API in the SDK around what is called instrumentation testing. It is very close to what is often called integration testing since it basically means that you write a test suite that is basically an application that uses your tested application and confirms it behaves the way it should. This whole testing approach therefore relies on the fact that both applications are deployed to a device or emulator.

Doing this sort of testing has been ok within the IDE in the past, but did not go much beyond. Specifically if you wanted to run the tests on a number of devices and emulators, have it all automated and wanted to capture the test results it quickly became painful to manage.

And this is exactly where my recent improvements are stepping up. As you might know I have been involved with the Maven Android Plugin as core committer for quite a while. Recently I have been doing a lot merging for the upcoming 3.0.0 release. That resulted in a bunch of release becoming available that contain the critical enhancements I made for automated testing.

With the move away from the adb command line call to using ddmlib directly I introduced automated multi-device handling for the deploy, undeploy and redeploy goals.

Using the latest 3.0.0-alpha-6 release of the Maven Android Plugin you can now have multiple emulators running and devices attached and issue a command like

mvn android:deploy

and it will automatically iterate through and deploy the application to all of them. The same applies to pull and push goals of the plugin copying files to or from the device.

You can also narrow down the devices affected to only all emulators or only all devices or only a specific devices or emulator.

mvn android:deploy -Dandroid.device=emulator
mvn android:deploy -Dandroid.device=usb
mvn android:deploy -Dandroid.device=emulator-5554
mvn android:deploy -Dandroid.device=someAvdName
mvn android:deploy -Dandroid.device=deviceSerialNumber

So now you are ready to deploy your application in one sweep to all attached devices and test your awesomeness on a 10 inch tablet, a 4 inch phone and some other device all manually. Well hold on. You forgot about the automated part of testing.

If you set up your instrumentation testing with Maven like the MorseFlash project in the official samples of the Maven Android Plugin the multi device/emulator support will also work for that. So it will deploy the application and the test suite to all of them and then run the tests on the lot one after another.

And that is where my next enhancement comes in – JUnit compatible test report files. One of the problems with the instrumentation testing as supplied by the Android SDK is the fact that no report files are created. Luckily a few projects have stepped up and produced implementations of the InstrumentationTestRunner class that create JUnit compatible test reports. The problem is that by using them you have to include the class in your test project or all of the different test projects, modify build scripts and so on. All this can become a maintenance headache. Beyond that you have to deal with the fact that the report file is created on the device/emulator. That is a problem if you want to e.g. look at the files conveniently or get them post processed into nice HTML for access e.g. in a CI server like Hudson. Typically you then have to integrate adb pull operations into your build to get the file off the emulator.

So rather than producing the report file on the device I modified the TestRunListener I already implemented for logging on the host to also spit out a correctly formatted xml file for each test run on each device.

The files will be created in the standard Maven test report output location target/surefire-reports and be named so that the file is picked up by tools like Hudson automatically. The file names contain AVD name, serial number, manufacturer and model so that the are more or less human readable. E.g. for my Motorola XOOM, my Nexus S and an emulator named 233 the files look sort of like that

TEST-0212324143a0d357_Motorola_MZ604.xml
TEST-123453F1031F00EC_samsung_NexusS.xml
TEST-emulator-5554_233_unknown_sdk.xml

The file will contain all system properties from the host, all device properties from the emulator or device as well as any testing related data like stack traces, failure texts and so on. If you run your Android application build with a Maven build like this on the Eclipse Hudson CI server you get great report accessibility for a test class.

Stack trace and so on for failed tests:

And overall results view

as well as historic tracking of changes from build to build. And all that happens without additional configuration of the build or the test project…

I hope this will edge you all one step closer to write tests as part of your Android development and use a CI server like Hudson.

If you want to see more of that sort of stuff you should come to my testing workshop and Android application development with Maven presentation at AnDevCon later this year and register with my last name as discount code ;-)

Enjoy

manfred

PS: I am a committer on Eclipse Hudson, the Maven Android Plugin and a bunch of other things and can easily be found if you need help.

PPS: This should work on other CI servers like Jenkins and so on as well.

UPDATE: It seems like a regression kicked in later that disabled the default generation of the report. You can force it on with the plugin configuration

<test>
  <createReport>true</createReport>
</test>

or with the command line parameter -Dandroid.test.createReport=true

11 comments » Write a comment

  1. thanks Manfred for contributing such facilitators for testing !!!
    Now, the bonus question : how about code coverage ? I haven t tried yet (just in case you did) but a tool like jacoco (the follow up of eclemma) may run since it does not require to instrument code before executing the tests…
    I mean, if there is a way to launch the vm running your android app on the device with the jacoco java agent, that could be possible to retrieve the code coverage execution report (and then adb pull and then format in xml…)
    Anyway, this is just me thinking out loud, thanks for the android maven plugin, it does rock !

  2. No idea about coverage apart from the fact that there is already a coverage flag exposed in the config for the test run that I implemented on top of ddmlib. However I am not sure what happens when you activate it… let me know if you find out ;-)

  3. I can’t seem to get this working using the 3.0.0-alpha-13 version of the plugin. Anything special I should do to have this surefire report generation enabled?

  4. No.. it is enabled by default. Is it working for you with the morseflash samples application?

  5. No it’s not working for the sample app, mvn clean install runs the tests but there is only an empty surefire folder and no surefire-reports in the morseflash-instrumentation/target

  6. Seems like you found a regression. The default setting to true is being lost. Force it on with

    <test><createReport>true</createReport></test>

    true in the configuration or -Dandroid.test.createReport=true on the command line.

  7. And thanks for helping us getting rid of custom test runners!

Leave a Reply

Required fields are marked *.


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>