Your Unit tests might not be as reliable as you thought

Sorry for the radio silence recently. I've taken a position as a DevOps (my other hat) so have been immersed in the world of Jenkins, Puppet, etc. But my love for Android still exists. :)

Recently I saw a Twitter post about unit testing and it mentioned the phrase "100% reliable". It reminded me of something that happened to me at my previous place where a unit test was passing but the app crashed on some devices.

Take this unit test:

1
2
3
4
5
6
7

public void validateDateParsing() throws Exception {
    SimpleDateFormat fmt = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSXXX");
    Date d = fmt.parse("2001-02-07T12:08:56.235-07:00");
    Calendar c = Calendar.getInstance(Locale.UK);
    c.setTime(d);
    assertEquals(c.get(Calendar.HOUR_OF_DAY), 19);
}

It's a noddy test to check that the parsing of a date string gives the correct hour. It runs. It passes.

So we put this code into our activity.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    TextView tv = (TextView) findViewById(R.id.text1);

    try {
        SimpleDateFormat fmt = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSXXX");
        Date d = fmt.parse("2001-02-07T12:08:56.235-07:00");
        Calendar c = Calendar.getInstance(Locale.UK);
        c.setTime(d);
        tv.setText("Hour is " + c.get(Calendar.HOUR_OF_DAY));
    } catch (ParseException e) {
        tv.setText("Cannot parse date");
    }
}

Run that on a modern device or an emulator running a recent Android API level. It will be fine and will display "Hour is 19".

BUT...try running it on an emulator that is running API 16. You'll see this:

But hold on. The unit test is fine. It passes. So why the crash? Well it turns out that the 'XXX' formatting instruction was only added in Android 4.3. This is from the java.text package and is not Android specific.

So the moral is that unit tests might not be 100% reliable after all. Ultimately you are running them on the JVM that is installed on the computer that you are developing on, and this is not the same as the JVM that runs on the Android device.

Darren @ Æ