Sometimes tests pass when they shouldn't

The code was right. The test was wrong. But it still passed.

The app that we've been working on has a ViewPager and we're using Espresso for functional testing. One of the tests swipes between the pages of the ViewPager and checks for some text on each page.

Later on we added a DrawerLayout to the app. All the tests still passed - all was well with the world. Luckily one of the team is noticed that the swiping test was pulling out the drawer instead of swiping the ViewPager. They don't call him eagle-eyed Pete for nothing. (Actually they don't call him that as his name isn't Pete.) In any case the test wasn't failing even though it was doing something different to what was being tested for.

Investigation initiated.

The reason why the drawer was being pulled out is down to the Expresso swipeRight(). It started the swipe from the very left of the display, which caused the drawer to pop out.

That was fixed by writing an alternative swiping action that starts from a little inside the view, thusly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

static ViewAction slowSwipeRightFromLeftIndented() {
  return actionWithAssertions(new GeneralSwipeAction(Swipe.SLOW, CustomLocation.TOP_LEFT, CustomLocation.TOP_RIGHT, Press.FINGER));
}

public enum CustomLocation implements CoordinatesProvider {
    TOP_LEFT {
        @Override
        public float[] calculateCoordinates(View view) {
            final int[] xy = new int[2];
            view.getLocationOnScreen(xy);
            final float x = xy[0] + 100;
            final float y = xy[1];
            return new float[]{x, y};
        }
    },
    TOP_RIGHT {
        @Override
        public float[] calculateCoordinates(View view) {
            final int[] xy = new int[2];
            view.getLocationOnScreen(xy);
            final float x = xy[0] + view.getWidth() - 50;
            final float y = xy[1];
            return new float[]{x, y};
        }
    }
}

The 100 in line 11 is the number of pixels in from the left to start the swipe. On line 21 the swipe is stopped 50 pixels from the right of the view. There's no real reason for this. :)

But why was the test passing? The ViewPager creates not just the view you can see but the ones to the left and to the right as well. The test did a swipe and then checked for some text in the view but not that it was visible. Espresso looks through the whole view hierarchy, not just the view you can see, and that includes the pages to either side. The text being checked for existed.

Once the swiping fix was put in the test passed as expected. To make it then fail¹ the test was updated to check for the text and that it was visible on the screen using the isDisplayed() matcher.

Darren @ Æ