UI Testing

UI Testing Faces Inherent Challenges

As I mentioned in the section on state testing, UI testing faces inherent challenges. Animations inherently involve asynchronous delays, which introduce indeterminacy and make debugging difficult. Widgets are often poorly designed for programatic validation, as that is not their primary purpose. Indeed, the goals of an well-implemented widget (for example, a ListView that does not waste time rendering items that are off-screen), is often in direct conflict with easy programatic validation.

Consquently, in my own work with AFib, I allocate most of my testing effort to state-level testing. If functionality can be tested at the state test level, I test it there.

Push business level functionality into the SPI

The more you push business level logic into the SPI, the easier it becomes to validate your app in state tests. Before you begin writing a UI test, make sure that the functionality you are validating is actually UI logic, and not business logic that has leaked into your UI code.

Running UI tests

Running UI tests from the command line

Running the command:

dart bin/xxx_afib.dart test

Will execute all tests, including your UI prototype tests. The line for each UI prototype tests shows its full test id. If you add that ID to the command above, you will run only that test.

Running UI tests from the UI

AFib UI prototype tests can be executed from within prototype mode. You can watch them execute, which often reveals false assumptions or strange state changes that would be time-consuming to detect in the debugger.

To do so, navigate to the UI prototype in prototype mode, then swipe in the prototype mode drawer from the right. Click on the test panel, then click the 'run' button for the smoke test.

Writing UI Tests

How AFib's tests differ from Flutter's

When I first started using Flutter, I ran into some trouble with UI actions not invoking the expected widget handlers. The reasons for those issues were not clear to me, and may have been due to my inexperience with Flutter at the time.

At that time, stack overflow contained answers referencing this kind of issue. They often recommended that you simply lookup the widget definition, and call its even handler directly (e.g. find the OutlineButton, and call its onPressed member variable function).

All of AFib's UI testing works this way. Rather than working at the level of hit tests, it works at the level of the conceptual widget hierarchy, finding widget definitions and then invoking their handlers.

You must await everything except e.expect!

In your AFib UI tests, you must await every statement, except e.expect. Failing to do so will cause confusing bugs, because your test will continue executing prior to completing the statement that is missing the await. For example:

  prototype.defineSmokeTest( 
    body: (e) async {
      await e.matchText(EVEXWidgetID.textCountRouteParam, ft.equals("0"));
      await e.applyTap(EVEXWidgetID.buttonIncrementRouteParam);

      // THIS IS A NASTY BUG: Your test is going to continue executing without finishing this action.  
      e.applyTap(EVEXWidgetID.buttonIncrementRouteParam);
      await e.applyTap(EVEXWidgetID.buttonIncrementRouteParam);
      await e.matchText(EVEXWidgetID.textCountRouteParam, ft.equals("3"));

      await e.applyTap(EVEXWidgetID.buttonSaveTransientCount, verify: (verifyContext) {
        final write = verifyContext.accessOneQuery<WriteCountHistoryItemQuery>();
        e.expect(write.item.count, ft.equals(3));
      });

Use the verify parameter to validate actions

The apply methods support a verify parameter which yields an AFUIVerifyContext. That context will have captured any route parameter updates, navigation actions, or query executions that resulted from the apply action. You can use the AFUIVerifyContext to access those actions and verify their properties.

Use e.match... to validate data in the UI

You can use the various e.match variants to validate widget values. You can use e.matchWidget to directly access the a widget definition.

Use e.apply... to manipulate the UI

You can use the various e.apply variants to manipulate widget values.

Specifying test screen size

When I first started using Flutter, I was thrown off by the fact that my emulator screen size and the size of the virtual test screen were not the same. For that reason, running the command-line tests explicitly reports the screen size used by the test:

------------------------------------------------------------
Running at size phone-standard:portrait / 1170.0 w x 2532.0 h
------------------------------------------------------------

  // --test-size                The size used for command line tests, often used in conjunction with test-orientation
  // 
  //       [*]                  Or, [width]x[height], e.g. 1000x2000
  //       [phone-large]        1284.0 w x 2778.0 h
  //       [phone-standard]     1170.0 w x 2532.0 h
  //       [tablet-large]       2048.0 w x 2732.0 h
  //       [tablet-small]       1536.0 w x 2048.0 h
  //       [tablet-standard]    1640.0 w x 2360.0 h
  config.setValue("test-size", "tablet-large");

  // --test-orientation    The orientation used in command line tests
  // 
  //       [landscape]     Landscape, width larger than height
  //       [portrait]      Portrait, height larger than width
  config.setValue("test-orientation", "portrait");

Handling scrolling

I have mentioned several times that the need to scroll a UI to discover widgets that exists is problematic. Currently, AFib has logic which will auto-scroll to attempt to find a widget if it is referenced in a test and not initially found.

However, this is an area that needs to be made much more robust, and is unlikely to work properly in your app. Because the need to scroll is unpredictably dependent on screen size, I believe that AFib should handle it automatically (with perhaps some hints from the test), rather than forcing the test itself to contain explicit scrolling logic.

Efficiently Writing Tests with 3rd Party Widgets

Use matchWidget as a one-off escape

If you are working with a third-party widget, AFib will not know how to match/apply it. In that case, you can use:

final widget = await e.matchWidget<YourCustomWidgetType>(EVEXWidgetID.yourCustomWidget);

Having done so, you can either validate its properties, or you can call the event handler function associated with it.

Register custom extractors and applicators to teach AFib about third party widgets

If you have a third-party widget that you interact with from many tests, please see AFTestExtensionContext.registerApplicator and AFTestExtensionContext.registerExtractor. You can register small custom classes that teach AFib how to match/apply to that specific widget.

Please, don't waste too much time on UI testing

One last time: state-based testing is vastly more efficient than UI testing. Be careful before wasting too much time on UI testing.