[integration_test] Recommend tests to be in integration_test/, fix example

[jiahaog]

Description

In this way, there is a clear distinction between integration tests that run on a device (in integration_test/, and widget tests that run with the flutter tester in test/.

Related Issues

flutter/flutter#64690

[dnfield]

LGTM

@nturgut @ditman any input on this?

[ditman]

Really small correction about integration tests not being supported in dev mode in web, and about the structure of web plugins.

[nturgut]

In the cirrus file we are still using test_driver/ directory even though it is not the recommended directory?

[nturgut]

Oh sorry, I just read the recommended package structure.

Is there a reason for these recommendations? One idea that comes to my mind is: integration_test_driver.dart won't be the same for different tests.

[jiahaog]

It won't be the same but I would think for most use cases the bulk of the testing code should go into the test script that runs on the device, and the driver is mainly for reporting.

Currently, some packages place the test scripts in test/foo_e2e.dart or test_driver/foo_e2e.dart. This works today, because of the old e2e naming, but if they were to rename their tests to like test/foo_integration_test.dart and test_driver/foo_integration_test.dart, this would conflict with globbing rules today that check for the *_test.dart suffix, hence the recommendation of a new integration_test/ directory for such tests

[nturgut]

We did the renaming like this last week in the for the web engine: flutter/engine#20954

[nturgut]

I think this approach blocks the possibilities that can be tested with e2e tests. Let me explain more in details:

For example in the extended test example we are creating a driver and give it to the integrationDriver method. We also give a handler for handling the screenshots.

This way, depending on the repository, developers can handle the screenshots differently. Even better, FlutterDriver can be created differently, for example a WebDriver can enable accessibility in advance. Or FlutterDriver can be provided with different options for the driver.

Also, the same application, can use different driver files since they can provide different functionalities.

Due to this reasons, we will probably won't switch to this usage in web integration tests on the engine side. I also recommend keeping the other option in the documentation for power users who wants to provide a wider set of tests.

[jiahaog]

@nturgut Sorry, I might have missed something. Can you elaborate more about how this approach blocks certain testing use cases? Customizing the driver is still possible with a custom driver file – in this PR, I have moved the extended test example to follow this proposed structure. There is a change to the command used to kick off the tests, but there is no change in functionality.

[nturgut]

@nturgut Sorry, I might have missed something. Can you elaborate more about how this approach blocks certain testing use cases? Customizing the driver is still possible with a custom driver file – in this PR, I have moved the extended test example to follow this proposed structure. There is a change to the command used to kick off the tests, but there is no change in functionality.

Sorry, I might be missing something, the README suggestion is:

An accompanying driver script will be needed that can be shared across all
integration tests. Create a `integration_test_driver.dart` in the `test_driver/`
directory with the following contents:

This suggestion might hide other possible usages. For example we can add:

• a note saying multiple driver files can be used
• we can also add an example of how we can pass a FlutterDriver

In my opinion, if developer adds multiple driver files, then the advantage of the new structure is not very clear:

lib/
  ...
integration_test/
  foo_test.dart
  bar_test.dart
test/
  # Other unit tests go here.
test_driver/
  integration_test_driver.dart
  integration_test_driver_with_screenshot.dart
  integration_test_driver_with_cstm_setup.dart

[jiahaog]

Thanks for clarifying – you're right that I wasn't clear enough in how we can use multiple driver files, I have updated the PR and linked it to the extended driver script.

My understanding is that one of the goals for package:integration_test is to push as much logic as possible into the test script, so that the test will be compatible with tools that do not provide access to the host, like Firebase Test Lab. I have not been exposed too much to web use cases, so let me know if I'm wrong, but I was going with the assumption that customizing the driver script should be only done rarely (such as to pull artifacts from the device), and there should be a significantly larger number of test scripts that run on the device to driver scripts which this recommendation will benefit. Taking a brief look at the e2etests/ directory of the engine, I see that most of the driver scripts are delegating to integrationDriver, so sharing the same driver script might be helpful there.

[nturgut]

Thanks for updating the documentation.

[jiahaog]

Thanks!