Add a trybot for flutter customer tests

[srawlins]

I have landed several changes into dart-lang/sdk, which have each ultimately caused some bot somewhere to fail, as each change causes one or most customer tests to fail in the flutter/tests repo. There is no trybot for these "flutter customer tests."

This current feedback loop is very expensive for me and for the flutter engine rollers:

1. it takes several days for me to find out the change causes new failures in flutter/tests
2. it breaks some flutter auto-roller or other
3. breaking the auto-roller requires humans to identify the culprit, and revert my change
4. I must fix each customer's codebase to be compatible with my change
5. I must bump the commit SHA of each customer's codebase in flutter/tests
6. I land again, without any customer test trybots, cycling back to the top of this feedback loop.

CC @Hixie @jason-simmons @iskakaushik to perhaps fill in details I got wrong above

[srawlins]

I can find more references, but as one data point, here are links for the latest cycle:

• My CL: https://dart-review.googlesource.com/c/sdk/+/277700
• Flutter Engine failure: Roll Flutter Engine from f79030440948 to 5aee7914c7cd (1 revision) flutter/flutter#118515
• log: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8791703831221213185/+/u/Customer_testing/customer_testing/test_stdout
• subsequent issue to track customer issues: Flutter customer tests override operator == with a dynamic parameter. #51038, with links to my PRs to fix customer issues

[Hixie]

The flutter/tests tests are designed to be runnable from any repo, so it should be possible to run them upstream. I forget if we have an environment variable to force a local engine to be used (I presume you'd have to build a local engine and use that to run the tests to test Dart changes), but if we don't we should be able to add one. cc @christopherfujino

[whesse]

Regarding local engine: The engine_v2 recipes run tests with a custom engine by passing a custom FLUTTER_STORAGE_BASE_URL and using flutter tools, not using the local-engine flag.

These tests are run through the adhoc_validation recipe module, and not through the standard test script 'dev/bots/test.dart'.

So to add this (to monorepo testing), we will create a new validation_tester recipe that can be triggered from the engine_v2 api, that spawns testers in new builds after the engine build is complete. We already have an engine_v2/tester that runs dev/bots/test.dart, but this should be a different recipe to run adhoc_validation steps instead.

We will have to make sure that the adhoc_validation steps run correctly on a flutter/flutter checkout with a custom artifact download URL - we have a bug filed because custom artifact download URLs break some tests that run flutter tools with a --fatal-warnings flag, but are working on fixing this.

This change would need to be made in the flutter recipes anyway, as part of the new engine_v2 Build/Archive/Test separation they are moving their builds to, so we will just do it earlier.

CC @godofredoc

The timeline for implementing this will be weeks, not days.

[srawlins]

Thanks much for all of the context, @whesse !!!

The timeline for implementing this will be weeks, not days.

Very understandable, haha. Having this implemented in weeks would be a huge win for the analyzer team at least, and probably more. Thanks!

[srawlins]

Is it possible this might be prioritized for Q3? Thanks!

[a-siva]

Is it possible this might be prioritized for Q3? Thanks!

@godofredoc any possibility work on this could be bumped, when we run into issues with this codebase it is problematic for us to see if we have a fix or not without having the ability to run the changes locally

[godofredoc]

I can take a look next week to see if we can move customer testing to dev/bots/test.dart as it is only 4 commands in bash https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/adhoc_validation/resources/customer_testing.sh

[kallentu]

I ran into this a couple days ago where my change in the Dart SDK broke a Flutter customer. And there were two layers of errors, so it took a day or two to figure out it was my change and then my first forward fix fixed the first layer of errors, but there were more that I was unaware about until another day or two after. Then when I was creating a fix, the turnaround time for testing it would be up to half a day.

It's not a great experience to be mostly certain about the effects of your changes and need to test by waiting for multiple versions and rolls to go through.

Would love to see additional testing on the Dart pipeline 👍

[whesse]

This is now on my list of priority tasks to do, it is clearly an important request.

[srawlins]

Thanks much Bill, really super appreciated!

[srawlins]

If it's helpful to have some examples to play with, I'd like to run these three CRs against the flutter customer tests in try-bots:

• https://dart-review.googlesource.com/c/sdk/+/330561
• https://dart-review.googlesource.com/c/sdk/+/321361
• https://dart-review.googlesource.com/c/sdk/+/291042

[godofredoc]

Customer_testing is now a shard test and it is ready to be added to the monorepo configs. @whesse FYI

[whesse]

Customer_testing has been added to monorepo, and is now running on CI. It is available as a try job for Dart SDK CLs by adding the flutter-linux-try builder with the "Choose Tryjobs" link.

[srawlins]

Thanks so much, Bill!!! I'll be using this a lot this week. Much appreciated!

[godofredoc]

Thank you @whesse