Skip to content

Windows build_tests_2_9 is 2.13% flaky #163118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fluttergithubbot opened this issue Feb 12, 2025 · 10 comments
Closed

Windows build_tests_2_9 is 2.13% flaky #163118

fluttergithubbot opened this issue Feb 12, 2025 · 10 comments
Assignees
Labels
c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-tool Owned by Flutter Tool team

Comments

@fluttergithubbot
Copy link
Contributor

The post-submit test builder Windows build_tests_2_9 had a flaky ratio 2.13% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20build_tests_2_9/1018
Commit: 78e5d79

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20build_tests_2_9/1018
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20build_tests_2_9/1013

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Windows%20build_tests_2_9

Please follow https://github.com/flutter/flutter/blob/master/docs/infra/Reducing-Test-Flakiness.md#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

@fluttergithubbot fluttergithubbot added c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-tool Owned by Flutter Tool team labels Feb 12, 2025
@harri35
Copy link
Contributor

harri35 commented Feb 18, 2025

@bkonyi
Same issue and recommendation as #163121 (comment)

@harri35
Copy link
Contributor

harri35 commented Feb 21, 2025

Waiting to see if #163798 will fix it

@harri35
Copy link
Contributor

harri35 commented Feb 25, 2025

@bkonyi same here - this should be fixed by #163798 (merged on 21.02.2025)
Currently, the logs look clean.
If the logs remain clean, I recommend closing this in a day or two.

@fluttergithubbot
Copy link
Contributor Author

[prod pool] flaky ratio for the past (up to) 100 commits between 2025-02-20 and 2025-02-25 is 0.00%. Flaky number: 0; total number: 92.

@harri35
Copy link
Contributor

harri35 commented Feb 27, 2025

@jmagman (cc: @bkony) I recommend we close this one:

  1. Fix was done in Update gradle memory properties in example and test projects #163798 and merged on 21st
  2. Flakyiness decreased

@fluttergithubbot
Copy link
Contributor Author

[prod pool] flaky ratio for the past (up to) 100 commits between 2025-02-26 and 2025-03-04 is 1.04%. Flaky number: 1; total number: 96.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20build_tests_2_9/1416
Commit: 87c87c0
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20build_tests_2_9/1416

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Windows%20build_tests_2_9

@harri35
Copy link
Contributor

harri35 commented Mar 5, 2025

@jmagman FYI: One more Java heap space issue, this is after the #163798 was marged: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8721622604347154753/+/u/run_test.dart_for_build_tests_shard_and_subshard_2_9/stdout

This is from flutter\examples\layers\android app and I checked, in 87c87c0 it had the updated values:

org.gradle.jvmargs=-Xmx8G -XX:MaxMetaspaceSize=4G -XX:ReservedCodeCacheSize=512m -XX:+HeapDumpOnOutOfMemoryError

Other previously affected ones look good, no more memory issues there:

As already discussed in multiple threads (#158889, #163798 (comment)), hard to debug these as it is not transparent how and why Gradle uses the memory and are the memory requirements OK or is there a bug somewhere in Gradle.

Should there be some kind of a follow-up issue here? Or let's just wait and see for now, as it seems the problem was alleviated.

@harri35
Copy link
Contributor

harri35 commented Mar 5, 2025

There is a similar one here: #164182 (comment)

@jmagman
Copy link
Member

jmagman commented Mar 7, 2025

Should there be some kind of a follow-up issue here? Or let's just wait and see for now, as it seems the problem was alleviated.

#158889 is to do something better than guessing, but we're unlikely to spend much time on it if my change reduced the flake rate, and we're only seeing the issue in CI and not reported by real customers.

I think we should close the issues that were opened before my change, and let the flaky bot tell of it's still an issue.

Thanks so much for following up on this, @harri35!

@jmagman jmagman closed this as completed Mar 7, 2025
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 21, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-tool Owned by Flutter Tool team
Projects
None yet
Development

No branches or pull requests

4 participants