Upload xcresults to LUCI cloud storage #41647

cyanglaz · 2023-05-01T20:54:39Z

Taking over from #41644

fixes: flutter/flutter#125823

Steps to verify

CI should pass with only the test script change, no xcresults are uploaded.
[draft ]Upload xcresult with scenario test fail #41652 should fail and xcresult should be uploaded.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

cyanglaz · 2023-05-10T22:13:29Z

Scenario app test failure and result uploaded: #41652

cyanglaz · 2023-05-11T04:20:40Z

Unittest failed and result uploaded: #41929

upload scenario app test result fix script

gaaclarke · 2023-05-12T16:21:55Z

testing/run_tests.py

@@ -752,6 +753,12 @@ def run_objc_tests(ios_variant='ios_debug_sim_unopt', test_filter=None):
    ios_unit_test_dir = os.path.join(
        BUILDROOT_DIR, 'flutter', 'testing', 'ios', 'IosUnitTests'
    )
+
+    result_bundle_temp = tempfile.TemporaryDirectory(


The way you use TemporaryDirectory is with a with statement to make sure that it gets cleaned up.

gaaclarke · 2023-05-12T16:22:46Z

testing/run_tests.py

@@ -760,13 +767,27 @@ def run_objc_tests(ios_variant='ios_debug_sim_unopt', test_filter=None):
        'xcodebuild '
        '-sdk iphonesimulator '
        '-scheme IosUnitTests '
-        "-destination name='" + new_simulator_name + "' "
+        '-resultBundlePath ' + result_bundle_path + " -destination name='" +


Please follow the formatting of having a flag at the start of each line.

gaaclarke · 2023-05-12T16:23:05Z

testing/run_tests.py

+    # Upload the xcresult when the tests fail.
+    luci_test_outputs_path = os.environ.get('FLUTTER_TEST_OUTPUTS_DIR')
+    xcresult_bundle = os.path.join(result_bundle_temp, 'ios_embedding.xcresult')
+    print(xcresult_bundle)


Stray print statement.

gaaclarke · 2023-05-12T16:24:44Z

testing/scenario_app/run_ios_tests.sh

@@ -41,25 +41,57 @@ fi
 # Can also be set via Simulator app Device > Rotate Device Automatically
 defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1

-cd $SRC_DIR/out/$FLUTTER_ENGINE/scenario_app/Scenarios
+SCENARIO_PATH=$SRC_DIR/out/$FLUTTER_ENGINE/scenario_app/Scenarios
+cd $SCENARIO_PATH


I'd pushd and pop around here to make sure you don't effect any downstream commands.

Oh I'm just realizing you didn't change this cd, just shifted the code. The push and pop aren't necessary but don't hurt. Thanks for looking into it.

gaaclarke · 2023-05-12T16:26:12Z

testing/scenario_app/run_ios_tests.sh

+# Zip and upload xcresult to luci.
+# First parameter ($1) is the zip output name.
+ZIP_AND_UPLOAD_XCRESULT_TO_LUCI () {
+  # Using absolute directory causes the zip containing all the sub directories.


Can you clear this up please? You are probably running into the difference between having a trailing slash or not fwiw.

gaaclarke · 2023-05-12T16:26:36Z

testing/scenario_app/run_ios_tests.sh

+ZIP_AND_UPLOAD_XCRESULT_TO_LUCI () {
+  # Using absolute directory causes the zip containing all the sub directories.
+  # So use relative directory (./$RESULT_BUNDLE_FOLDER) instead.
+  echo $1


Remove the echo or make it more useful by adding a label.

gaaclarke · 2023-05-12T16:28:17Z

testing/scenario_app/run_ios_tests.sh

+
+# Zip and upload xcresult to luci.
+# First parameter ($1) is the zip output name.
+ZIP_AND_UPLOAD_XCRESULT_TO_LUCI () {


Make the function name lowercase please.

gaaclarke

lgtm, thanks!

cyanglaz · 2023-05-15T19:48:20Z

testing/run_tests.py

+        test_command[0] = test_command[0] + ' -only-testing:%s' % test_filter
+      run_cmd(test_command, cwd=ios_unit_test_dir, shell=True)
+
+      # except:


@gaaclarke Just found this is mistakenly commented out, I will create a patch to fix it.

The PR landed before I can remove the auto_submit label :( I was a split second slower :(

Ahh, nice job seeing the problem. This wasn't showing up as an error in CI? Do we need another test?

It wouldn't as I also missed "try"

…126876) flutter/engine@00f20fb...027ca79 2023-05-15 [email protected] Use release_build and os dimension consistently. (flutter/engine#42012) 2023-05-15 [email protected] [Impeller] Add interactive DrawPaint blend test (flutter/engine#42031) 2023-05-15 [email protected] [Impeller] Limit subpass textures and backdrop blurs to the current clip (flutter/engine#42039) 2023-05-15 [email protected] Roll Dart SDK from c302a0252785 to d2b2ac829842 (1 revision) (flutter/engine#42051) 2023-05-15 [email protected] Upload xcresults to LUCI cloud storage (flutter/engine#41647) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

The original PR #41647 forgot to add try-except-raise fixes flutter/flutter#125823 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

…lutter#126876) flutter/engine@00f20fb...027ca79 2023-05-15 [email protected] Use release_build and os dimension consistently. (flutter/engine#42012) 2023-05-15 [email protected] [Impeller] Add interactive DrawPaint blend test (flutter/engine#42031) 2023-05-15 [email protected] [Impeller] Limit subpass textures and backdrop blurs to the current clip (flutter/engine#42039) 2023-05-15 [email protected] Roll Dart SDK from c302a0252785 to d2b2ac829842 (1 revision) (flutter/engine#42051) 2023-05-15 [email protected] Upload xcresults to LUCI cloud storage (flutter/engine#41647) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

cyanglaz mentioned this pull request May 1, 2023

[draft ]Upload xcresult with scenario test fail #41652

Closed

jmagman mentioned this pull request May 4, 2023

iOS objc FlutterChannelsTest tests flaky on an arm64 build flutter/flutter#126013

Closed

cyanglaz mentioned this pull request May 11, 2023

[draft] Upload xcresult with unittest fail #41929

Closed

Upload xcresults to LUCI cloud storage

233fc22

upload scenario app test result fix script

cyanglaz force-pushed the upload_xcresult branch from e38a4e1 to 233fc22 Compare May 11, 2023 04:21

Chris Yang added 3 commits May 11, 2023 09:45

fix

244d39e

remove result bundle after zipped and moved

df346c9

remove

d6c732f

cyanglaz marked this pull request as ready for review May 12, 2023 15:36

cyanglaz requested a review from gaaclarke May 12, 2023 15:36

gaaclarke suggested changes May 12, 2023

View reviewed changes

Chris Yang added 6 commits May 12, 2023 09:51

fix py

348a709

some bash fixes

6f7a7ca

format

6193320

zip comment

0493106

format python

c94b62b

push and pop

6fddd21

cyanglaz requested a review from gaaclarke May 15, 2023 18:28

gaaclarke approved these changes May 15, 2023

View reviewed changes

cyanglaz added the autosubmit Merge PR when tree becomes green via auto submit App label May 15, 2023

auto-submit bot merged commit 0a8308a into flutter:main May 15, 2023

cyanglaz deleted the upload_xcresult branch May 15, 2023 19:07

cyanglaz commented May 15, 2023

View reviewed changes

engine-flutter-autoroll mentioned this pull request May 15, 2023

Roll Flutter Engine from 00f20fbab038 to 027ca797985f (5 revisions) flutter/flutter#126876

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 15, 2023

0a8308a30 Upload xcresults to LUCI cloud storage (flutter/engine#41647)

999ca19

cyanglaz mentioned this pull request May 15, 2023

Fix upload xcresult script in run_tests.py #42056

Merged

8 tasks

cyanglaz mentioned this pull request May 17, 2023

Mac_ios microbenchmarks_ios is 7.00% flaky. XCode build concurrent build failure. flutter/flutter#126412

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Upload xcresults to LUCI cloud storage #41647

Upload xcresults to LUCI cloud storage #41647

Uh oh!

cyanglaz commented May 1, 2023 •

edited

Loading

Uh oh!

cyanglaz commented May 10, 2023

Uh oh!

cyanglaz commented May 11, 2023

Uh oh!

gaaclarke May 12, 2023

Uh oh!

gaaclarke May 12, 2023

Uh oh!

gaaclarke May 12, 2023

Uh oh!

gaaclarke May 12, 2023

Uh oh!

gaaclarke May 15, 2023

Uh oh!

gaaclarke May 12, 2023

Uh oh!

gaaclarke May 12, 2023

Uh oh!

gaaclarke May 12, 2023

Uh oh!

gaaclarke left a comment

Uh oh!

cyanglaz May 15, 2023

Uh oh!

cyanglaz May 15, 2023

Uh oh!

gaaclarke May 15, 2023

Uh oh!

cyanglaz May 15, 2023

Uh oh!

Uh oh!

Upload xcresults to LUCI cloud storage #41647

Upload xcresults to LUCI cloud storage #41647

Uh oh!

Conversation

cyanglaz commented May 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

cyanglaz commented May 10, 2023

Uh oh!

cyanglaz commented May 11, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaaclarke left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cyanglaz commented May 1, 2023 •

edited

Loading