Skip to content

ci: Fix flaky tests #1668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 26, 2023
Merged

Conversation

dplewis
Copy link
Member

@dplewis dplewis commented Jan 26, 2023

New Pull Request Checklist

Issue Description

The integration tests are ran in random order. This will allow us to catch invariants within the tests. Meaning the state of the entire suite should be the same before and after each test within a spec file. Most flaky tests can be found by running them standalone with fit or running the suite randomly and check the order than ran.

Sometimes the job status is returned as running when it should be successful. The status is returned too quickly before the job is complete. There potentially could be performance improvements on the server side.
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4001584644/jobs/6867964716
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4011556752/jobs/6889541063

The schema isn't cleared before each test. When the server starts there is a _User and _Role schema created
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4011007512/jobs/6888113575

These tests relied on the state of previous tests, specifically when Parse.User.enableUnsafeCurrentUser should be used
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4006643738/jobs/6878441649
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4006643738/jobs/6878442018

The MongoDB query optimizer can return different results when using explain() + hint(). https://www.mongodb.com/docs/manual/reference/explain-results/#std-label-queryPlanner

1) Parse Query can query with hint
  - TypeError: Cannot read properties of undefined (reading 'inputStage')

TODOs before merging

  • Add tests
  • Add changes to documentation (guides, repository pages, in-code descriptions)

@parse-github-assistant
Copy link

parse-github-assistant bot commented Jan 26, 2023

Thanks for opening this pull request!

  • ❌ Please edit your post and use the provided template when creating a new pull request. This helps everyone to understand your post better and asks for essential information to quicker review the pull request.

@codecov
Copy link

codecov bot commented Jan 26, 2023

Codecov Report

Base: 99.89% // Head: 99.89% // No change to project coverage 👍

Coverage data is based on head (9a10aae) compared to base (0b2985c).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff           @@
##            alpha    #1668   +/-   ##
=======================================
  Coverage   99.89%   99.89%           
=======================================
  Files          61       61           
  Lines        5973     5973           
  Branches     1367     1367           
=======================================
  Hits         5967     5967           
  Misses          6        6           
Impacted Files Coverage Δ
src/ParseQuery.js 100.00% <100.00%> (ø)
src/StorageController.react-native.js 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@dplewis dplewis requested a review from a team January 26, 2023 01:13
@dplewis
Copy link
Member Author

dplewis commented Jan 26, 2023

@mtrezza This is ready for review

@mtrezza mtrezza requested a review from a team January 26, 2023 02:57
@mtrezza mtrezza changed the title fix: Flaky test ci: Fix flaky tests Jan 26, 2023
@dplewis
Copy link
Member Author

dplewis commented Jan 26, 2023

@mtrezza There are a few that I can't figure out, run the following commands a few time to replicate this run a few times. This will produce the same exact random run as the job. You can find the randomizer seed from jasmine. I'll add these in a separate PR if I find a fix.

npm run integration -- --seed=31902

- Error: Timeout - Async function did not complete within 20000ms (set by jasmine.DEFAULT_TIMEOUT_INTERVAL)
        - MongoNotConnectedError: Client must be connected before running operations

https://github.com/parse-community/Parse-SDK-JS/actions/runs/4011772368/jobs/6889651856

npm run integration -- --seed=69922

Parse LiveQuery can unsubscribe with await to multiple queries different class
  - Unhandled promise rejection: TypeError: message.split is not a function

https://github.com/parse-community/Parse-SDK-JS/actions/runs/4012974929/jobs/6891853452
Edit: For the LiveQuery error its a server side issue. I opened a issue parse-community/parse-server#8406

@mtrezza
Copy link
Member

mtrezza commented Jan 26, 2023

Well investigated, we can merge even if there are some flaky tests left. They can be done in a separate PR as this PR is already an improvement. Just let me know when you think it's ready.

@dplewis
Copy link
Member Author

dplewis commented Jan 26, 2023

@mtrezza Its ready

Copy link
Member

@mtrezza mtrezza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@mtrezza mtrezza merged commit eaa1f10 into parse-community:alpha Jan 26, 2023
@dplewis dplewis deleted the flaky-test-cloud-job branch January 26, 2023 17:53
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 4.0.0-alpha.6

@parseplatformorg parseplatformorg added the state:released-alpha Released as alpha version label Jan 27, 2023
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 4.0.1-beta.1

@parseplatformorg parseplatformorg added the state:released-beta Released as beta version label Jan 31, 2023
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 4.0.1

@parseplatformorg parseplatformorg added the state:released Released as stable version label Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
state:released Released as stable version state:released-alpha Released as alpha version state:released-beta Released as beta version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants