-
-
Notifications
You must be signed in to change notification settings - Fork 596
ci: Fix flaky tests #1668
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: Fix flaky tests #1668
Conversation
Thanks for opening this pull request!
|
Codecov ReportBase: 99.89% // Head: 99.89% // No change to project coverage 👍
Additional details and impacted files@@ Coverage Diff @@
## alpha #1668 +/- ##
=======================================
Coverage 99.89% 99.89%
=======================================
Files 61 61
Lines 5973 5973
Branches 1367 1367
=======================================
Hits 5967 5967
Misses 6 6
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
@mtrezza This is ready for review |
@mtrezza There are a few that I can't figure out, run the following commands a few time to replicate this run a few times. This will produce the same exact random run as the job. You can find the randomizer seed from jasmine. I'll add these in a separate PR if I find a fix.
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4012974929/jobs/6891853452 |
Well investigated, we can merge even if there are some flaky tests left. They can be done in a separate PR as this PR is already an improvement. Just let me know when you think it's ready. |
@mtrezza Its ready |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
🎉 This change has been released in version 4.0.0-alpha.6 |
🎉 This change has been released in version 4.0.1-beta.1 |
🎉 This change has been released in version 4.0.1 |
New Pull Request Checklist
Issue Description
The integration tests are ran in random order. This will allow us to catch invariants within the tests. Meaning the state of the entire suite should be the same before and after each test within a spec file. Most flaky tests can be found by running them standalone with
fit
or running the suite randomly and check the order than ran.Sometimes the job status is returned as
running
when it should besuccessful
. The status is returned too quickly before the job is complete. There potentially could be performance improvements on the server side.https://github.com/parse-community/Parse-SDK-JS/actions/runs/4001584644/jobs/6867964716
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4011556752/jobs/6889541063
The schema isn't cleared before each test. When the server starts there is a
_User
and_Role
schema createdhttps://github.com/parse-community/Parse-SDK-JS/actions/runs/4011007512/jobs/6888113575
These tests relied on the state of previous tests, specifically when
Parse.User.enableUnsafeCurrentUser
should be usedhttps://github.com/parse-community/Parse-SDK-JS/actions/runs/4006643738/jobs/6878441649
https://github.com/parse-community/Parse-SDK-JS/actions/runs/4006643738/jobs/6878442018
The MongoDB query optimizer can return different results when using explain() + hint(). https://www.mongodb.com/docs/manual/reference/explain-results/#std-label-queryPlanner
TODOs before merging