Reliability improvement for input date E2E tests #35505
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've run the Components E2E tests on a VM in a loop about 30 times, and the only ones that ever failed are:
They each only failed only 1 or 2 times over the 30 runs, but the fact that they are all roughly the same test suggests it's something problematic about those tests, and not just pure randomness. It's hard to know why they occasionally fail, but the screenshots captured in failing runs imply that their
ApplyInvalidInputDateValue
logic either doesn't run or has no effect when it does.Strictly speaking,
ApplyInvalidInputDateValue
is a bit artificial because it's simulating doing something that an end user could not do. It's creating an invalid state (via JavaScript) that you couldn't actually create using mouse and keyboard gestures. So I'm going to make the case that it's not really a valid thing for us to do anyway.In this PR,
ApplyInvalidInputDateValue
from all the tests, and replaced it with real user gesturesNow, with these improvements, I've run the full test suite in my VM another 10 times and it's had a 100% pass rate across that set. I'll keep the loop running.
Update
After running 78 more times, there were 4 failing runs:
ReloadingThePage_GracefullyDisconnects_TheCurrentCircuit
CanFocusDuringOnAfterRenderAsyncWithFocusInEvent
performFocusDuringOnAfterRender
flag, since this test (very unusually) causes a render to be triggered from a background thread, and reads/writes that flag simultaneously on different threads.