Skip to content

iOS testbed runner prone to failure when two instances start near the same time #129200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
freakboy3742 opened this issue Jan 22, 2025 · 2 comments · Fixed by python/buildmaster-config#573
Labels
OS-ios type-bug An unexpected behavior, bug, or error

Comments

@freakboy3742
Copy link
Contributor

freakboy3742 commented Jan 22, 2025

Bug report

Bug description:

The iOS testbed runner starts an iOS simulator; however, the mechanism that starts the simulator doesn't return the ID of the simulator that was started. The runner needs to know the ID of the simulator that was started so that future commands can be directed at that simulator.

The current implementation of the testbed finds this ID by generating a list of know simulators, and then waiting until another entry appears. This works fine until there are 2 test runs started at near the same time, and the runner is unable to identify which simulator this test run has started. This results in test failures like the following:

note: Run script build phase 'Prepare Python Binary Modules' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'iOSTestbed' from project 'iOSTestbed')
note: Run script build phase 'Install Target Specific Python Standard Library' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'iOSTestbed' from project 'iOSTestbed')
Found more than one new device: {'5CAA0336-9CE1-4222-BFE3-ADA405F766DE', 'DD108383-685A-4400-BF30-013AA82C4A61'}
make: *** [testios] Error 1
program finished with exit code 2

A re-run of the same test job will usually fix the problem, but it shouldn't happen in the first place.

This almost never happens locally, because users don't normally start to test runs in parallel. However, it's a set of conditions that is reasonably likely on the CI machine, as it's very easy for a PR and a main CI job to start at the same time (most commonly, when you merge a PR and immediately backport to a previous release).

CPython versions tested on:

3.13

Operating systems tested on:

macOS, Other

Linked PRs

@freakboy3742
Copy link
Contributor Author

python/buildmaster-config#573 should be an immediate workaround to avoid the error in CI.

@freakboy3742
Copy link
Contributor Author

I'm going to re-open this; it's going to be a problem for other users who want to use the testbed in CI configurations, as it doesn't play nicely with pytest-xdist. Parallelized test runs can cause multiple iOS tests to run, triggering this issue.

It would also be desirable to increase the parallelization on the iOS worker as well - the CPU is able to handle the load, it's purely the startup ordering sequence that is an issue.

@freakboy3742 freakboy3742 reopened this Feb 26, 2025
freakboy3742 added a commit that referenced this issue Feb 27, 2025
Add a lock to ensure that only one iOS testbed per user can start at a time, so
that the simulator discovery process doesn't collide between instances.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Feb 27, 2025
…thonGH-130564)

Add a lock to ensure that only one iOS testbed per user can start at a time, so
that the simulator discovery process doesn't collide between instances.
(cherry picked from commit 9211b3d)

Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742 added a commit that referenced this issue Feb 28, 2025
…H-130564) (#130657)

Add a lock to ensure that only one iOS testbed per user can start at a time, so
that the simulator discovery process doesn't collide between instances.
(cherry picked from commit 9211b3d)

Co-authored-by: Russell Keith-Magee <[email protected]>
freakboy3742 added a commit to freakboy3742/buildmaster-config that referenced this issue Feb 28, 2025
zware pushed a commit to python/buildmaster-config that referenced this issue Feb 28, 2025
seehwan pushed a commit to seehwan/cpython that referenced this issue Apr 16, 2025
…thon#130564)

Add a lock to ensure that only one iOS testbed per user can start at a time, so
that the simulator discovery process doesn't collide between instances.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OS-ios type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant