Skip to content

Fix flaky test: use retry_asserts for observer event propagation#21417

Open
devin-ai-integration[bot] wants to merge 1 commit intomainfrom
devin/1775243804-fix-flaky-observer-test
Open

Fix flaky test: use retry_asserts for observer event propagation#21417
devin-ai-integration[bot] wants to merge 1 commit intomainfrom
devin/1775243804-fix-flaky-observer-test

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

Fixes flaky test_consume_events_ignores_non_in_flight_flow_runs (and the similar test_observer_consumes_cancelling_events) in tests/test_observers.py.

Root cause: Both tests emit events via emit_event() and then assert the observer's callback was invoked after a fixed asyncio.sleep(0.2). Events must travel through the EventsWorker → websocket → PrefectEventSubscriber pipeline before reaching the observer's _consume_events loop, and 0.2 seconds is insufficient under CI load.

Fix: Replace the fixed sleep + immediate assertion with the existing retry_asserts utility (from prefect._internal.testing), retrying up to 5 times with a 0.5 s delay between attempts. This is the same pattern used elsewhere in the test suite for eventual-consistency scenarios.

Failed workflow run: https://github.com/PrefectHQ/prefect/actions/runs/23958380669

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • No existing issue; this is a small flaky-test fix on main.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • If this pull request removes docs files, it includes redirect settings in mint.json.
  • If this pull request adds functions or classes, it includes helpful docstrings.

Link to Devin session: https://app.devin.ai/sessions/ed3a43bb4a694ba79c59949174bd7d8c

The test_consume_events_ignores_non_in_flight_flow_runs and
test_observer_consumes_cancelling_events tests relied on a fixed
asyncio.sleep(0.2) for events to propagate through the websocket
pipeline. Under CI load, this delay is insufficient, causing the
callback to not be called in time.

Replace the fixed sleep + immediate assertion with retry_asserts
(max_attempts=5, delay=0.5) which retries the assertion up to 5
times with 0.5s between attempts, properly handling event
propagation delays.

Co-authored-by: apk <apk@cognition.ai>
Co-Authored-By: bot_apk <apk@cognition.ai>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@devin-ai-integration devin-ai-integration bot added the development Tech debt, refactors, CI, tests, and other related work. label Apr 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development Tech debt, refactors, CI, tests, and other related work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant