|
| 1 | +# Fix Flaky Skedda E2E Tests |
| 2 | + |
| 3 | +You must read, understand, and follow all instructions in `./README.md` when planning and implementing this feature. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +Several E2E tests in `tests/e2e/test_skedda_display.py` are flaky - they pass intermittently locally but fail in CI. These tests verify the Skedda calendar display functionality. |
| 8 | + |
| 9 | +## Affected Tests |
| 10 | + |
| 11 | +1. **`test_skedda_calendar_space_columns`** - Fails waiting for `.skedda-space-header` element |
| 12 | +2. **`test_skedda_calendar_shows_recurring_events_differently`** - Fails waiting for `.skedda-event` element |
| 13 | + |
| 14 | +## Observed Behavior |
| 15 | + |
| 16 | +### Local Runs |
| 17 | +- Tests pass when run individually: `poetry run -- nox -s test-e2e -- tests/e2e/test_skedda_display.py::TestSkeddaCalendarDisplayE2E::test_skedda_calendar_shows_recurring_events_differently` |
| 18 | +- Tests sometimes pass when run as part of full test suite |
| 19 | +- Tests sometimes fail locally with same error as CI |
| 20 | + |
| 21 | +### CI Failures |
| 22 | +From GitHub Actions logs: |
| 23 | + |
| 24 | +``` |
| 25 | +FAILED tests/e2e/test_skedda_display.py::TestSkeddaCalendarDisplayE2E::test_skedda_calendar_space_columns[chromium] |
| 26 | +AssertionError: Locator expected to be visible |
| 27 | +Actual value: None |
| 28 | +Error: element(s) not found |
| 29 | +Call log: |
| 30 | + - Expect "to_be_visible" with timeout 10000ms |
| 31 | + - waiting for locator(".skedda-space-header").first |
| 32 | +
|
| 33 | +FAILED tests/e2e/test_skedda_display.py::TestSkeddaCalendarDisplayE2E::test_skedda_calendar_shows_recurring_events_differently[chromium] |
| 34 | +AssertionError: Locator expected to be visible |
| 35 | +Actual value: None |
| 36 | +Error: element(s) not found |
| 37 | +Call log: |
| 38 | + - Expect "to_be_visible" with timeout 10000ms |
| 39 | + - waiting for locator(".skedda-event").first |
| 40 | +``` |
| 41 | + |
| 42 | +### Server Logs During Failure |
| 43 | +The server logs show: |
| 44 | +- Display page renders successfully (200 response) |
| 45 | +- SSE connection established |
| 46 | +- **Skedda data endpoint returns 404**: `GET /api/v1/display/skedda-test-display/skedda-data/1 -> 404` |
| 47 | + |
| 48 | +This 404 on the skedda-data endpoint appears to be the root cause - the calendar renders but has no event data to display. |
| 49 | + |
| 50 | +## Test File Location |
| 51 | + |
| 52 | +`tests/e2e/test_skedda_display.py` |
| 53 | + |
| 54 | +## Related Components |
| 55 | + |
| 56 | +- Skedda calendar JavaScript rendering in display view |
| 57 | +- `/api/v1/display/<name>/skedda-data/<feed_id>` endpoint |
| 58 | +- Test fixtures that set up Skedda slideshow data |
| 59 | + |
| 60 | +## Notes |
| 61 | + |
| 62 | +- Other Skedda tests in the same file pass consistently: |
| 63 | + - `test_skedda_calendar_renders_grid` |
| 64 | + - `test_skedda_calendar_displays_events` |
| 65 | + - `test_skedda_calendar_loading_state` |
| 66 | +- The flaky tests may have timing/race condition issues or fixture setup problems specific to CI environment |
| 67 | + |
| 68 | +--- |
| 69 | + |
| 70 | +## Implementation Plan |
| 71 | + |
| 72 | +### Root Cause Analysis |
| 73 | + |
| 74 | +The tests fail because of **SQLite cross-process visibility issues**. The test data flow is: |
| 75 | + |
| 76 | +1. `_create_session_test_data()` creates test data (display, slideshow, skedda item, events) |
| 77 | +2. WAL checkpoint is performed to sync data to disk |
| 78 | +3. `live_server` fixture forks a subprocess running Flask |
| 79 | +4. Test navigates to display page - renders successfully because `SlideshowItem.query.filter_by(slideshow_id=X)` through relationship works |
| 80 | +5. JavaScript calls `/api/v1/display/skedda-test-display/skedda-data/{item_id}` |
| 81 | +6. API endpoint queries `SlideshowItem.query.filter_by(id=item_id).first()` - returns `None` → 404 |
| 82 | + |
| 83 | +The issue is that `SlideshowItem.query.filter_by(id=item_id)` in the forked server process sometimes cannot see data created before the fork, even with WAL checkpoint. This is a known SQLite cross-process visibility issue that manifests as a race condition - more likely to fail in CI due to timing differences. |
| 84 | + |
| 85 | +**Why some tests pass consistently:** |
| 86 | +- `test_skedda_calendar_renders_grid` - checks structural elements (calendar grid) that render even if API fails |
| 87 | +- `test_skedda_calendar_loading_state` - only checks loading state transitions, not event data |
| 88 | +- The failing tests specifically require event data from the API endpoint |
| 89 | + |
| 90 | +### Solution |
| 91 | + |
| 92 | +**Primary Fix:** Ensure complete database session cleanup and reconnection before the live_server fork to guarantee cross-process visibility. |
| 93 | + |
| 94 | +### Milestones |
| 95 | + |
| 96 | +#### Milestone 1: Improve Database Synchronization (FST-1) |
| 97 | + |
| 98 | +**Task 1.1:** Modify `tests/e2e/conftest.py` to ensure complete database session cleanup after creating test data: |
| 99 | +- After the WAL checkpoint, close the session completely with `db.session.close()` |
| 100 | +- Remove the session from the scoped session registry with `db.session.remove()` |
| 101 | +- Dispose of the engine to close all connections with `db.engine.dispose()` |
| 102 | +- This ensures the forked server process creates fresh connections that will see the committed data |
| 103 | + |
| 104 | +**Task 1.2:** Run E2E tests locally multiple times to verify fix: |
| 105 | +- Run `poetry run -- nox -s test-e2e -- tests/e2e/test_skedda_display.py` at least 5 times |
| 106 | +- All runs should pass consistently |
| 107 | + |
| 108 | +#### Milestone 2: Acceptance Criteria (FST-2) |
| 109 | + |
| 110 | +**Task 2.1:** Ensure all nox sessions pass: |
| 111 | +- Run `poetry run -- nox -s test-e2e` - all tests pass |
| 112 | +- Run `poetry run -- nox -s test-3.14` - unit tests still pass |
| 113 | +- Run `poetry run -- nox -s lint` - no lint errors |
| 114 | +- Run `poetry run -- nox -s type_check` - no type errors |
| 115 | + |
| 116 | +**Task 2.2:** Verify CI passes: |
| 117 | +- Push branch and create PR |
| 118 | +- All GitHub Actions checks must pass |
| 119 | + |
| 120 | +**Task 2.3:** Move feature file to completed: |
| 121 | +- Move `docs/features/fix-flaky-skedda-e2e-tests.md` to `docs/features/completed/` |
| 122 | + |
| 123 | +### Progress |
| 124 | + |
| 125 | +- [x] Milestone 1: Improve Database Synchronization (FST-1) |
| 126 | + - [x] Task 1.1: Modify conftest.py for complete session cleanup |
| 127 | + - [x] Task 1.2: Verify fix with multiple local test runs (10/10 passed) |
| 128 | +- [x] Milestone 2: Acceptance Criteria (FST-2) |
| 129 | + - [x] Task 2.1: All nox sessions pass |
| 130 | + - [x] Task 2.2: CI passes (PR #9) |
| 131 | + - [x] Task 2.3: Move feature file to completed |
| 132 | + |
| 133 | +### Implementation Notes |
| 134 | + |
| 135 | +The original plan needed to be expanded during implementation. Through investigation, we discovered: |
| 136 | + |
| 137 | +1. The root cause was more nuanced: `Display.query.filter_by(name=X).first()` would return `None` even when `Display.query.all()` showed the record existed in the database. This is a SQLAlchemy/SQLite session state issue. |
| 138 | + |
| 139 | +2. The fix required changes in two places: |
| 140 | + - **tests/e2e/conftest.py**: Use NullPool, TRUNCATE WAL checkpoint mode, and session cleanup |
| 141 | + - **kiosk_show_replacement/api/v1.py**: Add `db.session.expire_all()` and fallback logic in `get_display_skedda_data()` to handle cases where filter_by queries fail but .all() queries succeed |
0 commit comments