Add ASM (Atomic Slot Migration) cluster maintenance-notification scenario tests - #4158
Open
kiryazovi-redis wants to merge 5 commits into
Open
Add ASM (Atomic Slot Migration) cluster maintenance-notification scenario tests#4158kiryazovi-redis wants to merge 5 commits into
kiryazovi-redis wants to merge 5 commits into
Conversation
petyaslavova
requested changes
Jul 2, 2026
petyaslavova
force-pushed
the
feat_add_sch_async
branch
2 times, most recently
from
July 9, 2026 11:32
4b18f44 to
5602af3
Compare
kiryazovi-redis
force-pushed
the
feat/asm-cluster-scenario-tests
branch
from
July 10, 2026 19:32
4b85297 to
aa38fc4
Compare
kiryazovi-redis
force-pushed
the
feat/asm-cluster-scenario-tests
branch
from
July 10, 2026 19:37
aa38fc4 to
e7f6396
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
Reviewed by Cursor Bugbot for commit 51a8fd6. Configure here.
…ove) + generic node_add The FI's SLOT_MIGRATE_ASM toggle routes the generic cluster tests through the ASM `reshard` trigger. ASM scale (add/remove) is a cascade of SMIGRATING/SMIGRATED, so the client reconciles its node cache over several notifications after the op completes: - Add a `_wait_for_node_cache_delta` settle-poll helper; use it in node_remove and the new node_add test before the node-delta assertion (single-step migrate/failover return immediately). Relax node_remove reconnect count to >= 1 (a cascade may touch more than one). - Add generic `test_notification_handling_with_node_add` (+1) mirroring node_remove. - Skip `test_new_connections_receive_last_smigrating_smigrated_notification` for reshard add/remove (catching the in-flight notification on a new connection is a single-window property; ASM scale is a multi-window cascade). slot-shuffle keeps a single window.
…on catch The new-connection-catches-SMIGRATING assertion needs a maintenance window that stays open long enough to open a fresh connection mid-flight. ASM `reshard` slot-shuffle (single migrate_slots over a small slot range) closes ~1s after SMIGRATING (SMIGRATING t+0 -> SMIGRATED t+1.2s in CI), so a connection opened after the first wait/assert loop legitimately observes MaintenanceState.NONE. This is the same "ends too fast" class already skipped for slot_shuffle+failover. Existing-connection SMIGRATING/SMIGRATED handling for slot-shuffle+reshard stays covered by the handling and cmd-execution tests; new-connection catch is still exercised via slot_shuffle+migrate and remove_add+reshard (longer windows).
Guard against env0 reshard-latency flakiness. The client's patience was smaller
than what the fault injector allows a reshard to take:
FI: slot-shuffle migrate_slots up to 300s, ASM scale up to 600s (blocking).
client (before): trigger op-wait 120s, SMIGRATING waits 60s, SMIGRATED 40s,
per-test cap 300s.
So a slow-but-successful reshard was counted as a failure - and when the trigger
thread gave up at 120s the command-execution workers were orphaned and their
join() hung to the 300s pytest cap.
Raise the ceilings to cover the FI's real budget (raising a timeout only affects
the failure path, so it is free on green runs):
- SLOT_SHUFFLE_TIMEOUT 120 -> 300, SMIGRATED_TIMEOUT 40 -> 120
- trigger get_operation_result wait -> RESHARD_OP_TIMEOUT (600), so the trigger
thread never gives up before the FI finishes
- per-test cap -> RESHARD_TEST_TIMEOUT (660) on the OSS reshard tests only
(standalone/non-reshard tests keep 300s)
…ut guard - Pytest cap vs drain (Medium): RESHARD_TEST_TIMEOUT is now RESHARD_OP_TIMEOUT + SMIGRATED_TIMEOUT + margin (900s), so a slow ASM scale that uses the full op-wait and then drains the node cache post-join is not killed mid-reconciliation. - Shared trigger timeout leaking to standalone (Low): revert _trigger_effect's default op-wait to a modest DEFAULT_TRIGGER_OP_TIMEOUT (180s, under the standalone tests' 300s cap); the 6 OSS reshard tests now pass RESHARD_OP_TIMEOUT (600s) explicitly. - Drain helper silent timeout (Medium): _wait_for_node_cache_delta now pytest.fail()s on timeout instead of returning silently, so a failed reconciliation surfaces clearly instead of as a misleading node-count assert. - node_add missing invariants (Low): assert cache size unchanged + all initial node keys present during SMIGRATING, mirroring test_notification_handling_with_node_remove.
kiryazovi-redis
force-pushed
the
feat/asm-cluster-scenario-tests
branch
from
July 13, 2026 15:58
96889a7 to
d1fd9b3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description of change
Cluster client maintenance-notification scenario tests for the ASM (Atomic Slot Migration) path. All ASM logic lives in the fault injector behind its
SLOT_MIGRATE_ASMtoggle, so the client stays generic (no ASM-specific code): when the FI runs in ASM mode, the existing effect-trigger cluster tests exercise the FI'sreshardtrigger and assertSMIGRATING→MaintenanceState.MAINTENANCE→SMIGRATEDwith node-cache reconciliation.This PR is therefore small and test-only.
Changes
test_notification_handling_with_node_add(+1 node), the mirror oftest_notification_handling_with_node_remove._wait_for_node_cache_deltasettle-poll on the cluster test base: ASM scale (add/remove) is a cascade ofSMIGRATING/SMIGRATED(the decision engine moves slots across shards), so the client reconciles its node cache over several notifications after the FI op completes — drain + poll until the ±1 delta before asserting.node_removenow joins the trigger thread + waits for −1 first; reconnect count relaxed to>= 1(a cascade can mark more than one).test_new_connections_receive_last_smigrating_smigrated_notificationfor allreshardeffects (slot-shuffle, add, remove): "a new connection catches the in-flightSMIGRATING" needs a maintenance window that stays open long enough to open a fresh connection mid-flight. ASM add/remove is a multi-window cascade, and a single ASM slot-shuffle over a small slot range closes (SMIGRATED) ~1s afterSMIGRATING— the window shuts before a new connection can be opened. Same "ends too fast" class already skipped forslot_shuffle+failover. Existing-connection handling for slot-shuffle stays covered by the handling and cmd-execution tests.SMIGRATINGwaits 60s,SMIGRATED40s, per-test cap 300s), so a slow reshard flaked and the trigger thread could give up early and leave the command-execution workers hanging. Now: triggerget_operation_resultwaitsRESHARD_OP_TIMEOUT(600s),SLOT_SHUFFLE_TIMEOUT120→300,SMIGRATED_TIMEOUT40→120, and the OSS reshard tests use aRESHARD_TEST_TIMEOUT(660s) cap (standalone/non-reshard tests keep 300s).Requirements
Needs the fault injector with the
reshardtrigger +SLOT_MIGRATE_ASMtoggle. CI runs the generic tests against an ASM-mode fault injector (asm=true).Pull Request check-list
new_connections[slot-shuffle-reshard]skipped as designed.asm=true); green end-to-end.History (condensed)
Earlier revisions added an
AsmTriggersenum,include_asmthreading, and a dedicatedTestClusterClientPushNotificationsWithAsmEffectTriggerwith a node-cache "drain until settled" loop. Per review, all ASM logic moved into the FI behind the toggle, so those client-side constructs were removed and the change collapsed to the small, generic diff above. The branch was rebased onto the currentfeat_add_sch_asyncbase (the earlier huge diff was stale-base divergence, not real change). Follow-up commits then made the generic tests robust to real ASM reshard latency (new-connection skip + FI-budget timeouts) after live ASM CI runs.Note
Low Risk
Test-only changes in scenario integration tests; no production client or library behavior is modified.
Overview
Extends cluster maintenance-notification scenario tests for ASM (
reshard) fault-injector runs without changing redis-py client code.Timeouts are aligned with the FI reshard budget (
RESHARD_OP_TIMEOUT600s, longerSMIGRATED/ slot-shuffle waits,RESHARD_TEST_TIMEOUTon OSS reshard tests) so slow-but-successful reshards are not treated as failures and trigger threads do not exit early and leave workers hanging. Non-reshard triggers keep a shorter default op-wait viaDEFAULT_TRIGGER_OP_TIMEOUT.Adds
_wait_for_node_cache_deltato drain push notifications and poll until the cluster client’s node cache reaches the expected ±1 change after the FI op completes—needed because ASM scale add/remove is a cascade ofSMIGRATING/SMIGRATEDevents.test_notification_handling_with_node_addmirrors node-remove for a +1 cache delta; node-remove joins the trigger first, waits for −1, and relaxes “marked for reconnect” to>= 1.Skips
test_new_connections_receive_last_smigrating_smigrated_notificationwhentrigger == "reshard"for add/remove/slot-shuffle (maintenance windows too short to open a fresh connection mid-flight). Reshard effect-trigger tests passRESHARD_OP_TIMEOUTinto_trigger_effect.Reviewed by Cursor Bugbot for commit 851a112. Bugbot is set up for automated code reviews on this repo. Configure here.