fix(sentinel): disconnect on read errors by default - #4292
dylanpulver wants to merge 2 commits into
Conversation
SentinelManagedConnection.read_response() defaulted disconnect_on_error to False, so a connection whose reply was never read went back to the pool and the next command read the previous command's response. Every other read_response in the library defaults to True, including the async twin and the base Connection.read_response, whose except BaseException branch exists for this case (redis#1128, redis#2499). Both defaults were written in 35b7e09 for redis#2754, which took the value from the PubSub.parse_response call site. PubSub passes disconnect_on_error explicitly, so it is unaffected.
petyaslavova
left a comment
There was a problem hiding this comment.
Hey @dylanpulver, thank you for your contribution!
Confirmed, and the impact is wider than the description argues. Every read_response caller that omits disconnect_on_error is a strict command/response pair that needs True - Redis.parse_response, the on_connect handshake, _send_ping, re_auth. The three callers that genuinely want False (PubSub, the pending-push drain, _process_pending_invalidations) all pass it explicitly, so the current default has no beneficiary. It was never a design choice: it came from the #2754 hotfix, where the sync half was amended in last (4598d805d) and the author noted they were unfamiliar with Sentinel - which is why the async twin got True in the same commit.
Two things before merge:
- Please drop
test_default_disconnect_on_error_matches_base_connection. Asserting the default throughinspect.signaturerestates the source and would still pass if the flag were never forwarded; the four assertions you updated already pin it. - Please add a pipeline regression test.
_execute_transactionreads N+2 replies in a loop, and aBaseExceptionat reply k strands the rest:Retrydoes not treat it as retryable,execute()catches onlyException,reset()disconnects only when watching, andrelease()only whenshould_reconnect(). The connection goes back to_available_connectionsdesynced, which is a worse failure than the single-command case you reproduced.
The signature test restated the source and would have passed even if the flag were never forwarded; the four updated assertions already pin the default. The pipeline test covers what the single-command case does not. _execute_transaction reads MULTI, one reply per queued command, then EXEC, and a BaseException part way through that loop strands the rest: Retry does not treat it as retryable, execute() catches only Exception, reset() disconnects only while watching, and release() only when should_reconnect(). Without the fix the recycled connection answers a later ECHO with the QUEUED left over from the abandoned transaction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011M5uTyCU4WcNTsPvGrErDo
Description of change
SentinelManagedConnection.read_response()defaultsdisconnect_on_errortoFalse(redis/sentinel.py:73), so a connection whose reply was never read goes back to the pool and the next command gets the previous one's response. Against a real sentinel, after an interrupt at the socket read:Every other
read_responsedefaults toTrue, including the async twin atredis/asyncio/sentinel.py:76and baseConnection.read_response, whoseexcept BaseExceptionbranch exists for this case (#1128, #2499). Both were written in one commit,35b7e09afor #2754, which took the value from thePubSub.parse_responsecall site, whereFalseis passed explicitly.PubSubpasses the value explicitly, so it is unaffected.ReadOnlyErrortoo, since the parser returns error replies rather than raising them and they surface after the guarded block.Why no existing test caught it: they patch out
Connection.read_response, the only code that reads the flag, so they check the forwarded value and never its effect. Four assertions pinnedFalseand now pinTrue; the new tests assert socket state and the next reply.Ran
tests/standalone plus asyncio against a local sentinel setup, with a failure set identical to master's (the rest need redis-stack or TLS).invoke lintersclean on the pinned ruff 0.9.6. Reverting the one-line change fails three of the four new tests; the fourth guards the explicitFalseopt-out and passes either way.Prepared with Claude (Opus 5).
Pull Request check-list
Note
Medium Risk
Changes default connection lifecycle for all Sentinel-managed reads except explicit
disconnect_on_error=False; low blast radius but affects pooled connection reuse after read failures.Overview
Fixes stale replies on Sentinel master connections when a read is interrupted before the full response is consumed (e.g.
KeyboardInterruptat the socket read).SentinelManagedConnection.read_response()now defaultsdisconnect_on_errortoTrue, matching baseConnection, the async Sentinel connection, and the behavior intended in #1128. With the old default ofFalse, the socket stayed open and pooled connections could return the previous command’s reply on the next call (e.g.ping()→ wrong result,get()→PONG). Callers that need to keep the connection open (e.g. PubSub with an explicitdisconnect_on_error=False) are unchanged.Tests are updated to expect the new default and add sync/async coverage that checks socket teardown and the next reply, including an interrupted transactional pipeline recycled through
SentinelConnectionPool.Reviewed by Cursor Bugbot for commit ed39ad8. Bugbot is set up for automated code reviews on this repo. Configure here.