Skip to content

Unexpected Connection closed by server behavior on redis==8.1.0 #4252

Description

@barela14
[BUG] asyncio: ConnectionError: Connection closed by server. surfaces to the caller when a pooled connection is closed while idle (regression in 8.1.0)
Version
redis-py: 8.1.0 (works on 8.0.1)
Python: 3.13
Server: Redis (docker redis:latest), standalone, RESP3
Platform: Linux container / macOS
hiredis: reproduced both with and without hiredis installed
Description

With redis.asyncio and a ConnectionPool configured with health_check_interval and a Retry, a connection that the server closed while it sat idle in the pool causes the next command to raise ConnectionError: Connection closed by server. to the application, instead of being transparently retried on a fresh connection.

On 8.0.1 the identical script recovers and returns the value. On 8.1.0 it raises. No application-side change; only the redis-py version differs.

The failure surfaces from the PING inside check_health(): the health check detects the dead socket (correctly), but the resulting ConnectionError propagates out of Redis.execute_command rather than being retried.

Client config

import redis.asyncio as redis
from redis.backoff import ExponentialBackoff
from redis.retry import Retry

pool = redis.ConnectionPool.from_url(
    url=f"redis://{host}:{port}/0",
    encoding="utf-8",
    decode_responses=True,
    health_check_interval=10,
    retry=Retry(ExponentialBackoff(), 3),
    retry_on_error=[redis.ConnectionError, redis.TimeoutError, RuntimeError],
    socket_connect_timeout=5,
    retry_on_timeout=True,
    socket_keepalive=True,
)
client = redis.Redis.from_pool(connection_pool=pool)
I verified the retry object actually reaches the connection — on both versions conn.retry.get_retries() == 3 and the supported errors are (TimeoutError, ConnectionError, RuntimeError, asyncio.TimeoutError). So this is not a case of the retry policy being dropped at construction.

Reproduction
Standalone script. It kills the pooled connection server-side with CLIENT KILL, which is what an idle-timeout / server-side reap looks like to the client. next_health_check = 0 just forces the health check to be due immediately, so you don't have to wait out health_check_interval.


import asyncio
import redis
import redis.asyncio as aredis
from redis.backoff import ExponentialBackoff
from redis.retry import Retry


async def client_id(conn):
    await conn.send_command("CLIENT", "ID", check_health=False)
    return await conn.read_response()


async def main():
    pool = aredis.ConnectionPool.from_url(
        url="redis://localhost:6379/0",
        decode_responses=True,
        health_check_interval=10,
        retry=Retry(ExponentialBackoff(), 3),
        retry_on_error=[aredis.ConnectionError, aredis.TimeoutError, RuntimeError],
        socket_connect_timeout=5,
        retry_on_timeout=True,
        socket_keepalive=True,
    )
    c = aredis.Redis.from_pool(connection_pool=pool)

    await c.hset("h", "k", "v")
    print("before:", await c.hget(name="h", key="k"))

    # The connection is idle in the pool. Make the health check due, then have
    # the server close it — same as an idle timeout / server-side reap.
    conn = pool._available_connections[0]
    conn.next_health_check = 0

    admin = aredis.Redis.from_url("redis://localhost:6379/0", decode_responses=True)
    await admin.execute_command("CLIENT", "KILL", "ID", str(await client_id(conn)))
    await admin.aclose()

    try:
        print("after: ", await c.hget(name="h", key="k"))
        print("RESULT: RECOVERED")
    except Exception as e:
        print(f"RESULT: RAISED {type(e).__name__}: {e}")

    await c.aclose()


print("redis-py", redis.__version__)
asyncio.run(main())
Actual (8.1.0)

redis-py 8.1.0
before: v
RESULT: RAISED ConnectionError: Connection closed by server.
Expected — and what 8.0.1 actually does

redis-py 8.0.1
before: v
after:  v
RESULT: RECOVERED
Traceback (from the real application)

    customer = await self.redis_client.hget(name=hset_name, key=hashed_api_key)
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/client.py", line 946, in execute_command
    result = await conn.retry.call_with_retry(
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/client.py", line 870, in _send_command_parse_response
    await conn.send_command(*args)
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/connection.py", line 1214, in send_command
    await self.send_packed_command(
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/connection.py", line 1177, in send_packed_command
    await self.check_health()
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/connection.py", line 1163, in check_health
    await self.retry.call_with_retry(
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/connection.py", line 1148, in _send_ping
    if str_if_bytes(await self.read_response()) != "PONG":
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/connection.py", line 1286, in read_response
    response = await self._read_response_from_parser(
  File "/usr/local/lib/python3.13/site-packages/redis/asyncio/connection.py", line 1335, in _read_response_from_parser
    return await self._parser.read_response(
  File "/usr/local/lib/python3.13/site-packages/redis/_parsers/resp3.py", line 185, in read_response
    response = await self._read_response(
  File "/usr/local/lib/python3.13/site-packages/redis/_parsers/resp3.py", line 197, in _read_response
    raw = await self._readline()
  File "/usr/local/lib/python3.13/site-packages/redis/_parsers/base.py", line 589, in _readline
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
Notes from narrowing it down
A few things I checked so they can be ruled out:

Not the retry configuration. conn.retry carries retries=3 and includes ConnectionError in its supported errors on both versions.
Not retry.py. redis/retry.py and redis/asyncio/retry.py are byte-identical between 8.0.1 and 8.1.0.
Not the parser. My first comparison was confounded by hiredis being absent in the 8.1.0 venv (so it picked _AsyncRESP3Parser vs _AsyncHiredisParser). After pinning hiredis==3.3.1 in both, the parser matches and 8.1.0 still fails — so the parser is not the cause.
Not should_reconnect(). Forcing it to False on 8.1.0 does not change the outcome.
Every function in the traceback (check_health, _send_ping, _ping_failed, send_packed_command) is byte-identical between the two versions, and calling conn.check_health() directly raises the same ConnectionError on both. So the difference is not in raising the error — it is in whether the caller retries it.
Instrumenting Retry.call_with_retry shows the difference. On 8.0.1 the failing command drives repeated health checks and recovers; on 8.1.0 _ping_failed is never invoked and the error escapes on the first failure:


=== 8.0.1 ===                        === 8.1.0 ===
disconnect(...)                      check_health
check_health                         disconnect(...)
check_health                         -> ConnectionError raised
check_health
check_health
-> recovered
So on 8.1.0 the ConnectionError raised by the health-check PING is escaping the retry loop in execute_command instead of triggering a reconnect-and-retry.


Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions