Skip to content

Keep a partly received pubsub message instead of reconnecting - #4613

Open
yamamoto7 wants to merge 2 commits into
redis:masterfrom
yamamoto7:fix/pubsub-partial-message-timeout
Open

yamamoto7 wants to merge 2 commits into
redis:masterfrom
yamamoto7:fix/pubsub-partial-message-timeout

Conversation

@yamamoto7

@yamamoto7 yamamoto7 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Description of change

Since #3982, the sync PubSub.get_message() passes its timeout to read_response(). If only part of a message has arrived, the read times out with TimeoutError. _execute() hands that to the retry layer, which calls _reconnect(): the connection is closed and resubscribed, and the partly read message (plus anything published during the reconnect) is lost. The caller just gets None.

Impact

  • Affects get_message() with the default timeout=0, run_in_thread() with the default sleep_time=0.0, and keyspace notifications built on them.
  • With a local Redis 8.10.3, 10 published 1 MB messages are all lost on 8.1.0 and master, and all received on 7.4.1.
  • Through a proxy that sends each reply in two writes 50 ms apart, 100-byte messages are lost too, with both timeout=0 and timeout=0.01.
  • Wait on the socket by default in run_in_thread workers #4610 (a 0.1 s default for run_in_thread()) narrows the window but does not close it.

Fix

For non-blocking reads, try_read() now returns None on TimeoutError instead of letting it reach the retry layer. The parser keeps what it has read, so the next call returns the whole message. With a push_handler_func, the TimeoutError may come from the handler instead, so that case still goes to the retry layer as on master. get_message() still never hangs, which was the goal of #3982, and blocking reads still raise. The asyncio client already works this way.

Tests

  • New test test_get_message_resumes_a_partly_received_message in TestPubSubTimeoutPropagation. It makes one read raise TimeoutError, as the parser does for an incomplete message, then checks that there is no reconnect and that the next call returns the message. It fails on master and passes with the fix. Resuming a partly read reply is already covered for each parser by test_connection_parse_response_resume, so the new test mocks read_response() and checks only the PubSub side; a version with a real 1 MB message failed on master only about half the time on Linux.
  • New test test_get_message_timeout_with_push_handler_reaches_retry_layer checks that with a push handler the timeout still reaches the retry layer. It fails if that case is suppressed too.
  • I ran the CI standalone and cluster pytest commands against redislabs/client-libs-test:8.10.3 (standalone and a 6-node cluster; no TLS, sentinel or replica), with and without hiredis, on RESP2 and RESP3. In all 8 runs the failing tests match master, apart from the new test, which passes. After the push handler change I reran tests/test_pubsub.py and tests/test_keyspace_notifications.py in the same 8 configurations and the fixed_client subset, with the same result. invoke linters passes.

Pull Request check-list

  • Do tests and lints pass with this change?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)? (no API change)
  • Is there an example added to the examples folder (if applicable)? (n/a)

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T06:29:18.971348Z 7b923e8 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 091781519c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread redis/client.py
return conn.read_response(
disconnect_on_error=False, push_request=True, timeout=read_timeout
)
except TimeoutError:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Propagate timeouts raised by RESP3 push handlers

When RESP3 is used and push_handler_func raises redis.exceptions.TimeoutError (for example, its own Redis I/O times out), Connection.read_response() routes that callback exception through this catch too, so nonblocking get_message() returns None rather than surfacing the handler failure. The Python RESP3 parser rewinds the frame on that exception, causing each poll to invoke the same failing handler again, while hiredis has already consumed the frame and silently drops the failure/message; the async PubSub path still propagates Redis TimeoutError. Narrow the suppression to a timeout produced by the bounded socket read.

AGENTS.md reference: AGENTS.md:L147-L148

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. A TimeoutError from a push_handler_func reaches try_read() the same way as a read timeout, so the two cannot be told apart there without relying on the implicit exception chain. With a push handler installed, the timeout now goes to the retry layer as on master; only reads without one return None. test_get_message_timeout_with_push_handler_reaches_retry_layer covers this.

One detail: on master, with the default retry, the handler failure is not surfaced either. The retry layer reconnects and the next read returns the resubscribe reply, so the failing message and the one after it are lost without an error. The regression was elsewhere: the pure-Python parser ran a handler that keeps failing again on every poll, so the subscriber spun without delivering anything. With this change the push handler case behaves as on master.

Comment thread tests/test_pubsub.py Outdated
returned by a later call, not dropped by a reconnect.
"""
p = r.pubsub()
p.subscribe("foo")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a test-specific pub/sub channel

In the cluster test profile, this unmarked test shares foo with many other cases. The existing wait_for_handled_message helper (lines 72-75) and adjacent test_timeout_with_pattern_subscribe explicitly account for an earlier PUBLISH foo arriving late over the cluster bus; if that happens after this subscription, can_read() observes the stale frame and the final b"hello" assertion fails even though resume behavior is correct. Use a channel unique to this test for both calls.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: the test now uses its own channel, partly-received.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant