fix: retry transient transport errors on a fresh async connection#77
Conversation
httpx reuses pooled keep-alive connections; when the server has already closed
one, the next async request fails immediately with a transient transport error
— commonly `ReadError('')` (empty message), also `RemoteProtocolError`,
`ConnectError`, pool/connect timeouts. `async_execute` previously retried only
on a closed event loop and surfaced everything else, so a single stale socket
became a hard failure (after 3.8.3, visible as `[{'message': "ReadError('')"}]`).
It now drops the stale client and retries once on a fresh connection for those
transient transport errors. `ReadTimeout` is intentionally excluded — a genuinely
slow request would just time out again. New `_should_retry_on_fresh_connection`
helper + `TRANSIENT_TRANSPORT_ERRORS`; sync `execute` already had this behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uhj6eA8d9jcAzzHuzqtPod
There was a problem hiding this comment.
Overall Assessment
This PR adds retry logic to async_execute for transient transport errors (stale keep-alive sockets manifesting as ReadError(''), RemoteProtocolError, etc.), mirroring the existing sync execute behavior. The implementation is sound: a new _should_retry_on_fresh_connection predicate classifies errors, ReadTimeout is correctly excluded to avoid doubling latency on slow requests, and the retry path drops the stale client before creating a fresh one. No blocking bugs found.
Findings
No actionable findings.
Notes
- Test coverage is strong: the new
test_transient_transport_retry.pyprovides parametrized predicate coverage plus an integration test that verifies the retry path (firstReadErrortriggers_drop_async_client+ fresh client, second call succeeds) and a negative test confirmingReadTimeoutis not retried. - The docstring on
_should_retry_on_fresh_connectionand the module-levelTRANSIENT_TRANSPORT_ERRORScomment clearly document the design rationale (whyReadTimeoutis excluded, whyNetworkError/RemoteProtocolErrorare included). - The PR follows the repo's TDD and release conventions (test added, version bumped, CHANGELOG entry written).
There was a problem hiding this comment.
Pull request overview
This PR hardens GraphQLClient.async_execute against stale pooled HTTP connections by retrying exactly once on a freshly-created httpx.AsyncClient when specific transient transport errors occur, aligning the async path with the existing sync behavior.
Changes:
- Added
TRANSIENT_TRANSPORT_ERRORSand_should_retry_on_fresh_connection()to centralize the async retry predicate. - Updated
async_executeto drop the stale async client and retry once on a new client for transient transport failures (excludingReadTimeout). - Added a new test module covering the retry predicate and async retry behavior; bumped version to 3.8.4 and documented the change in
CHANGELOG.md.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
pygqlc/GraphQLClient.py |
Adds transient transport error classification + a helper predicate, and retries async_execute once after dropping the async client. |
tests/pygqlc/gql_client/test_transient_transport_retry.py |
New tests for retry predicate and async retry/no-retry behavior. |
pygqlc/__version__.py |
Bumps package version to 3.8.4. |
CHANGELOG.md |
Documents the async fresh-connection retry behavior and rationale. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Retry once on a fresh connection when the failure is a closed event | ||
| # loop or a transient transport error (e.g. a stale keep-alive socket | ||
| # the server already closed, surfacing as ReadError/RemoteProtocolError). | ||
| # Anything else (real timeouts, genuine network outages) re-raises. |
| payload = {"data": {"createBulkThings": {"successful": True}}} | ||
| client = AsyncMock() | ||
| # First post hits a stale socket (ReadError); the retry on a fresh client succeeds. | ||
| client.post = AsyncMock(side_effect=[httpx.ReadError(""), _fake_response(payload)]) | ||
|
|
||
| monkeypatch.setattr(gql_env, "_get_async_client", AsyncMock(return_value=client)) | ||
| dropped = AsyncMock() | ||
| monkeypatch.setattr(gql_env, "_drop_async_client", dropped) | ||
|
|
||
| result = await gql_env.async_execute("query { things { id } }") | ||
|
|
||
| assert result == payload | ||
| assert client.post.call_count == 2 # original + one retry | ||
| dropped.assert_awaited_once() # stale client was dropped before retrying |
…nts/docs Address Copilot review on #77: - async_execute retry test now uses two distinct clients (stale -> fresh) so it actually asserts a brand-new connection is used on retry. - Fix misleading inline comment (the block does retry ConnectTimeout/ConnectError); shortened the comments, helper docstring, and CHANGELOG entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uhj6eA8d9jcAzzHuzqtPod
|
Addressed @Copilot's review:
|
There was a problem hiding this comment.
Overall Assessment
This PR adds retry-on-fresh-connection logic to async_execute for transient transport errors (httpx.NetworkError, RemoteProtocolError, PoolTimeout, ConnectTimeout) plus the existing "Event loop is closed" case. The sync execute path already retried on any exception; the async path now does so selectively. Implementation and tests are sound; no blocking bugs found.
Findings
No actionable findings.
Notes
- The predicate
_should_retry_on_fresh_connectionis extracted as a static method with a clear docstring, and theTRANSIENT_TRANSPORT_ERRORStuple is documented at the module level. ReadTimeoutis correctly excluded from the retry set (it is caught byexcept httpx.RequestErrorbut_should_retry...returns False, causing a re-raise);ConnectTimeout/PoolTimeoutare included because they indicate connection establishment failure rather than request slowness.- The new test file provides parametrized coverage of the predicate plus an integration test that verifies the stale→fresh client swap and a negative test for
ReadTimeout. All tests are fast and hermetic (no real sockets). - The sync
executeretries on a bareException; the async change narrows this to a well-defined set while preserving the one-shot retry pattern.
Problem
After 3.8.3 made transport failures legible, a recurring
ReadError('')surfaced onUPSERT_KPI_DATA(femsaKPI_ENGINE_ON_DEMAND) aserrors=[{'message': "ReadError('')"}]. An empty-messageReadErroris the classic signature of a stale keep-alive connection: httpx reuses a pooled connection the server has already closed, so the next request fails instantly.async_executeonly retried on a closed event loop and surfaced everything else, so one stale socket = one hard failure.Fix
async_executenow drops the stale client and retries once on a fresh connection for transient transport errors:ReadTimeoutis intentionally excluded — a genuinely slow request would just time out again and double the latency. The syncexecutepath already retried once on a fresh client, so only the async path changed.Tests
tests/pygqlc/gql_client/test_transient_transport_retry.py: parametrized coverage of the retry predicate (ReadError/RemoteProtocolError/ConnectTimeout/PoolTimeout/ConnectError/event-loop → retry; ReadTimeout/generic → no), plus an integration test thatasync_executeretries on a fresh connection after a first-callReadErrorand a negative test thatReadTimeoutis not retried. 10 pass; existing timeout test still green; no new lint.Context
This is the transport-layer root cause behind the KPI
ReadErrorfailures. The companion fix repairs the dead@retry(theretryinglibrary is a no-op onasync def) in thekpispackage.🤖 Generated with Claude Code