PHOENIX-7988 Idle co-active RegionServer stays wedged in STORE_AND_FORWARD, pinning the group at ACTIVE_NOT_IN_SYNC - #2600
Conversation
…RWARD, pinning the group at ACTIVE_NOT_IN_SYNC A co-active RS in STORE_AND_FORWARD promotes back to SYNC_AND_FORWARD via processNoMoreRoundsLeft only when the caught-up guard passes. That guard scanned the next round's shard, which is a shared directory holding every co-active RS's live OPENFORWRITE rotation writer, so an idle or out-claimed RS never saw it empty and stayed pinned in STORE_AND_FORWARD, wedging the group at ACTIVE_NOT_IN_SYNC indefinitely. Split the guard: the RS's own mode flip is gated only on an empty in-progress directory (the per-RS forward-health signal), while the shared in-sync status claim keeps the full caught-up guard. The mode flip is self-validating since SyncAndForwardModeImpl.onEnter must reach the peer, so an optimistic promotion against a dead peer bounces back to STORE_AND_FORWARD. Adds a VisibleForTesting tracker-injecting constructor and three focused tests covering the wedge fix, the peer-down backlog case (no promotion), and the fully-caught-up case.
There was a problem hiding this comment.
Pull request overview
Fixes idle co-active RegionServers remaining in STORE_AND_FORWARD because shared live rotation files prevent the caught-up guard from passing.
Changes:
- Separates local mode promotion from the shared in-sync claim.
- Adds tracker injection for focused testing.
- Adds tests for backlog, catch-up, and peer failure scenarios.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
phoenix-core-server/.../ReplicationLogDiscoveryForwarder.java |
Splits mode-promotion and group-sync guards. |
phoenix-core/.../ReplicationLogDiscoveryForwarderTest.java |
Tests promotion, backlog, catch-up, and peer failure behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| LOG.info("Processed all the replication log files for {}", logGroup); | ||
| if (syncUpdateTS <= EnvironmentEdgeManager.currentTimeMillis()) { | ||
| try { | ||
| long waitTime = logGroup.setHAGroupStatusToSync(); |
There was a problem hiding this comment.
It's pre-existing, not introduced in this PR. It also self-heals. When onEnter bounces back, StoreAndForwardModeImpl.onEnter restarts the periodic re-assertion task that rewrites ACTIVE_NOT_IN_SYNC every 0.7 * zkSessionTimeout, so any transient false ACTIVE_IN_SYNC is overwritten within one interval.
The residual risk is a narrow, self-correcting window (≤ ~0.7 * zkTimeout) advertising ACTIVE_IN_SYNC against a dead peer.
A co-active RS in STORE_AND_FORWARD promotes back to SYNC_AND_FORWARD via processNoMoreRoundsLeft only when the caught-up guard passes. That guard scanned the next round's shard, which is a shared directory holding every co-active RS's live OPENFORWRITE rotation writer, so an idle or out-claimed RS never saw it empty and stayed pinned in STORE_AND_FORWARD, wedging the group at ACTIVE_NOT_IN_SYNC indefinitely.
Split the guard: the RS's own mode flip is gated only on an empty in-progress directory (the per-RS forward-health signal), while the shared in-sync status claim keeps the full caught-up guard. The mode flip is self-validating since SyncAndForwardModeImpl.onEnter must reach the peer, so an optimistic promotion against a dead peer bounces back to STORE_AND_FORWARD.
Adds a VisibleForTesting tracker-injecting constructor and three focused tests covering the wedge fix, the peer-down backlog case (no promotion), and the fully-caught-up case.