feat(mailserver): stop reconciling history if we were online - #7673
Open
jrainville wants to merge 1 commit into
Open
feat(mailserver): stop reconciling history if we were online#7673jrainville wants to merge 1 commit into
jrainville wants to merge 1 commit into
Conversation
Fixes #7568 ## Summary - bound automatic history reconciliation to observed unreliable-delivery windows - introduce typed reconciliation events carrying fixed `From` and `To` bounds - retain, retry, and coalesce pending windows while preserving disjoint outages - prevent stable online periods from being queried during later reconciliation - use `mailserver_topics.last_request` as a monotonic “known complete through” cursor - checkpoint initialized topic cursors during reliable full-node connectivity - schedule bounded reconciliation for offline recovery, sleep/wake, and pause/resume - remove duplicate startup fetching and retain one cursor-based startup catch-up - wake retained work when mobile-network syncing is enabled - bound newest-community-description queries to the reconciliation window - preserve existing behavior for initial topic history, manual requests, and archive backfills ## Testing - added coverage for reliable and unreliable connectivity transitions - added overlapping/disjoint window coalescing and concurrent queue tests - added fixed retry-bound and blocked-work retention coverage - added cursor initialization, checkpointing, and monotonic update tests - added pause/resume and community-description window tests - ran focused Waku and protocol tests, including race coverage - ran messaging package compilation and `go vet` - ran protocol tests with `-tags lint` because `libsds.h` is unavailable in this checkout
✅ Companion PR Verified#21714 - chore(status-go): up status-go to stop reconciling when online |
There was a problem hiding this comment.
Pull request overview
This PR refines mailserver (store node) history reconciliation so that the protocol only queries history for periods where live delivery may have been missed (offline/unreliable windows), while tracking a monotonic per-topic “known complete through” cursor and checkpointing it during reliably connected full-node operation.
Changes:
- Introduces typed reconciliation windows (
HistoryReconcileWindow) and updates the Waku → Transport → Messaging API surface accordingly. - Reworks Messenger historic sync into a queued, bounded-window worker that coalesces/retries reconciliation requests without expanding into reliably-online time.
- Updates mailserver topic cursor persistence semantics (0 = uninitialized; monotonic advancement) and adds unit coverage around windowing, coalescing, pause/resume, and cursor checkpointing.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| services/mailservers/database.go | Makes topic upserts monotonic and adds AdvanceLastRequest to checkpoint initialized topic cursors. |
| services/mailservers/api_test.go | Extends DB tests for cursor advancement and non-regression behavior. |
| protocol/messenger.go | Adjusts pause/resume and connectivity handling to enqueue bounded reconciliation windows and checkpoint cursors. |
| protocol/messenger_settings.go | Wakes the historic sync worker when mobile-network syncing is enabled. |
| protocol/messenger_pause_play_test.go | Adds coverage ensuring resume enqueues a single bounded catch-up window. |
| protocol/messenger_mailserver.go | Adds window-aware historic sync options, bounds reconciliation queries, and makes cursor updates monotonic. |
| protocol/messenger_mailserver_window_test.go | New tests for window flooring, window translation, and concurrent/coalesced queue behavior. |
| protocol/messenger_mailserver_test.go | Adds coverage that community-description fetch respects reconciliation window bounds. |
| protocol/messenger_mailserver_cycle.go | Implements queued historic sync requests, coalescing, retry/defer semantics, and periodic cursor checkpointing. |
| pkg/messaging/waku/types/waku.go | Updates the Waku interface to emit typed reconcile windows and report delivery reliability. |
| pkg/messaging/waku/types/connection.go | Defines the HistoryReconcileWindow type. |
| pkg/messaging/waku/history_reconcile.go | Emits unreliable-delivery windows instead of a level-trigger signal; adds reliability tracking and HistoryDeliveryReliable. |
| pkg/messaging/waku/history_reconcile_test.go | Adds tests for window tracking bounds/disjointness and light-node reliability behavior. |
| pkg/messaging/waku/gowaku.go | Updates the reconcile channel type/buffering to carry windows. |
| pkg/messaging/types/connection_status.go | Re-exports HistoryReconcileWindow from the waku layer. |
| pkg/messaging/layers/transport/transport.go | Surfaces typed reconcile windows and delivery-reliability reporting from transport. |
| pkg/messaging/api.go | Exposes typed reconcile windows and delivery-reliability reporting on the messaging API. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
110
to
113
| select { | ||
| case w.historyReconcileNeeded <- struct{}{}: | ||
| default: // a signal is already pending; coalesce | ||
| case w.historyReconcileNeeded <- window: | ||
| case <-w.ctx.Done(): | ||
| } |
Member
Jenkins Builds
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #7673 +/- ##
===========================================
- Coverage 62.28% 62.25% -0.03%
===========================================
Files 868 868
Lines 120152 120461 +309
===========================================
+ Hits 74834 74997 +163
- Misses 37674 37811 +137
- Partials 7644 7653 +9
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7568
Status-app PR: status-im/status-app#21714
Summary
FromandToboundsmailserver_topics.last_requestas a monotonic “known complete through” cursorTesting
go vet-tags lintbecauselibsds.his unavailable in this checkout