Relay sync #764
Merged
Relay sync #764
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y sync - Add stream_events_from method to NostrClient for efficient event streaming - Update sync_event_type_to_multiple to use streaming instead of batch fetching - Streaming approach processes events as they arrive rather than waiting for all results - More efficient for large result sets and reduces memory usage
- When all relays are new (e.g., new account creation), there are no source relays to sync from - Previously would just log a warning and leave relays in Pending state - Now marks all pending relays as Completed immediately in this scenario - Prevents unnecessary sync attempts when there's nothing to sync
- The relays parameter already contains the user's configured relays - user_relays field was duplicating this information unnecessarily - Updated NostrClient::new() to remove user_relays parameter - Changed sync methods to use self.relays instead of self.user_relays - Updated all call sites to pass only one relay list
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements relay synchronization functionality that syncs historical events to newly added or re-added relays in a user's configuration. When a relay is added (or re-added after removal), the system now streams events from existing relays and publishes them to the new relay, allowing it to catch up to current state. Failed sync attempts are tracked in a retry queue with up to 10 retry attempts, and sync can resume from the last successful timestamp if interrupted.
Key Changes:
- Adds relay sync tracking with database persistence for sync status and retry queues
- Implements streaming-based event sync with configurable rate limiting (50ms delay between events)
- Integrates two new background jobs (relay_sync_job and relay_retry_sync_job) that run periodically
- Provides gap detection logic to differentiate between briefly offline relays and those that were removed/re-added
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/bcr-ebill-transport/src/relay_sync.rs | New module containing core sync logic including event streaming, progress tracking, and retry queue management |
| crates/bcr-ebill-transport/src/nostr.rs | Adds sync_relays() and retry_failed_syncs() methods to NostrClient with concurrency protection via AtomicBool |
| crates/bcr-ebill-transport/src/transport_service.rs | Delegates sync operations to underlying Nostr transport implementation |
| crates/bcr-ebill-persistence/src/nostr.rs | Extends NostrStoreApi trait (renamed from NostrContactStoreApi) with relay sync status and retry queue methods |
| crates/bcr-ebill-persistence/src/db/nostr_contact_store.rs | Implements new NostrStoreApi methods in SurrealNostrStore with two new database tables |
| crates/bcr-ebill-wasm/src/job.rs | Adds run_relay_sync_job() and run_relay_retry_sync_job() to periodic job runner |
| crates/bcr-ebill-api/src/constants.rs | Defines relay sync configuration constants (delay, max retries, batch size, gap threshold) |
| crates/bcr-ebill-api/src/service/transport_service/transport_client.rs | Adds sync_relays() and retry_failed_syncs() to TransportClientApi trait |
| crates/bcr-ebill-api/src/service/transport_service/transport.rs | Adds sync_relays() and retry_failed_syncs() to TransportServiceApi trait |
| crates/bcr-ebill-transport/src/test_utils.rs | Updates mock implementations with new sync methods |
| crates/bcr-ebill-api/src/tests/mod.rs | Updates mock implementations with new sync methods and imports |
| crates/bcr-ebill-persistence/src/lib.rs | Adds backwards compatibility aliases (NostrContactStoreApi → NostrStoreApi, SurrealNostrContactStore → SurrealNostrStore) |
| crates/bcr-ebill-api/src/lib.rs | Updates imports to use SurrealNostrStore while maintaining backwards compatibility |
| crates/bcr-ebill-transport/Cargo.toml | Adds futures dependency for join_all functionality |
| .gitignore | Adds /docs/plans to ignored paths |
- Optimize database queries in retry queue methods to use WHERE clauses instead of loading all records and filtering in memory - Add detailed documentation for gap detection logic - Improve error logging for failed retry queue additions (ERROR level for critical failures) - Add missing 'error' log import
tompro
assigned
codingpeanut157 and
stefanbitcr and unassigned
zupzup,
mtbitcr,
codingpeanut157 and
stefanbitcr
tompro
requested review from
codingpeanut157,
mtbitcr,
stefanbitcr and
zupzup
zupzup
approved these changes
Dec 17, 2025
Collaborator
zupzup
left a comment
zupzup
left a comment
There was a problem hiding this comment.
Looks very good, great job 👍
| // instead of resuming from last_synced_timestamp. This prevents syncing | ||
| // large amounts of historical data when a relay is deliberately removed and | ||
| // later re-added to the configuration. | ||
| let gap_threshold = Timestamp::now().inner() - RELAY_SYNC_GAP_THRESHOLD_SECONDS; |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📝 Description
Syncs a relay up to current state when added (or re-added) to the users set of relays. Retries failed relay message sync for up to 10x. Resumes sync after being interrupted.
Relates to #757
✅ Checklist
Please ensure the following tasks are completed before requesting a review:
cargo fmt.cargo clippy.🚀 Changes Made
📋 Review Guidelines
Please focus on the following while reviewing: