feat(media-store): cloud media storage backend, phase 1 (foundation)#550
Merged
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
There was a problem hiding this comment.
Pull request overview
Implements Phase 1 (foundation) of the Media Store feature: adds a user-owned, S3-compatible cloud “canonical home” for media bytes, with synced metadata stamps (content_hash, upload confirmations) and a local, content-addressed cache + upload queue to support cross-device media display and resilience.
Changes:
- Adds Media Store schema foundations:
mediacontent identity + upload stamps and new syncedmedia_storesdescriptor table, wired into the sync engine. - Introduces a file-based
MediaObjectStoreabstraction with an S3 adapter, plus store marker identity (smv1/store.json) and key derivation (smv1/objects/...). - Adds per-device local-cache DB v2 tables for transfer queue + cache index, upload pipeline/worker, store-backed resolver fallback, and a new Settings → Media Storage page + localized strings.
Reviewed changes
Copilot reviewed 71 out of 72 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| test/helpers/in_memory_media_object_store.dart | In-memory MediaObjectStore test double with failure injection |
| test/features/media/presentation/media_item_view_store_fallback_test.dart | Widget test: MediaItemView falls back to store when native source unavailable |
| test/features/media/data/media_store_resolver_test.dart | Unit tests for store-backed resolver (cache + hash verification) |
| test/features/media/data/media_repository_store_stamps_test.dart | Tests for new media repository stamp fields round-tripping |
| test/features/media_store/support/fake_local_file_resolver.dart | Test resolver for localFile sources in media-store tests |
| test/features/media_store/media_upload_pipeline_test.dart | Pipeline tests: upload, dedup, replay, failure scheduling, eligibility |
| test/features/media_store/media_transfer_queue_repository_test.dart | Queue repository tests + local cache DB v2 migration coverage |
| test/features/media_store/media_store_service_test.dart | MediaStoreService tests for connect/test/disconnect flows |
| test/features/media_store/media_store_end_to_end_test.dart | E2E test: device A upload → device B resolve via shared store |
| test/features/media_store/media_storage_page_test.dart | Widget test for Media Storage settings page form/connect behavior |
| test/features/media_store/media_cache_store_test.dart | Cache store tests: put/get, LRU eviction, self-heal, pool separation |
| test/core/services/sync/sync_media_stores_test.dart | Sync tests for new mediaStores replication plumbing |
| test/core/services/media_store/store_marker_test.dart | Store marker ensure/read/corrupt-json behavior tests |
| test/core/services/media_store/store_keys_test.dart | StoreKeys + streamed SHA-256 tests |
| test/core/services/media_store/s3_media_object_store_test.dart | S3 adapter tests via MockClient |
| test/core/services/media_store/media_store_credentials_test.dart | Keychain credential store + attach-state round-trip tests |
| test/core/services/media_store/media_object_store_contract.dart | Behavioral contract tests for MediaObjectStore implementations |
| test/core/services/media_store/media_object_store_contract_test.dart | Contract test runner for InMemoryMediaObjectStore |
| test/core/database/migration_v103_media_store_test.dart | Main DB v103 migration + backstop coverage |
| lib/l10n/arb/app_zh.arb | Adds Media Storage strings (zh) |
| lib/l10n/arb/app_pt.arb | Adds Media Storage strings (pt) |
| lib/l10n/arb/app_nl.arb | Adds Media Storage strings (nl) |
| lib/l10n/arb/app_localizations.dart | Generated localization API: new Media Storage keys |
| lib/l10n/arb/app_localizations_zh.dart | Generated zh localization implementation updates |
| lib/l10n/arb/app_localizations_pt.dart | Generated pt localization implementation updates |
| lib/l10n/arb/app_localizations_nl.dart | Generated nl localization implementation updates |
| lib/l10n/arb/app_localizations_it.dart | Generated it localization implementation updates |
| lib/l10n/arb/app_localizations_hu.dart | Generated hu localization implementation updates |
| lib/l10n/arb/app_localizations_he.dart | Generated he localization implementation updates |
| lib/l10n/arb/app_localizations_fr.dart | Generated fr localization implementation updates |
| lib/l10n/arb/app_localizations_es.dart | Generated es localization implementation updates |
| lib/l10n/arb/app_localizations_en.dart | Generated en localization implementation updates |
| lib/l10n/arb/app_localizations_de.dart | Generated de localization implementation updates |
| lib/l10n/arb/app_localizations_ar.dart | Generated ar localization implementation updates |
| lib/l10n/arb/app_it.arb | Adds Media Storage strings (it) |
| lib/l10n/arb/app_hu.arb | Adds Media Storage strings (hu) |
| lib/l10n/arb/app_he.arb | Adds Media Storage strings (he) |
| lib/l10n/arb/app_fr.arb | Adds Media Storage strings (fr) |
| lib/l10n/arb/app_es.arb | Adds Media Storage strings (es) |
| lib/l10n/arb/app_en.arb | Adds Media Storage strings (en) + formatting adjustments |
| lib/l10n/arb/app_de.arb | Adds Media Storage strings (de) |
| lib/l10n/arb/app_ar.arb | Adds Media Storage strings (ar) |
| lib/features/settings/presentation/pages/settings_page.dart | Adds Settings tile linking to Media Storage page |
| lib/features/media/presentation/widgets/media_item_view.dart | Adds store-backed fallback resolution when native source unavailable |
| lib/features/media/presentation/providers/photo_picker_providers.dart | Enqueues uploads on media creation via provider hook |
| lib/features/media/domain/entities/media_item.dart | Adds content hash/size + remote upload stamp fields to domain model |
| lib/features/media/data/services/media_import_service.dart | Calls onMediaCreated hook after create/import for upload enqueue |
| lib/features/media/data/resolvers/media_store_resolver.dart | Store-backed resolver: download → hash verify → cache → FileData |
| lib/features/media/data/repositories/media_repository.dart | Persists new media fields + adds stampContentIdentity/stampRemoteUploaded |
| lib/features/media_store/presentation/providers/media_store_providers.dart | Runtime construction: store, cache, resolver, upload worker + status hint |
| lib/features/media_store/presentation/providers/media_store_enqueue_provider.dart | Small bridge provider for enqueue hook wiring |
| lib/features/media_store/presentation/pages/media_storage_page.dart | New S3 media-store configuration UI + connect/test/disconnect flows |
| lib/features/media_store/data/media_upload_pipeline.dart | Upload pipeline: materialize, hash, dedup via head, upload, stamp, retry |
| lib/features/media_store/data/media_transfer_queue_repository.dart | Local queue repo: enqueue, nextPending, state transitions, backoff |
| lib/features/media_store/data/media_stores_repository.dart | Synced secret-free store descriptor persistence + pending sync marking |
| lib/features/media_store/data/media_store_worker.dart | Single-flight queue drainer + enqueue-and-kick |
| lib/features/media_store/data/media_store_service.dart | Connect/test/disconnect service using marker + credential persistence |
| lib/features/media_store/data/media_cache_store.dart | Content-addressed cache + index + LRU eviction across two pools |
| lib/core/services/sync/sync_service.dart | Registers mediaStores in sync merge order and updatedAt map |
| lib/core/services/sync/sync_data_serializer.dart | Adds mediaStores to SyncData + export/import/upsert/delete plumbing |
| lib/core/services/media_store/store_marker.dart | Store marker read/ensure implementation (smv1/store.json) |
| lib/core/services/media_store/store_keys.dart | Key derivation + streamed sha256OfFile helper |
| lib/core/services/media_store/s3_media_object_store.dart | S3-backed MediaObjectStore adapter over S3ApiClient |
| lib/core/services/media_store/media_store_credentials_store.dart | Keychain storage for media store S3 config under its own key |
| lib/core/services/media_store/media_store_attach_state.dart | SharedPreferences attach state (storeId) |
| lib/core/services/media_store/media_object_store.dart | MediaObjectStore interface + StoreObjectInfo + error taxonomy |
| lib/core/router/app_router.dart | Adds /settings/media-storage route |
| lib/core/database/local_cache_database.dart | Local cache DB v2: transfer queue + cache entries tables + migration |
| lib/core/database/database.dart | Main DB v103: media store columns + media_stores table + backstop DDL |
| lib/core/data/repositories/sync_repository.dart | Registers mediaStores entity mapping for sync pending records |
| docs/superpowers/specs/2026-07-10-s3-media-storage-design.md | Media Store design spec (Phase plan + goals + architecture) |
ericgriffin
force-pushed
the
worktree-s3-media-store
branch
from
5719d23 to
845e022
Compare
3 tasks
feat(media-store): backends, phase 4 (dropbox, google drive, icloud)
feat(media-store): large objects, phase 3 (multipart, resume, video)
feat(media-store): UX layer, phase 2 (thumbnails, transfers, backfill)
# Conflicts: # lib/core/database/database.dart
github-project-automation
Bot
moved this from In review
to Done
in Submersion Release Tracker
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 (Foundation) of the Media Store: a user-owned cloud location (any S3-compatible service) becomes the canonical home for photo bytes. Photos attached on one device now display on every synced device through the store, instead of today's fragile gallery re-matching, and survive device loss. Design spec:
docs/superpowers/specs/2026-07-10-s3-media-storage-design.md(all four goals, independent config from sync, photos+video across four backends over five phases); implementation plan:docs/superpowers/plans/2026-07-10-media-store-phase1.md.Bytes live at content-addressed keys (
smv1/objects/<aa>/<sha256>.<ext>) under the user's bucket prefix; the syncedmediarow carriescontent_hash+remote_uploaded_atconfirmation stamps; display falls back from the native resolver to a hash-verified, LRU-evicted local cache. Astore.jsonidentity marker detects bucket wipe/repoint and suspends transfers (same defensive pattern as sync's library epoch).Changes
mediacolumns (content_hash,content_size_bytes,remote_uploaded_at,remote_thumb_uploaded_at) plus the synced, secret-freemedia_storesdescriptor table; idempotent DDL with abeforeOpenbackstop against parallel-branch version collisionsmediaStoresregistered across the sync engine (serializer, merge order,_hlcTargets), mirroringbuddyRoles; covered by the existing registration guard testsmedia_transfer_queue(retry with backoff, terminal after 5 attempts) andmedia_cache_entriestableslib/core/services/media_store/:MediaObjectStoreinterface (file-based, Phase 3-ready), S3 adapter over the existingS3ApiClient/SigV4 stack (single-shot; multipart lands in Phase 3),smv1key derivation, streamed SHA-256, store identity marker, keychain credentials store (media_store_s3_config), SharedPreferences attach statehead()dedup (content addressing makes cross-device races converge), upload, stamp; enqueued automatically on photo attach; single-flight worker with a marker preflightMediaStoreResolverengaged as aMediaItemViewfallback (deliberately NOT registered under aMediaSourceType; rows keep their native source type, so disconnecting degrades to exactly today's behavior); downloads are hash-verified before entering the content-addressed cache (originals 2 GiB / thumbs 256 MiB pools)settings_mediaStorage_*strings added to all 11 localesTest Plan
flutter testpasses (targeted per repo practice: sync suite 422, database suite 133, media + media_store 605, settings 608 -- includes new migration, sync round-trip/incremental/e2e, pipeline, cache eviction, resolver fallback widget tests, and a two-device store round-trip)flutter analyzepasses (no issues)Screenshots
New Settings -> Media Storage page follows the existing S3 sync config form; screenshots to follow with the macOS manual pass.
Phases 2-5 (thumbnails/transfers UI/backfill, multipart + video, Drive/Dropbox/iCloud adapters, deletion/GC/verify) follow as separate PRs per the spec's delivery plan.