177 lines
8.9 KiB
Markdown
177 lines
8.9 KiB
Markdown
# Dendrite to Synapse Migration - TODO
|
|
|
|
## Goal
|
|
Migrate local PostgreSQL data from Dendrite to Synapse. Minimum: users, rooms, messages, files.
|
|
|
|
## Status: ALL PHASES COMPLETE AND VALIDATED
|
|
|
|
Tested against real Dendrite DB dump (bocken.org):
|
|
- 1194 users, 492 rooms, 51474 events, 2747 media files, 3309 thumbnails
|
|
- Full migration runs in ~8 seconds
|
|
- Synapse starts cleanly, admin API returns correct data
|
|
- Messages, room state, memberships, media metadata all verified
|
|
|
|
## Architecture Notes
|
|
|
|
### Dendrite Schema (Go, NID-based)
|
|
- Uses numeric IDs (NIDs) for rooms, events, event types, state keys
|
|
- Event JSON stored separately in `roomserver_event_json`
|
|
- Event types mapped via `roomserver_event_types` (nid -> string)
|
|
- State keys mapped via `roomserver_event_state_keys` (nid -> string)
|
|
- Membership uses numeric nid references (target_nid = event_state_key_nid of user)
|
|
- Media uses `base64hash` (SHA-256) for dedup, stored in `mediaapi_media_repository`
|
|
- Media files: `{base}/{hash[0]}/{hash[1]}/{hash[2:]}/file`
|
|
- Thumbnails: `{base}/{hash[0]}/{hash[1]}/{hash[2:]}/thumbnail-{w}x{h}-{method}`
|
|
- Accounts in `userapi_accounts`, profiles in `userapi_profiles`
|
|
|
|
### Synapse Schema (Python, text-based)
|
|
- Uses text IDs directly everywhere
|
|
- Event JSON in `event_json` table, metadata in `events` table
|
|
- State managed via `state_groups` + `state_groups_state` with delta chains
|
|
- Membership in `room_memberships` + `local_current_membership`
|
|
- Media in `local_media_repository` (uses media_id as filesystem key)
|
|
- Media files: `{base}/local_content/{id[0:2]}/{id[2:4]}/{id[4:]}`
|
|
- Thumbnails: `{base}/local_thumbnails/{id[0:2]}/{id[2:4]}/{id[4:]}/{w}-{h}-{top}-{sub}-{method}`
|
|
- Accounts in `users`, profiles in `profiles`
|
|
- Schema versioned (currently v93-94), needs Synapse pre-init to create schema
|
|
|
|
### Key Mapping: Dendrite -> Synapse
|
|
| Dendrite Table | Synapse Table | Notes |
|
|
|---|---|---|
|
|
| userapi_accounts | users | password_hash, created_ts (ms->s), account_type->is_guest/admin |
|
|
| userapi_profiles | profiles | user_id=localpart, full_user_id=@user:server |
|
|
| userapi_devices | devices + access_tokens | direct map; access_token preserved so clients don't re-login |
|
|
| roomserver_rooms | rooms | room_id, room_version; creator from m.room.create events |
|
|
| roomserver_events + event_json | events + event_json | denormalize NIDs, topological_ordering=depth |
|
|
| syncapi_current_room_state | current_state_events | direct map |
|
|
| syncapi_current_room_state (member) | room_memberships + local_current_membership | |
|
|
| mediaapi_media_repository | local_media_repository | media_id, type, size, upload_name, user_id |
|
|
| mediaapi_thumbnail | local_media_repository_thumbnails | |
|
|
| syncapi_receipts | receipts_linearized + receipts_graph | partial unique index for NULL thread_id |
|
|
| roomserver_redactions | redactions | |
|
|
|
|
## Tasks
|
|
|
|
### Phase 0: Setup
|
|
- [x] Explore Dendrite schema
|
|
- [x] Explore Synapse schema
|
|
- [x] Create migration plan
|
|
- [x] Create script skeleton with connection handling + CLI args
|
|
|
|
### Phase 1: Users & Profiles
|
|
- [x] Migrate userapi_accounts -> users (created_ts ms->s conversion)
|
|
- [x] Migrate userapi_profiles -> profiles (user_id=localpart, full_user_id=@user:server)
|
|
- [x] Migrate userapi_devices -> devices
|
|
- [x] Tested: 1194 users, 1194 profiles, 13 devices
|
|
|
|
### Phase 2: Rooms
|
|
- [x] Migrate roomserver_rooms -> rooms
|
|
- [x] Extract room creator from m.room.create events
|
|
- [x] Migrate roomserver_room_aliases -> room_aliases + room_alias_servers
|
|
- [x] Tested: 492 rooms, correct creators
|
|
|
|
### Phase 3: Events (Core)
|
|
- [x] Build event_type NID->string and state_key NID->string lookups
|
|
- [x] Migrate events with denormalized types/state_keys
|
|
- [x] stream_ordering = global sequential, topological_ordering = depth
|
|
- [x] internal_metadata = "{}" (stream_ordering/outlier read from events columns)
|
|
- [x] format_version mapped from room version (v1-2->1, v3->2, v4-10->3, v11+->4)
|
|
- [x] processed = True for migrated events
|
|
- [x] Migrate event_json with correct format
|
|
- [x] Populate state_events (events where state_key IS NOT NULL)
|
|
- [x] Build event_edges from prev_events in event JSON
|
|
- [x] Build event_auth from auth_events in event JSON
|
|
- [x] Forward extremities from Dendrite's latest_event_nids
|
|
- [x] room_depth from MIN(depth) per room
|
|
- [x] Tested: 51474 events, 24609 state events, 489 fwd extremities
|
|
|
|
### Phase 4: Room State
|
|
- [x] current_state_events from syncapi_current_room_state
|
|
- [x] Incremental state groups: one per state event, delta chains via state_group_edges
|
|
- [x] All events mapped to correct state group via event_to_state_groups
|
|
- [x] Tested: 24609 state groups, 51474 event mappings, 0 unmapped events
|
|
|
|
### Phase 5: Membership
|
|
- [x] Migrate from syncapi_current_room_state (type=m.room.member) -> room_memberships
|
|
- [x] Populate local_current_membership for local users
|
|
- [x] Include event_stream_ordering FK
|
|
- [x] Tested: 7254 memberships, 3220 local memberships
|
|
|
|
### Phase 6: Media
|
|
- [x] Migrate mediaapi_media_repository -> local_media_repository
|
|
- [x] Migrate mediaapi_thumbnail -> local_media_repository_thumbnails
|
|
- [x] Copy content files: Dendrite `{base}/{hash[0]}/{hash[1]}/{hash[2:]}/file` -> Synapse `{base}/local_content/{id[0:2]}/{id[2:4]}/{id[4:]}`
|
|
- [x] Copy thumbnails: Dendrite `thumbnail-{w}x{h}-{method}` -> Synapse `{w}-{h}-{top}-{sub}-{method}`
|
|
- [x] Tested: 2747 media, 3309 thumbnails, file paths verified
|
|
|
|
### Phase 7: Auxiliary Data
|
|
- [x] Migrate receipts (receipts_linearized + receipts_graph, partial unique index)
|
|
- [x] Migrate redactions
|
|
- [x] Populate room_stats_current (member counts by type)
|
|
- [x] Populate room_stats_state (room name, topic, encryption, etc.)
|
|
- [x] Update events_stream_seq sequence
|
|
- [x] Populate user_stats_current
|
|
- [x] Tested: 857 receipts, 216 redactions, 492 room stats
|
|
|
|
### Validation
|
|
- [x] Synapse starts against migrated DB without errors
|
|
- [x] Admin API: 488 rooms visible with correct names and member counts
|
|
- [x] Messages accessible and readable via API
|
|
- [x] Room state correct (creator, version, state types)
|
|
- [x] Media metadata accessible via admin statistics API
|
|
- [x] Background updates run normally post-migration
|
|
|
|
## Findings / Issues Log
|
|
- Dendrite event_state_key_nid 0 = not a state event, nid 1 = '' (empty string)
|
|
- Dendrite event_type_nid preassigned: 1=m.room.create, 2=power_levels, 3=join_rules, 4=third_party_invite, 5=member, 6=redaction, 7=history_visibility
|
|
- Synapse topological_ordering = depth (NOT a per-room counter)
|
|
- Synapse internal_metadata JSON should be "{}" - stream_ordering and outlier loaded from events table columns
|
|
- Synapse format_version: room v1-2=1, v3=2, v4-10=3, v11+=4
|
|
- Synapse receipts_linearized has partial unique index WHERE thread_id IS NULL
|
|
- Synapse room_alias_servers has no unique constraint - must check-before-insert
|
|
- Synapse profiles unique on user_id (localpart), NOT on full_user_id
|
|
- Forward extremities: use Dendrite's latest_event_nids, don't compute from graph
|
|
- 2262 rejected events in Dendrite skipped during migration
|
|
- 5548 orphan event edges (referencing federated events we don't have) - normal
|
|
- Synapse background updates recalculate some stats after startup - normal
|
|
- E2EE: three things must be migrated together for encrypted history to survive —
|
|
(1) `userapi_devices.access_token` -> `access_tokens` so clients don't re-login
|
|
(re-login usually wipes the local Megolm store and always changes device_id,
|
|
breaking Olm continuity),
|
|
(2) `syncapi_send_to_device` -> `device_inbox` so undelivered m.room.encrypted
|
|
Olm messages (Megolm key shares to offline devices) reach the recipient,
|
|
(3) `device_lists_stream` seeded from local devices so clients re-verify
|
|
e2e_device_keys_json on first sync (otherwise: stale cache mismatches).
|
|
v1 of the migration only moved the Synapse-native E2EE tables; that left
|
|
1480 pending to-device messages stranded and all clients forced to re-login,
|
|
which is the root cause of "partial key loss" reported against v1.
|
|
- E2EE: `keyserver_fallback_keys` -> `e2e_fallback_keys_json` added so new Olm
|
|
sessions still succeed after OTKs are exhausted.
|
|
- E2EE: `e2e_cross_signing_keys.stream_id` now drawn from the matching
|
|
sequence (`e2e_cross_signing_keys_sequence`) to avoid UNIQUE(stream_id)
|
|
collisions on subsequent Synapse writes.
|
|
- Re-running Phase 8 duplicates `e2e_cross_signing_signatures` rows (no unique
|
|
constraint on the Synapse side). TRUNCATE before re-running, or run once.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Prerequisites: Synapse must be initialized first (creates schema)
|
|
python3 -m synapse.app.homeserver --config-path homeserver.yaml # start+stop once
|
|
|
|
# Full migration
|
|
python3 migrate.py \
|
|
--dendrite-db "dbname=dendrite host=/run/postgresql" \
|
|
--synapse-db "dbname=synapse host=/run/postgresql" \
|
|
--server-name "example.com" \
|
|
--dendrite-media-path /var/lib/dendrite/media \
|
|
--synapse-media-path /var/lib/synapse/media_store \
|
|
--phase 1,2,3,4,5,6,7
|
|
|
|
# Selective phases (e.g., just re-run media)
|
|
python3 migrate.py ... --phase 6
|
|
|
|
# Dry run (no commits)
|
|
python3 migrate.py ... --dry-run
|
|
```
|