# Dendrite to Synapse Migration

Migrate a Matrix homeserver from [Dendrite](https://github.com/matrix-org/dendrite) to [Synapse](https://github.com/element-hq/synapse), preserving users, rooms, messages, media, E2EE keys, and all associated metadata.

Tested against a real production Dendrite instance (bocken.org): 1194 users, 492 rooms, 51,474 events, 2,747 media files, 1,480 pending to-device messages — full migration completes in ~8 seconds. Users, rooms, messages, media, devices, sessions (access tokens), cross-signing, key backup, and the to-device inbox are all ported. Encrypted history clients can decrypt locally continues to decrypt; undelivered Megolm key shares are flushed to Synapse's `device_inbox` on next sync.

## What gets migrated

| Phase | Data | Dendrite source | Synapse target |
|-------|------|-----------------|----------------|
| 1 | Users, profiles, devices, sessions | `userapi_accounts`, `userapi_profiles`, `userapi_devices` | `users`, `profiles`, `devices`, `access_tokens` |
| 2 | Rooms and aliases | `roomserver_rooms`, `roomserver_room_aliases` | `rooms`, `room_aliases` |
| 3 | Events (messages, state) | `roomserver_events`, `roomserver_event_json` | `events`, `event_json`, `state_events`, `event_edges`, `event_auth` |
| 4 | Room state snapshots | `syncapi_current_room_state` | `current_state_events`, `state_groups`, `state_groups_state` |
| 5 | Memberships | `syncapi_current_room_state` (member events) | `room_memberships`, `local_current_membership` |
| 6 | Media files + thumbnails | `mediaapi_media_repository`, `mediaapi_thumbnail` | `local_media_repository`, `local_media_repository_thumbnails` |
| 7 | Receipts, redactions, stats | `syncapi_receipts`, `roomserver_redactions` | `receipts_linearized`, `redactions`, `room_stats_*` |
| 8 | E2EE keys, to-device inbox, device-list stream | `userapi_key_backup_*`, `keyserver_*`, `syncapi_send_to_device` | `e2e_room_keys*`, `e2e_device_keys_json`, `e2e_fallback_keys_json`, `e2e_cross_signing_*`, `device_inbox`, `device_lists_stream` |

## Prerequisites

- PostgreSQL for both databases
- Python 3.10+ with `psycopg2`
- Dendrite should be stopped (DB not actively written to)
- Synapse must be initialized first (schema created)

```bash
pip install psycopg2-binary
```

## Step-by-step guide

This documents the full process used to migrate bocken.org from Dendrite to Synapse.

### 1. Dump the Dendrite database

On the server running Dendrite, dump the PostgreSQL database:

```bash
# Stop Dendrite first to ensure a clean dump
sudo systemctl stop dendrite

# Dump in custom format (compressed, restorable)
pg_dump -Fc -f dendrite_dump.custom dendrite
```

Transfer the dump to your local machine:

```bash
scp server:dendrite_dump.custom .
```

### 2. Set up the local porting environment

Restore the Dendrite dump into a local PostgreSQL instance:

```bash
# Create a fresh database for the Dendrite data
createdb dendrite_from_dump

# Restore the dump
pg_restore -d dendrite_from_dump dendrite_dump.custom
```

Create an empty Synapse database and initialize the schema:

```bash
# Create the Synapse database
createdb synapse_from_dendrite

# Clone Synapse and set up a minimal config
git clone https://github.com/element-hq/synapse.git
cd synapse
pip install -e ".[postgres]"

# Generate a minimal homeserver.yaml
python -m synapse.app.homeserver \
    --server-name your-domain.com \
    --config-path homeserver.yaml \
    --generate-config \
    --report-stats no

# Edit homeserver.yaml — set the database to PostgreSQL:
#   database:
#     name: psycopg2
#     args:
#       database: synapse_from_dendrite

# Start Synapse once so it creates the schema, then stop it
python -m synapse.app.homeserver --config-path homeserver.yaml
# Wait for "Synapse now listening on..." then Ctrl+C
```

### 3. Run the migration

```bash
# Full migration (all 8 phases)
python3 migrate.py \
    --dendrite-db "dbname=dendrite_from_dump" \
    --synapse-db "dbname=synapse_from_dendrite" \
    --server-name your-domain.com

# With media file copying (if you have the media directory locally)
python3 migrate.py \
    --dendrite-db "dbname=dendrite_from_dump" \
    --synapse-db "dbname=synapse_from_dendrite" \
    --server-name your-domain.com \
    --dendrite-media-path ./media \
    --synapse-media-path ./media_store
```

#### Useful flags

```bash
# Dry run — inspect without committing any changes
python3 migrate.py ... --dry-run -v

# Run specific phases (e.g., re-run only media)
python3 migrate.py ... --phase 6

# Copy media files only (no database writes, no --synapse-db needed)
python3 migrate.py --media-only \
    --dendrite-db "dbname=dendrite_from_dump" \
    --server-name your-domain.com \
    --dendrite-media-path ./media \
    --synapse-media-path ./media_store
```

### 4. Verify locally

Start Synapse against the migrated database and check:

```bash
python -m synapse.app.homeserver --config-path homeserver.yaml

# In another terminal — verify via admin API
# (create an admin token first or use an existing one)
curl -s http://localhost:8008/_synapse/admin/v2/rooms | python3 -m json.tool | head
curl -s http://localhost:8008/_synapse/admin/v2/users | python3 -m json.tool | head
```

### 5. Deploy to the server

Dump the migrated Synapse database and transfer it:

```bash
# Dump the migrated Synapse DB
pg_dump -Fc -f synapse_from_dendrite.sql.custom synapse_from_dendrite

# Transfer to server
scp synapse_from_dendrite.sql.custom server:
```

On the server:

```bash
# Stop Dendrite
sudo systemctl stop dendrite

# Create the Synapse database and restore
sudo -u postgres createdb -O synapse synapse_bocken
pg_restore -d synapse_bocken synapse_from_dendrite.sql.custom

# If you ran --media-only locally or need to copy media on the server:
# Option A: copy the media_store directory you built locally
# Option B: run --media-only on the server pointing at Dendrite's media path
python3 migrate.py --media-only \
    --dendrite-db "dbname=dendrite" \
    --server-name your-domain.com \
    --dendrite-media-path /var/lib/dendrite/media \
    --synapse-media-path /var/lib/synapse/media_store

# Start Synapse
sudo systemctl start synapse
```

### 6. Post-migration: OIDC user linking

If you're switching to OIDC authentication and existing users already have accounts, Synapse will try to create new accounts (e.g., `alexander1` instead of `alexander`). Link existing accounts to their OIDC identities directly in the database:

```sql
-- Check if a stale mapping was created from a failed login attempt
SELECT * FROM user_external_ids
WHERE auth_provider = 'oidc'
  AND external_id = '<your-oidc-sub-claim>';

-- If it points to the wrong user (e.g., alexander1), update it
UPDATE user_external_ids
SET user_id = '@alexander:your-domain.com'
WHERE auth_provider = 'oidc'
  AND external_id = '<your-oidc-sub-claim>';

-- Clean up the accidentally created user if needed
DELETE FROM user_external_ids WHERE user_id = '@alexander1:your-domain.com';
```

The `sub` claim is the unique identifier from your OIDC provider (not the username) — typically a UUID or hash. Find it in your provider's admin panel or decode an ID token:

```bash
echo '<id_token>' | cut -d. -f2 | base64 -d 2>/dev/null | jq .sub
```

## Architecture notes

### Key differences between Dendrite and Synapse

- **Dendrite** uses numeric IDs (NIDs) for rooms, events, event types, state keys. Event JSON stored separately, types resolved via lookup tables.
- **Synapse** uses text IDs directly. State managed via delta-chained state groups. Media uses a different filesystem layout.

### Schema mapping

See [TODO.md](TODO.md) for the complete mapping table, findings log, and per-phase validation results.

### Media file layout conversion

```
Dendrite:  {base}/{hash[0]}/{hash[1]}/{hash[2:]}/file
Synapse:   {base}/local_content/{id[0:2]}/{id[2:4]}/{id[4:]}

Dendrite:  {base}/{hash[0]}/{hash[1]}/{hash[2:]}/thumbnail-{W}x{H}-{method}
Synapse:   {base}/local_thumbnails/{id[0:2]}/{id[2:4]}/{id[4:]}/{W}-{H}-{type}-{subtype}-{method}
```

## E2EE: what gets preserved, what depends on the client

The homeserver never sees Megolm message keys in the clear — those live in each client's key store. For a migration to preserve decryptable history end-to-end, three independent pieces all have to line up:

1. **Client device identity stays the same.** The client's `device_id`, `access_token`, `curve25519` identity key, and `ed25519` signing key must all match what the client has cached locally. The migration copies all four: `userapi_devices` → `devices` + `access_tokens` (so the client is not forced to re-login), and `keyserver_device_keys` → `e2e_device_keys_json` (so other users and the client itself see the same identity keys). Without `access_tokens`, clients get 401s on their bearer token, log in again, and generate a fresh device — at which point the local Megolm store on many clients becomes unusable.

2. **Undelivered room-key shares survive.** When a user shares a Megolm session, the key is wrapped in an `m.room.encrypted` Olm message and sent as a to-device message. If the recipient was offline, the message sits in Dendrite's `syncapi_send_to_device` until they next sync. The migration moves this queue verbatim into Synapse's `device_inbox`. Skipping this step is what caused "keys partially lost" in the first test run — offline recipients never got the room keys, and those rooms remained undecryptable until someone re-shared.

3. **Clients re-verify device lists on first sync.** Synapse tells clients which users' device lists have changed via `/sync`'s `device_lists.changed`. The migration seeds `device_lists_stream` with one entry per local device so every client treats every local device as "changed" once after migration, re-fetches `e2e_device_keys_json`, and avoids a stale-cache "device key mismatch → refuse to decrypt" failure.

What cannot be migrated server-side:

- **Client-side-only Megolm sessions.** If a client chose not to enable server-side key backup, the Megolm session keys for messages it received live only in that client's local store. The migration preserves device identity so the client keeps and uses those local keys, but there is no server-side copy to restore from.
- **Olm session state** (the double-ratchet chains between device pairs) is client-side only. Clients will transparently start new Olm sessions using the migrated `e2e_one_time_keys_json` / `e2e_fallback_keys_json` on the next message exchange — no user-visible effect.

## Known issues

- 2,262 rejected events in Dendrite are skipped (expected)
- ~5,500 orphan event edges referencing federated events we don't have (normal for any homeserver)
- Synapse runs background update tasks after first startup on the migrated DB — this is normal and may take a few minutes
- Re-running Phase 8 appends duplicate `e2e_cross_signing_signatures` rows (no unique constraint on the target table). Either run Phase 8 once, or `TRUNCATE e2e_cross_signing_signatures` before re-running.