dendrite2synapse/README.md

# Dendrite to Synapse Migration

Migrate a Matrix homeserver from [Dendrite](https://github.com/matrix-org/dendrite) to [Synapse](https://github.com/element-hq/synapse), preserving users, rooms, messages, media, E2EE keys, and all associated metadata.

Tested against a real production Dendrite instance (bocken.org): 1194 users, 492 rooms, 51,474 events, 2,747 media files — full migration completes in ~8 seconds. Encrypted rooms were ported, but it seems like keys were partially lost. Not a 100% success. Rooms and users persisted and are working.

## What gets migrated

| Phase | Data | Dendrite source | Synapse target |
|-------|------|-----------------|----------------|
| 1 | Users, profiles, devices | `userapi_accounts`, `userapi_profiles`, `userapi_devices` | `users`, `profiles`, `devices` |
| 2 | Rooms and aliases | `roomserver_rooms`, `roomserver_room_aliases` | `rooms`, `room_aliases` |
| 3 | Events (messages, state) | `roomserver_events`, `roomserver_event_json` | `events`, `event_json`, `state_events`, `event_edges`, `event_auth` |
| 4 | Room state snapshots | `syncapi_current_room_state` | `current_state_events`, `state_groups`, `state_groups_state` |
| 5 | Memberships | `syncapi_current_room_state` (member events) | `room_memberships`, `local_current_membership` |
| 6 | Media files + thumbnails | `mediaapi_media_repository`, `mediaapi_thumbnail` | `local_media_repository`, `local_media_repository_thumbnails` |
| 7 | Receipts, redactions, stats | `syncapi_receipts`, `roomserver_redactions` | `receipts_linearized`, `redactions`, `room_stats_*` |
| 8 | E2EE keys (backups, cross-signing) | `userapi_key_backup_*`, `keyserver_*` | `e2e_room_keys*`, `e2e_device_keys_json`, `e2e_cross_signing_*` |

## Prerequisites

- PostgreSQL for both databases
- Python 3.10+ with `psycopg2`
- Dendrite should be stopped (DB not actively written to)
- Synapse must be initialized first (schema created)

```bash
pip install psycopg2-binary
```

## Step-by-step guide

This documents the full process used to migrate bocken.org from Dendrite to Synapse.

### 1. Dump the Dendrite database

On the server running Dendrite, dump the PostgreSQL database:

```bash
# Stop Dendrite first to ensure a clean dump
sudo systemctl stop dendrite

# Dump in custom format (compressed, restorable)
pg_dump -Fc -f dendrite_dump.custom dendrite
```

Transfer the dump to your local machine:

```bash
scp server:dendrite_dump.custom .
```

### 2. Set up the local porting environment

Restore the Dendrite dump into a local PostgreSQL instance:

```bash
# Create a fresh database for the Dendrite data
createdb dendrite_from_dump

# Restore the dump
pg_restore -d dendrite_from_dump dendrite_dump.custom
```

Create an empty Synapse database and initialize the schema:

```bash
# Create the Synapse database
createdb synapse_from_dendrite

# Clone Synapse and set up a minimal config
git clone https://github.com/element-hq/synapse.git
cd synapse
pip install -e ".[postgres]"

# Generate a minimal homeserver.yaml
python -m synapse.app.homeserver \
    --server-name your-domain.com \
    --config-path homeserver.yaml \
    --generate-config \
    --report-stats no

# Edit homeserver.yaml — set the database to PostgreSQL:
#   database:
#     name: psycopg2
#     args:
#       database: synapse_from_dendrite

# Start Synapse once so it creates the schema, then stop it
python -m synapse.app.homeserver --config-path homeserver.yaml
# Wait for "Synapse now listening on..." then Ctrl+C
```

### 3. Run the migration

```bash
# Full migration (all 8 phases)
python3 migrate.py \
    --dendrite-db "dbname=dendrite_from_dump" \
    --synapse-db "dbname=synapse_from_dendrite" \
    --server-name your-domain.com

# With media file copying (if you have the media directory locally)
python3 migrate.py \
    --dendrite-db "dbname=dendrite_from_dump" \
    --synapse-db "dbname=synapse_from_dendrite" \
    --server-name your-domain.com \
    --dendrite-media-path ./media \
    --synapse-media-path ./media_store
```

#### Useful flags

```bash
# Dry run — inspect without committing any changes
python3 migrate.py ... --dry-run -v

# Run specific phases (e.g., re-run only media)
python3 migrate.py ... --phase 6

# Copy media files only (no database writes, no --synapse-db needed)
python3 migrate.py --media-only \
    --dendrite-db "dbname=dendrite_from_dump" \
    --server-name your-domain.com \
    --dendrite-media-path ./media \
    --synapse-media-path ./media_store
```

### 4. Verify locally

Start Synapse against the migrated database and check:

```bash
python -m synapse.app.homeserver --config-path homeserver.yaml

# In another terminal — verify via admin API
# (create an admin token first or use an existing one)
curl -s http://localhost:8008/_synapse/admin/v2/rooms | python3 -m json.tool | head
curl -s http://localhost:8008/_synapse/admin/v2/users | python3 -m json.tool | head
```

### 5. Deploy to the server

Dump the migrated Synapse database and transfer it:

```bash
# Dump the migrated Synapse DB
pg_dump -Fc -f synapse_from_dendrite.sql.custom synapse_from_dendrite

# Transfer to server
scp synapse_from_dendrite.sql.custom server:
```

On the server:

```bash
# Stop Dendrite
sudo systemctl stop dendrite

# Create the Synapse database and restore
sudo -u postgres createdb -O synapse synapse_bocken
pg_restore -d synapse_bocken synapse_from_dendrite.sql.custom

# If you ran --media-only locally or need to copy media on the server:
# Option A: copy the media_store directory you built locally
# Option B: run --media-only on the server pointing at Dendrite's media path
python3 migrate.py --media-only \
    --dendrite-db "dbname=dendrite" \
    --server-name your-domain.com \
    --dendrite-media-path /var/lib/dendrite/media \
    --synapse-media-path /var/lib/synapse/media_store

# Start Synapse
sudo systemctl start synapse
```

### 6. Post-migration: OIDC user linking

If you're switching to OIDC authentication and existing users already have accounts, Synapse will try to create new accounts (e.g., `alexander1` instead of `alexander`). Link existing accounts to their OIDC identities directly in the database:

```sql
-- Check if a stale mapping was created from a failed login attempt
SELECT * FROM user_external_ids
WHERE auth_provider = 'oidc'
  AND external_id = '<your-oidc-sub-claim>';

-- If it points to the wrong user (e.g., alexander1), update it
UPDATE user_external_ids
SET user_id = '@alexander:your-domain.com'
WHERE auth_provider = 'oidc'
  AND external_id = '<your-oidc-sub-claim>';

-- Clean up the accidentally created user if needed
DELETE FROM user_external_ids WHERE user_id = '@alexander1:your-domain.com';
```

The `sub` claim is the unique identifier from your OIDC provider (not the username) — typically a UUID or hash. Find it in your provider's admin panel or decode an ID token:

```bash
echo '<id_token>' | cut -d. -f2 | base64 -d 2>/dev/null | jq .sub
```

## Architecture notes

### Key differences between Dendrite and Synapse

- **Dendrite** uses numeric IDs (NIDs) for rooms, events, event types, state keys. Event JSON stored separately, types resolved via lookup tables.
- **Synapse** uses text IDs directly. State managed via delta-chained state groups. Media uses a different filesystem layout.

### Schema mapping

See [TODO.md](TODO.md) for the complete mapping table, findings log, and per-phase validation results.

### Media file layout conversion

```
Dendrite:  {base}/{hash[0]}/{hash[1]}/{hash[2:]}/file
Synapse:   {base}/local_content/{id[0:2]}/{id[2:4]}/{id[4:]}

Dendrite:  {base}/{hash[0]}/{hash[1]}/{hash[2:]}/thumbnail-{W}x{H}-{method}
Synapse:   {base}/local_thumbnails/{id[0:2]}/{id[2:4]}/{id[4:]}/{W}-{H}-{type}-{subtype}-{method}
```

## Known issues

- 2,262 rejected events in Dendrite are skipped (expected)
- ~5,500 orphan event edges referencing federated events we don't have (normal for any homeserver)
- Synapse runs background update tasks after first startup on the migrated DB — this is normal and may take a few minutes
- E2EE message history requires clients to have used server-side key backup; client-side-only keys cannot be migrated server-side