inital commit

2026-04-12 11:53:04 +02:00
commit 9693c714d6
4 changed files with 2592 additions and 0 deletions
@@ -0,0 +1,229 @@
+# Dendrite to Synapse Migration
+
+Migrate a Matrix homeserver from [Dendrite](https://github.com/matrix-org/dendrite) to [Synapse](https://github.com/element-hq/synapse), preserving users, rooms, messages, media, E2EE keys, and all associated metadata.
+
+Tested against a real production Dendrite instance (bocken.org): 1194 users, 492 rooms, 51,474 events, 2,747 media files — full migration completes in ~8 seconds. Encrypted rooms were ported, but it seems like keys were partially lost. Not a 100% success. Rooms and users persisted and are working.
+
+## What gets migrated
+
+| Phase | Data | Dendrite source | Synapse target |
+|-------|------|-----------------|----------------|
+| 1 | Users, profiles, devices | `userapi_accounts`, `userapi_profiles`, `userapi_devices` | `users`, `profiles`, `devices` |
+| 2 | Rooms and aliases | `roomserver_rooms`, `roomserver_room_aliases` | `rooms`, `room_aliases` |
+| 3 | Events (messages, state) | `roomserver_events`, `roomserver_event_json` | `events`, `event_json`, `state_events`, `event_edges`, `event_auth` |
+| 4 | Room state snapshots | `syncapi_current_room_state` | `current_state_events`, `state_groups`, `state_groups_state` |
+| 5 | Memberships | `syncapi_current_room_state` (member events) | `room_memberships`, `local_current_membership` |
+| 6 | Media files + thumbnails | `mediaapi_media_repository`, `mediaapi_thumbnail` | `local_media_repository`, `local_media_repository_thumbnails` |
+| 7 | Receipts, redactions, stats | `syncapi_receipts`, `roomserver_redactions` | `receipts_linearized`, `redactions`, `room_stats_*` |
+| 8 | E2EE keys (backups, cross-signing) | `userapi_key_backup_*`, `keyserver_*` | `e2e_room_keys*`, `e2e_device_keys_json`, `e2e_cross_signing_*` |
+
+## Prerequisites
+
+- PostgreSQL for both databases
+- Python 3.10+ with `psycopg2`
+- Dendrite should be stopped (DB not actively written to)
+- Synapse must be initialized first (schema created)
+
+```bash
+pip install psycopg2-binary
+```
+
+## Step-by-step guide
+
+This documents the full process used to migrate bocken.org from Dendrite to Synapse.
+
+### 1. Dump the Dendrite database
+
+On the server running Dendrite, dump the PostgreSQL database:
+
+```bash
+# Stop Dendrite first to ensure a clean dump
+sudo systemctl stop dendrite
+
+# Dump in custom format (compressed, restorable)
+pg_dump -Fc -f dendrite_dump.custom dendrite
+```
+
+Transfer the dump to your local machine:
+
+```bash
+scp server:dendrite_dump.custom .
+```
+
+### 2. Set up the local porting environment
+
+Restore the Dendrite dump into a local PostgreSQL instance:
+
+```bash
+# Create a fresh database for the Dendrite data
+createdb dendrite_from_dump
+
+# Restore the dump
+pg_restore -d dendrite_from_dump dendrite_dump.custom
+```
+
+Create an empty Synapse database and initialize the schema:
+
+```bash
+# Create the Synapse database
+createdb synapse_from_dendrite
+
+# Clone Synapse and set up a minimal config
+git clone https://github.com/element-hq/synapse.git
+cd synapse
+pip install -e ".[postgres]"
+
+# Generate a minimal homeserver.yaml
+python -m synapse.app.homeserver \
+    --server-name your-domain.com \
+    --config-path homeserver.yaml \
+    --generate-config \
+    --report-stats no
+
+# Edit homeserver.yaml — set the database to PostgreSQL:
+#   database:
+#     name: psycopg2
+#     args:
+#       database: synapse_from_dendrite
+
+# Start Synapse once so it creates the schema, then stop it
+python -m synapse.app.homeserver --config-path homeserver.yaml
+# Wait for "Synapse now listening on..." then Ctrl+C
+```
+
+### 3. Run the migration
+
+```bash
+# Full migration (all 8 phases)
+python3 migrate.py \
+    --dendrite-db "dbname=dendrite_from_dump" \
+    --synapse-db "dbname=synapse_from_dendrite" \
+    --server-name your-domain.com
+
+# With media file copying (if you have the media directory locally)
+python3 migrate.py \
+    --dendrite-db "dbname=dendrite_from_dump" \
+    --synapse-db "dbname=synapse_from_dendrite" \
+    --server-name your-domain.com \
+    --dendrite-media-path ./media \
+    --synapse-media-path ./media_store
+```
+
+#### Useful flags
+
+```bash
+# Dry run — inspect without committing any changes
+python3 migrate.py ... --dry-run -v
+
+# Run specific phases (e.g., re-run only media)
+python3 migrate.py ... --phase 6
+
+# Copy media files only (no database writes, no --synapse-db needed)
+python3 migrate.py --media-only \
+    --dendrite-db "dbname=dendrite_from_dump" \
+    --server-name your-domain.com \
+    --dendrite-media-path ./media \
+    --synapse-media-path ./media_store
+```
+
+### 4. Verify locally
+
+Start Synapse against the migrated database and check:
+
+```bash
+python -m synapse.app.homeserver --config-path homeserver.yaml
+
+# In another terminal — verify via admin API
+# (create an admin token first or use an existing one)
+curl -s http://localhost:8008/_synapse/admin/v2/rooms | python3 -m json.tool | head
+curl -s http://localhost:8008/_synapse/admin/v2/users | python3 -m json.tool | head
+```
+
+### 5. Deploy to the server
+
+Dump the migrated Synapse database and transfer it:
+
+```bash
+# Dump the migrated Synapse DB
+pg_dump -Fc -f synapse_from_dendrite.sql.custom synapse_from_dendrite
+
+# Transfer to server
+scp synapse_from_dendrite.sql.custom server:
+```
+
+On the server:
+
+```bash
+# Stop Dendrite
+sudo systemctl stop dendrite
+
+# Create the Synapse database and restore
+sudo -u postgres createdb -O synapse synapse_bocken
+pg_restore -d synapse_bocken synapse_from_dendrite.sql.custom
+
+# If you ran --media-only locally or need to copy media on the server:
+# Option A: copy the media_store directory you built locally
+# Option B: run --media-only on the server pointing at Dendrite's media path
+python3 migrate.py --media-only \
+    --dendrite-db "dbname=dendrite" \
+    --server-name your-domain.com \
+    --dendrite-media-path /var/lib/dendrite/media \
+    --synapse-media-path /var/lib/synapse/media_store
+
+# Start Synapse
+sudo systemctl start synapse
+```
+
+### 6. Post-migration: OIDC user linking
+
+If you're switching to OIDC authentication and existing users already have accounts, Synapse will try to create new accounts (e.g., `alexander1` instead of `alexander`). Link existing accounts to their OIDC identities directly in the database:
+
+```sql
+-- Check if a stale mapping was created from a failed login attempt
+SELECT * FROM user_external_ids
+WHERE auth_provider = 'oidc'
+  AND external_id = '<your-oidc-sub-claim>';
+
+-- If it points to the wrong user (e.g., alexander1), update it
+UPDATE user_external_ids
+SET user_id = '@alexander:your-domain.com'
+WHERE auth_provider = 'oidc'
+  AND external_id = '<your-oidc-sub-claim>';
+
+-- Clean up the accidentally created user if needed
+DELETE FROM user_external_ids WHERE user_id = '@alexander1:your-domain.com';
+```
+
+The `sub` claim is the unique identifier from your OIDC provider (not the username) — typically a UUID or hash. Find it in your provider's admin panel or decode an ID token:
+
+```bash
+echo '<id_token>' | cut -d. -f2 | base64 -d 2>/dev/null | jq .sub
+```
+
+## Architecture notes
+
+### Key differences between Dendrite and Synapse
+
+- **Dendrite** uses numeric IDs (NIDs) for rooms, events, event types, state keys. Event JSON stored separately, types resolved via lookup tables.
+- **Synapse** uses text IDs directly. State managed via delta-chained state groups. Media uses a different filesystem layout.
+
+### Schema mapping
+
+See [TODO.md](TODO.md) for the complete mapping table, findings log, and per-phase validation results.
+
+### Media file layout conversion
+
+```
+Dendrite:  {base}/{hash[0]}/{hash[1]}/{hash[2:]}/file
+Synapse:   {base}/local_content/{id[0:2]}/{id[2:4]}/{id[4:]}
+
+Dendrite:  {base}/{hash[0]}/{hash[1]}/{hash[2:]}/thumbnail-{W}x{H}-{method}
+Synapse:   {base}/local_thumbnails/{id[0:2]}/{id[2:4]}/{id[4:]}/{W}-{H}-{type}-{subtype}-{method}
+```
+
+## Known issues
+
+- 2,262 rejected events in Dendrite are skipped (expected)
+- ~5,500 orphan event edges referencing federated events we don't have (normal for any homeserver)
+- Synapse runs background update tasks after first startup on the migrated DB — this is normal and may take a few minutes
+- E2EE message history requires clients to have used server-side key backup; client-side-only keys cannot be migrated server-side
@@ -0,0 +1,158 @@
+# Dendrite to Synapse Migration - TODO
+
+## Goal
+Migrate local PostgreSQL data from Dendrite to Synapse. Minimum: users, rooms, messages, files.
+
+## Status: ALL PHASES COMPLETE AND VALIDATED
+
+Tested against real Dendrite DB dump (bocken.org):
+- 1194 users, 492 rooms, 51474 events, 2747 media files, 3309 thumbnails
+- Full migration runs in ~8 seconds
+- Synapse starts cleanly, admin API returns correct data
+- Messages, room state, memberships, media metadata all verified
+
+## Architecture Notes
+
+### Dendrite Schema (Go, NID-based)
+- Uses numeric IDs (NIDs) for rooms, events, event types, state keys
+- Event JSON stored separately in `roomserver_event_json`
+- Event types mapped via `roomserver_event_types` (nid -> string)
+- State keys mapped via `roomserver_event_state_keys` (nid -> string)
+- Membership uses numeric nid references (target_nid = event_state_key_nid of user)
+- Media uses `base64hash` (SHA-256) for dedup, stored in `mediaapi_media_repository`
+- Media files: `{base}/{hash[0]}/{hash[1]}/{hash[2:]}/file`
+- Thumbnails: `{base}/{hash[0]}/{hash[1]}/{hash[2:]}/thumbnail-{w}x{h}-{method}`
+- Accounts in `userapi_accounts`, profiles in `userapi_profiles`
+
+### Synapse Schema (Python, text-based)
+- Uses text IDs directly everywhere
+- Event JSON in `event_json` table, metadata in `events` table
+- State managed via `state_groups` + `state_groups_state` with delta chains
+- Membership in `room_memberships` + `local_current_membership`
+- Media in `local_media_repository` (uses media_id as filesystem key)
+- Media files: `{base}/local_content/{id[0:2]}/{id[2:4]}/{id[4:]}`
+- Thumbnails: `{base}/local_thumbnails/{id[0:2]}/{id[2:4]}/{id[4:]}/{w}-{h}-{top}-{sub}-{method}`
+- Accounts in `users`, profiles in `profiles`
+- Schema versioned (currently v93-94), needs Synapse pre-init to create schema
+
+### Key Mapping: Dendrite -> Synapse
+| Dendrite Table | Synapse Table | Notes |
+|---|---|---|
+| userapi_accounts | users | password_hash, created_ts (ms->s), account_type->is_guest/admin |
+| userapi_profiles | profiles | user_id=localpart, full_user_id=@user:server |
+| userapi_devices | devices | direct map |
+| roomserver_rooms | rooms | room_id, room_version; creator from m.room.create events |
+| roomserver_events + event_json | events + event_json | denormalize NIDs, topological_ordering=depth |
+| syncapi_current_room_state | current_state_events | direct map |
+| syncapi_current_room_state (member) | room_memberships + local_current_membership | |
+| mediaapi_media_repository | local_media_repository | media_id, type, size, upload_name, user_id |
+| mediaapi_thumbnail | local_media_repository_thumbnails | |
+| syncapi_receipts | receipts_linearized + receipts_graph | partial unique index for NULL thread_id |
+| roomserver_redactions | redactions | |
+
+## Tasks
+
+### Phase 0: Setup
+- [x] Explore Dendrite schema
+- [x] Explore Synapse schema
+- [x] Create migration plan
+- [x] Create script skeleton with connection handling + CLI args
+
+### Phase 1: Users & Profiles
+- [x] Migrate userapi_accounts -> users (created_ts ms->s conversion)
+- [x] Migrate userapi_profiles -> profiles (user_id=localpart, full_user_id=@user:server)
+- [x] Migrate userapi_devices -> devices
+- [x] Tested: 1194 users, 1194 profiles, 13 devices
+
+### Phase 2: Rooms
+- [x] Migrate roomserver_rooms -> rooms
+- [x] Extract room creator from m.room.create events
+- [x] Migrate roomserver_room_aliases -> room_aliases + room_alias_servers
+- [x] Tested: 492 rooms, correct creators
+
+### Phase 3: Events (Core)
+- [x] Build event_type NID->string and state_key NID->string lookups
+- [x] Migrate events with denormalized types/state_keys
+- [x] stream_ordering = global sequential, topological_ordering = depth
+- [x] internal_metadata = "{}" (stream_ordering/outlier read from events columns)
+- [x] format_version mapped from room version (v1-2->1, v3->2, v4-10->3, v11+->4)
+- [x] processed = True for migrated events
+- [x] Migrate event_json with correct format
+- [x] Populate state_events (events where state_key IS NOT NULL)
+- [x] Build event_edges from prev_events in event JSON
+- [x] Build event_auth from auth_events in event JSON
+- [x] Forward extremities from Dendrite's latest_event_nids
+- [x] room_depth from MIN(depth) per room
+- [x] Tested: 51474 events, 24609 state events, 489 fwd extremities
+
+### Phase 4: Room State
+- [x] current_state_events from syncapi_current_room_state
+- [x] Incremental state groups: one per state event, delta chains via state_group_edges
+- [x] All events mapped to correct state group via event_to_state_groups
+- [x] Tested: 24609 state groups, 51474 event mappings, 0 unmapped events
+
+### Phase 5: Membership
+- [x] Migrate from syncapi_current_room_state (type=m.room.member) -> room_memberships
+- [x] Populate local_current_membership for local users
+- [x] Include event_stream_ordering FK
+- [x] Tested: 7254 memberships, 3220 local memberships
+
+### Phase 6: Media
+- [x] Migrate mediaapi_media_repository -> local_media_repository
+- [x] Migrate mediaapi_thumbnail -> local_media_repository_thumbnails
+- [x] Copy content files: Dendrite `{base}/{hash[0]}/{hash[1]}/{hash[2:]}/file` -> Synapse `{base}/local_content/{id[0:2]}/{id[2:4]}/{id[4:]}`
+- [x] Copy thumbnails: Dendrite `thumbnail-{w}x{h}-{method}` -> Synapse `{w}-{h}-{top}-{sub}-{method}`
+- [x] Tested: 2747 media, 3309 thumbnails, file paths verified
+
+### Phase 7: Auxiliary Data
+- [x] Migrate receipts (receipts_linearized + receipts_graph, partial unique index)
+- [x] Migrate redactions
+- [x] Populate room_stats_current (member counts by type)
+- [x] Populate room_stats_state (room name, topic, encryption, etc.)
+- [x] Update events_stream_seq sequence
+- [x] Populate user_stats_current
+- [x] Tested: 857 receipts, 216 redactions, 492 room stats
+
+### Validation
+- [x] Synapse starts against migrated DB without errors
+- [x] Admin API: 488 rooms visible with correct names and member counts
+- [x] Messages accessible and readable via API
+- [x] Room state correct (creator, version, state types)
+- [x] Media metadata accessible via admin statistics API
+- [x] Background updates run normally post-migration
+
+## Findings / Issues Log
+- Dendrite event_state_key_nid 0 = not a state event, nid 1 = '' (empty string)
+- Dendrite event_type_nid preassigned: 1=m.room.create, 2=power_levels, 3=join_rules, 4=third_party_invite, 5=member, 6=redaction, 7=history_visibility
+- Synapse topological_ordering = depth (NOT a per-room counter)
+- Synapse internal_metadata JSON should be "{}" - stream_ordering and outlier loaded from events table columns
+- Synapse format_version: room v1-2=1, v3=2, v4-10=3, v11+=4
+- Synapse receipts_linearized has partial unique index WHERE thread_id IS NULL
+- Synapse room_alias_servers has no unique constraint - must check-before-insert
+- Synapse profiles unique on user_id (localpart), NOT on full_user_id
+- Forward extremities: use Dendrite's latest_event_nids, don't compute from graph
+- 2262 rejected events in Dendrite skipped during migration
+- 5548 orphan event edges (referencing federated events we don't have) - normal
+- Synapse background updates recalculate some stats after startup - normal
+
+## Usage
+
+```bash
+# Prerequisites: Synapse must be initialized first (creates schema)
+python3 -m synapse.app.homeserver --config-path homeserver.yaml  # start+stop once
+
+# Full migration
+python3 migrate.py \
+    --dendrite-db "dbname=dendrite host=/run/postgresql" \
+    --synapse-db "dbname=synapse host=/run/postgresql" \
+    --server-name "example.com" \
+    --dendrite-media-path /var/lib/dendrite/media \
+    --synapse-media-path /var/lib/synapse/media_store \
+    --phase 1,2,3,4,5,6,7
+
+# Selective phases (e.g., just re-run media)
+python3 migrate.py ... --phase 6
+
+# Dry run (no commits)
+python3 migrate.py ... --dry-run
+```
@@ -0,0 +1,303 @@
+-- Dendrite test schema + seed data
+-- This creates the minimum Dendrite tables needed for migration testing
+
+-- === Sequences ===
+CREATE SEQUENCE IF NOT EXISTS roomserver_room_nid_seq;
+CREATE SEQUENCE IF NOT EXISTS roomserver_event_nid_seq;
+CREATE SEQUENCE IF NOT EXISTS roomserver_event_type_nid_seq START 65536;
+CREATE SEQUENCE IF NOT EXISTS roomserver_event_state_key_nid_seq START 65536;
+CREATE SEQUENCE IF NOT EXISTS roomserver_state_snapshot_nid_seq;
+CREATE SEQUENCE IF NOT EXISTS roomserver_state_block_nid_seq;
+CREATE SEQUENCE IF NOT EXISTS syncapi_stream_id;
+CREATE SEQUENCE IF NOT EXISTS syncapi_receipt_id;
+
+-- === User tables ===
+CREATE TABLE IF NOT EXISTS userapi_accounts (
+    localpart TEXT NOT NULL,
+    server_name TEXT NOT NULL,
+    created_ts BIGINT NOT NULL,
+    password_hash TEXT,
+    appservice_id TEXT,
+    is_deactivated BOOLEAN DEFAULT FALSE,
+    account_type SMALLINT NOT NULL,
+    UNIQUE (localpart, server_name)
+);
+
+CREATE TABLE IF NOT EXISTS userapi_profiles (
+    localpart TEXT NOT NULL,
+    server_name TEXT NOT NULL,
+    display_name TEXT,
+    avatar_url TEXT,
+    UNIQUE (localpart, server_name)
+);
+
+CREATE TABLE IF NOT EXISTS userapi_devices (
+    access_token TEXT NOT NULL PRIMARY KEY,
+    device_id TEXT NOT NULL,
+    localpart TEXT NOT NULL,
+    server_name TEXT NOT NULL,
+    created_ts BIGINT NOT NULL,
+    display_name TEXT,
+    last_seen_ts BIGINT NOT NULL,
+    ip TEXT,
+    user_agent TEXT,
+    UNIQUE (localpart, server_name, device_id)
+);
+
+-- === Room tables ===
+CREATE TABLE IF NOT EXISTS roomserver_rooms (
+    room_nid BIGINT PRIMARY KEY DEFAULT nextval('roomserver_room_nid_seq'),
+    room_id TEXT NOT NULL UNIQUE,
+    latest_event_nids BIGINT[] NOT NULL DEFAULT '{}'::BIGINT[],
+    last_event_sent_nid BIGINT NOT NULL DEFAULT 0,
+    state_snapshot_nid BIGINT NOT NULL DEFAULT 0,
+    room_version TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS roomserver_room_aliases (
+    alias TEXT NOT NULL PRIMARY KEY,
+    room_id TEXT NOT NULL,
+    creator_id TEXT NOT NULL
+);
+
+-- === Event type/state key lookup tables ===
+CREATE TABLE IF NOT EXISTS roomserver_event_types (
+    event_type_nid BIGINT PRIMARY KEY DEFAULT nextval('roomserver_event_type_nid_seq'),
+    event_type TEXT NOT NULL UNIQUE
+);
+
+-- Preassigned event types
+INSERT INTO roomserver_event_types (event_type_nid, event_type) VALUES
+    (1, 'm.room.create'),
+    (2, 'm.room.power_levels'),
+    (3, 'm.room.join_rules'),
+    (4, 'm.room.third_party_invite'),
+    (5, 'm.room.member'),
+    (6, 'm.room.redaction'),
+    (7, 'm.room.history_visibility')
+ON CONFLICT DO NOTHING;
+
+CREATE TABLE IF NOT EXISTS roomserver_event_state_keys (
+    event_state_key_nid BIGINT PRIMARY KEY DEFAULT nextval('roomserver_event_state_key_nid_seq'),
+    event_state_key TEXT NOT NULL UNIQUE
+);
+
+-- Preassigned: nid 1 = empty string
+INSERT INTO roomserver_event_state_keys (event_state_key_nid, event_state_key) VALUES (1, '')
+ON CONFLICT DO NOTHING;
+
+-- === Event tables ===
+CREATE TABLE IF NOT EXISTS roomserver_events (
+    event_nid BIGINT PRIMARY KEY DEFAULT nextval('roomserver_event_nid_seq'),
+    room_nid BIGINT NOT NULL,
+    event_type_nid BIGINT NOT NULL,
+    event_state_key_nid BIGINT NOT NULL,
+    sent_to_output BOOLEAN NOT NULL DEFAULT FALSE,
+    state_snapshot_nid BIGINT NOT NULL DEFAULT 0,
+    depth BIGINT NOT NULL,
+    event_id TEXT NOT NULL UNIQUE,
+    auth_event_nids BIGINT[] NOT NULL,
+    is_rejected BOOLEAN NOT NULL DEFAULT FALSE
+);
+
+CREATE TABLE IF NOT EXISTS roomserver_event_json (
+    event_nid BIGINT NOT NULL PRIMARY KEY,
+    event_json TEXT NOT NULL
+);
+
+-- === Membership ===
+CREATE TABLE IF NOT EXISTS roomserver_membership (
+    room_nid BIGINT NOT NULL,
+    target_nid BIGINT NOT NULL,
+    sender_nid BIGINT NOT NULL DEFAULT 0,
+    membership_nid BIGINT NOT NULL DEFAULT 1,
+    event_nid BIGINT NOT NULL DEFAULT 0,
+    target_local BOOLEAN NOT NULL DEFAULT FALSE,
+    forgotten BOOLEAN NOT NULL DEFAULT FALSE,
+    UNIQUE (room_nid, target_nid)
+);
+
+-- === Sync API tables ===
+CREATE TABLE IF NOT EXISTS syncapi_current_room_state (
+    room_id TEXT NOT NULL,
+    event_id TEXT NOT NULL,
+    type TEXT NOT NULL,
+    sender TEXT NOT NULL,
+    contains_url BOOL NOT NULL,
+    state_key TEXT NOT NULL,
+    headered_event_json TEXT NOT NULL,
+    membership TEXT,
+    added_at BIGINT,
+    history_visibility SMALLINT NOT NULL DEFAULT 2,
+    UNIQUE (room_id, type, state_key)
+);
+
+CREATE TABLE IF NOT EXISTS syncapi_receipts (
+    id BIGINT PRIMARY KEY DEFAULT nextval('syncapi_receipt_id'),
+    room_id TEXT NOT NULL,
+    receipt_type TEXT NOT NULL,
+    user_id TEXT NOT NULL,
+    event_id TEXT NOT NULL,
+    receipt_ts BIGINT NOT NULL,
+    UNIQUE (room_id, receipt_type, user_id)
+);
+
+-- === Redactions ===
+CREATE TABLE IF NOT EXISTS roomserver_redactions (
+    redaction_event_id TEXT PRIMARY KEY,
+    redacts_event_id TEXT NOT NULL,
+    validated BOOLEAN NOT NULL
+);
+
+-- === Media ===
+CREATE TABLE IF NOT EXISTS mediaapi_media_repository (
+    media_id TEXT NOT NULL,
+    media_origin TEXT NOT NULL,
+    content_type TEXT NOT NULL,
+    file_size_bytes BIGINT NOT NULL,
+    creation_ts BIGINT NOT NULL,
+    upload_name TEXT NOT NULL,
+    base64hash TEXT NOT NULL,
+    user_id TEXT NOT NULL,
+    UNIQUE (media_id, media_origin)
+);
+
+CREATE TABLE IF NOT EXISTS mediaapi_thumbnail (
+    media_id TEXT NOT NULL,
+    media_origin TEXT NOT NULL,
+    content_type TEXT NOT NULL,
+    file_size_bytes BIGINT NOT NULL,
+    creation_ts BIGINT NOT NULL,
+    width INTEGER NOT NULL,
+    height INTEGER NOT NULL,
+    resize_method TEXT NOT NULL,
+    UNIQUE (media_id, media_origin, width, height, resize_method)
+);
+
+-- =============================================
+-- SEED TEST DATA
+-- =============================================
+
+-- Users
+INSERT INTO userapi_accounts (localpart, server_name, created_ts, password_hash, account_type)
+VALUES
+    ('alice', 'test.local', 1700000000000, '$2a$12$fakehash_alice', 1),
+    ('bob', 'test.local', 1700000100000, '$2a$12$fakehash_bob', 1),
+    ('admin', 'test.local', 1699999000000, '$2a$12$fakehash_admin', 3);
+
+INSERT INTO userapi_profiles (localpart, server_name, display_name, avatar_url)
+VALUES
+    ('alice', 'test.local', 'Alice Wonderland', 'mxc://test.local/alice_avatar'),
+    ('bob', 'test.local', 'Bob Builder', NULL),
+    ('admin', 'test.local', 'Admin', NULL);
+
+INSERT INTO userapi_devices (access_token, device_id, localpart, server_name, created_ts, display_name, last_seen_ts, ip, user_agent)
+VALUES
+    ('token_alice_1', 'DEVICE_A1', 'alice', 'test.local', 1700000000000, 'Alice Phone', 1700100000000, '192.168.1.10', 'Element/1.0'),
+    ('token_bob_1', 'DEVICE_B1', 'bob', 'test.local', 1700000100000, 'Bob Laptop', 1700100000000, '192.168.1.11', 'Element/1.0');
+
+-- Rooms
+INSERT INTO roomserver_rooms (room_nid, room_id, room_version) VALUES
+    (1, '!room1:test.local', '10'),
+    (2, '!room2:test.local', '10');
+
+-- Additional event type for m.room.message
+INSERT INTO roomserver_event_types (event_type_nid, event_type) VALUES (65536, 'm.room.message') ON CONFLICT DO NOTHING;
+-- Event type for m.room.name
+INSERT INTO roomserver_event_types (event_type_nid, event_type) VALUES (65537, 'm.room.name') ON CONFLICT DO NOTHING;
+
+-- State keys for users
+INSERT INTO roomserver_event_state_keys (event_state_key_nid, event_state_key) VALUES
+    (65536, '@alice:test.local'),
+    (65537, '@bob:test.local')
+ON CONFLICT DO NOTHING;
+
+-- Room aliases
+INSERT INTO roomserver_room_aliases (alias, room_id, creator_id) VALUES
+    ('#general:test.local', '!room1:test.local', '@alice:test.local');
+
+-- === Events for Room 1 ===
+
+-- m.room.create event
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (1, 1, 1, 1, 1, '$create_room1', '{}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (1, '{"event_id":"$create_room1","room_id":"!room1:test.local","type":"m.room.create","sender":"@alice:test.local","state_key":"","origin_server_ts":1700000001000,"content":{"creator":"@alice:test.local","room_version":"10"},"auth_events":[],"prev_events":[]}');
+
+-- m.room.member join alice
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (2, 1, 5, 65536, 2, '$join_alice_room1', '{1}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (2, '{"event_id":"$join_alice_room1","room_id":"!room1:test.local","type":"m.room.member","sender":"@alice:test.local","state_key":"@alice:test.local","origin_server_ts":1700000002000,"content":{"membership":"join","displayname":"Alice Wonderland"},"auth_events":["$create_room1"],"prev_events":["$create_room1"]}');
+
+-- m.room.member join bob
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (3, 1, 5, 65537, 3, '$join_bob_room1', '{1,2}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (3, '{"event_id":"$join_bob_room1","room_id":"!room1:test.local","type":"m.room.member","sender":"@bob:test.local","state_key":"@bob:test.local","origin_server_ts":1700000003000,"content":{"membership":"join","displayname":"Bob Builder"},"auth_events":["$create_room1","$join_alice_room1"],"prev_events":["$join_alice_room1"]}');
+
+-- m.room.name
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (4, 1, 65537, 1, 4, '$name_room1', '{1,2}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (4, '{"event_id":"$name_room1","room_id":"!room1:test.local","type":"m.room.name","sender":"@alice:test.local","state_key":"","origin_server_ts":1700000004000,"content":{"name":"General Chat"},"auth_events":["$create_room1","$join_alice_room1"],"prev_events":["$join_bob_room1"]}');
+
+-- m.room.message from alice
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (5, 1, 65536, 0, 5, '$msg1_room1', '{1,2}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (5, '{"event_id":"$msg1_room1","room_id":"!room1:test.local","type":"m.room.message","sender":"@alice:test.local","origin_server_ts":1700000010000,"content":{"msgtype":"m.text","body":"Hello everyone!"},"auth_events":["$create_room1","$join_alice_room1"],"prev_events":["$name_room1"]}');
+
+-- m.room.message from bob
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (6, 1, 65536, 0, 6, '$msg2_room1', '{1,3}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (6, '{"event_id":"$msg2_room1","room_id":"!room1:test.local","type":"m.room.message","sender":"@bob:test.local","origin_server_ts":1700000020000,"content":{"msgtype":"m.text","body":"Hi Alice!"},"auth_events":["$create_room1","$join_bob_room1"],"prev_events":["$msg1_room1"]}');
+
+-- m.room.message with URL
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (7, 1, 65536, 0, 7, '$msg3_room1', '{1,2}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (7, '{"event_id":"$msg3_room1","room_id":"!room1:test.local","type":"m.room.message","sender":"@alice:test.local","origin_server_ts":1700000030000,"content":{"msgtype":"m.image","body":"photo.jpg","url":"mxc://test.local/media123"},"auth_events":["$create_room1","$join_alice_room1"],"prev_events":["$msg2_room1"]}');
+
+-- === Events for Room 2 (small DM) ===
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (8, 2, 1, 1, 1, '$create_room2', '{}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (8, '{"event_id":"$create_room2","room_id":"!room2:test.local","type":"m.room.create","sender":"@bob:test.local","state_key":"","origin_server_ts":1700001000000,"content":{"creator":"@bob:test.local","room_version":"10"},"auth_events":[],"prev_events":[]}');
+
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids) VALUES
+    (9, 2, 5, 65537, 2, '$join_bob_room2', '{8}');
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (9, '{"event_id":"$join_bob_room2","room_id":"!room2:test.local","type":"m.room.member","sender":"@bob:test.local","state_key":"@bob:test.local","origin_server_ts":1700001001000,"content":{"membership":"join","displayname":"Bob Builder"},"auth_events":["$create_room2"],"prev_events":["$create_room2"]}');
+
+-- A rejected event (should be skipped)
+INSERT INTO roomserver_events (event_nid, room_nid, event_type_nid, event_state_key_nid, depth, event_id, auth_event_nids, is_rejected) VALUES
+    (10, 1, 65536, 0, 8, '$rejected_event', '{1}', TRUE);
+INSERT INTO roomserver_event_json (event_nid, event_json) VALUES
+    (10, '{"event_id":"$rejected_event","room_id":"!room1:test.local","type":"m.room.message","sender":"@evil:other.server","origin_server_ts":1700000050000,"content":{"msgtype":"m.text","body":"spam"},"auth_events":["$create_room1"],"prev_events":["$msg3_room1"]}');
+
+-- === Syncapi current room state ===
+INSERT INTO syncapi_current_room_state (room_id, event_id, type, sender, contains_url, state_key, headered_event_json, membership) VALUES
+    ('!room1:test.local', '$create_room1', 'm.room.create', '@alice:test.local', false, '', '{"event_id":"$create_room1","room_id":"!room1:test.local","type":"m.room.create","sender":"@alice:test.local","state_key":"","content":{"creator":"@alice:test.local","room_version":"10"}}', NULL),
+    ('!room1:test.local', '$join_alice_room1', 'm.room.member', '@alice:test.local', false, '@alice:test.local', '{"event_id":"$join_alice_room1","room_id":"!room1:test.local","type":"m.room.member","sender":"@alice:test.local","state_key":"@alice:test.local","content":{"membership":"join","displayname":"Alice Wonderland"}}', 'join'),
+    ('!room1:test.local', '$join_bob_room1', 'm.room.member', '@bob:test.local', false, '@bob:test.local', '{"event_id":"$join_bob_room1","room_id":"!room1:test.local","type":"m.room.member","sender":"@bob:test.local","state_key":"@bob:test.local","content":{"membership":"join","displayname":"Bob Builder"}}', 'join'),
+    ('!room1:test.local', '$name_room1', 'm.room.name', '@alice:test.local', false, '', '{"event_id":"$name_room1","room_id":"!room1:test.local","type":"m.room.name","sender":"@alice:test.local","state_key":"","content":{"name":"General Chat"}}', NULL),
+    ('!room2:test.local', '$create_room2', 'm.room.create', '@bob:test.local', false, '', '{"event_id":"$create_room2","room_id":"!room2:test.local","type":"m.room.create","sender":"@bob:test.local","state_key":"","content":{"creator":"@bob:test.local","room_version":"10"}}', NULL),
+    ('!room2:test.local', '$join_bob_room2', 'm.room.member', '@bob:test.local', false, '@bob:test.local', '{"event_id":"$join_bob_room2","room_id":"!room2:test.local","type":"m.room.member","sender":"@bob:test.local","state_key":"@bob:test.local","content":{"membership":"join","displayname":"Bob Builder"}}', 'join');
+
+-- === Receipts ===
+INSERT INTO syncapi_receipts (room_id, receipt_type, user_id, event_id, receipt_ts) VALUES
+    ('!room1:test.local', 'm.read', '@alice:test.local', '$msg2_room1', 1700000025000),
+    ('!room1:test.local', 'm.read', '@bob:test.local', '$msg3_room1', 1700000035000);
+
+-- === Redactions ===
+-- (none in test data, but table exists)
+
+-- === Media ===
+INSERT INTO mediaapi_media_repository (media_id, media_origin, content_type, file_size_bytes, creation_ts, upload_name, base64hash, user_id) VALUES
+    ('media123', 'test.local', 'image/jpeg', 12345, 1700000030000, 'photo.jpg', 'abc123def456ghi789', '@alice:test.local'),
+    ('media456', 'test.local', 'application/pdf', 98765, 1700000040000, 'document.pdf', 'xyz789abc123def456', '@bob:test.local');
+
+INSERT INTO mediaapi_thumbnail (media_id, media_origin, content_type, file_size_bytes, creation_ts, width, height, resize_method) VALUES
+    ('media123', 'test.local', 'image/jpeg', 3000, 1700000030000, 320, 240, 'scale');