Назад към всички

surreal-sync

// Data migration and synchronization to SurrealDB from MongoDB, PostgreSQL, MySQL, Neo4j, Kafka, and JSONL. Full and incremental CDC sync. Part of the surreal-skills collection.

$ git log --oneline --stat
stars:1,933
forks:367
updated:March 4, 2026
SKILL.mdreadonly
SKILL.md Frontmatter
namesurreal-sync
descriptionData migration and synchronization to SurrealDB from MongoDB, PostgreSQL, MySQL, Neo4j, Kafka, and JSONL. Full and incremental CDC sync. Part of the surreal-skills collection.
licenseMIT
metadata[object Object]

Surreal-Sync -- Data Migration and Synchronization

Surreal-Sync is a CLI tool for migrating data from various database sources to SurrealDB with full and incremental synchronization via Change Data Capture (CDC).

Supported Sources

SourceFull SyncIncremental CDCMethod
MongoDBYesYesChange streams
MySQLYesYesTrigger-based CDC + sequence checkpoints
PostgreSQL (triggers)YesYesTrigger-based CDC + sequence checkpoints
PostgreSQL (wal2json)YesYesLogical replication with wal2json plugin
Neo4jYesYesTimestamp-based tracking
JSONL FilesYesN/ABatch import from JSON Lines
Apache KafkaYesYesConsumer subscriptions with deduplication

Quick Start

# Install surreal-sync (Rust binary)
cargo install surreal-sync

# Full sync from PostgreSQL (trigger-based)
surreal-sync from postgres trigger-full \
  --connection-string "postgresql://user:pass@localhost/mydb" \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Incremental CDC from PostgreSQL (wal2json)
surreal-sync from postgres wal2json \
  --connection-string "postgresql://user:pass@localhost/mydb" \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Full sync from MongoDB
surreal-sync from mongo full \
  --connection-string "mongodb://localhost:27017/mydb" \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Batch import from JSONL
surreal-sync from jsonl import \
  --file data.jsonl \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Consume from Kafka
surreal-sync from kafka consume \
  --bootstrap-servers "localhost:9092" \
  --topic my-events \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

CLI Pattern

surreal-sync from <SOURCE> <COMMAND> \
  --connection-string [CONNECTION STRING] \
  --surreal-endpoint [SURREAL ENDPOINT] \
  --surreal-username [SURREAL USERNAME] \
  --surreal-password [SURREAL PASSWORD] \
  --to-namespace <NS> \
  --to-database <DB>

Key Features

  • Automatic schema inference and SurrealDB table creation
  • Record ID mapping from source primary keys
  • Relationship extraction and graph edge creation
  • Configurable batch sizes and parallelism
  • Resumable sync with checkpoint tracking
  • Deduplication for Kafka consumers

Full Documentation

See the main skill's rule file for complete guidance: