Conversation
…andling, shared e2e helpers, and documentation. Reuse the mysql-trigger-source pipeline with MariaDB-specific schema and type fixes, container test support, and full/incremental integration tests.
|
I don't know if any of this is useful or relevant, but there's a nuance with mariadb CDC stuff on more recent versions of mariadb. It's described in this issue conduitio-labs/conduit-connector-mysql#136 I ended up instigating a fix for the popular golang mysql CDC package. Perhaps some useful info there to double check what you have in rust. |
|
@nickchomey Thanks for the input! I think the link you kindly provided is to leverage MariaDB's own binlog. Unlike the PostgreSQL source, our MySQL/MariaDB source supports only trigger-based mode today, so the natural next step would be to implement binlog-based modes for those. I'll refer to it when tackling it! |
|
Oh, I wasn't aware that it was trigger based. Are you able to elaborate a bit on why? I have to figure there's already rust crates for binlog CDC |
|
Hey @nickchomey 👋 I got started with the trigger-based solution because it is more widely available than binlog, and I found no reliable crate for reading/decoding binlog. Anyways, I've added binlog-based MariaDB/MySQL sources that do DBLog-like watermark-based snapshotting and reconciliation using our own binlog reader/decoder via #105. I believe it should now be the go-to option for syncing MySQL/MariaDB to SurrealDB. I didn't pay much attention to making it reusable outside this project, but hopefully, it is good enough to serve as a reference. |
What is the motivation?
MariaDB is a common MySQL-compatible source; users should sync it with the same
from mysqlworkflow.What does this change do?
docs/mariadb.md)mysql://,from mysqlsubcommands)LONGTEXTWhat is your testing strategy?
make testmariadb:11mirror existing MySQL scenariosIs this related to any issues?
Closes #67
Have you read the Contributing Guidelines?