Various bugfixes - #121
Conversation
|
I see now that |
|
You work fast! However, I doubt I'm going to test the referenced changes, as I have enough experience with CDC now (particularly with mysql/mariadb) to know it's very tricky, so I have to figure that there's still all sorts of edge cases that are not being handled yet and won't be found til people stumble on and report them. Conversely, Debezium has a decade of scrutiny and improvements behind it, that come with being the gold standard for CDC. The mariadb test suite alone takes 25 min to run (I did so yesterday while adding v12.3 support). And it will have a simple native binary available soon. Plus, benthos makes it easy for me to do custom processing and custom surrealdb output connector. But I'm happy to provide feedback/brainstorm in #118 if you have any questions on use cases etc. |
This is purely vibe coded. I didnt even look at the code. You probably only want to look at it as reference. But I am now able to connect mysql and mariadb, and they sync some basic data.
It fixes 3 issues:
Fix 1: MySQL 8.0
caching_sha2_passwordRSA auth (wire.rs)negotiate_tls: For--tls-mode preferred, skip TLS entirely instead of trying and failing on self-signed certs. The RSA-based full auth path handles the password exchange.caching_sha2_passwordfull auth flow: request the RSA public key from MySQL, XOR the null-terminated password withSHA256(scramble), RSA-OAEP encrypt with SHA-1, and send.auth_plugin_data_lenincludes the trailing NUL byte, so we now subtract 1 to get the real 20-byteSCRAMBLE_LENGTHinstead of reading 21 bytes.Fix 2: MySQL zero dates (
0000-00-00) (reverse.rs + binlog.rs)extract_date,extract_time,extract_datetimecalls in the full_sync path to return null values instead of crashing when MySQL emits zero dates (e.g., WordPress auto-drafts withpost_date_gmt = 0000-00-00 00:00:00).Fix 3: Composite primary keys (rows.rs +
surreal2-sink/src/rows.rs)UniversalValue::Arrayhandling touniversal_value_to_surreal_id— tables with composite primary keys (likewp_term_relationshipswith(object_id, term_taxonomy_id)) now get a string record ID like"4_2"instead of crashing.I think you are definitely going to want to test all db sources with extreme (max, min, null, invalid) values for all data types. This is something I spent a significant amount of time doing when I initially built my own mysql/mariadb -> surreal stuff with Conduit then Benthos. I did it all via code in a private repo, so I dont have a sample SQL file that I can just share as a starting point with all the extreme values. I hope this helps