Add PostgreSQL pgoutput source with interleaved snapshot and replication tail - #108
Merged
Conversation
Introduce pgoutput-protocol and postgresql-wal-source crates, wire the source into the CLI, and add integration and e2e tests with documentation.
Bind BIGINT keyset cursors as int8 instead of narrowing to int4, and defer replication slot advancement until every peeked wal2json change is surfaced so concurrent writes are not dropped during interleaved snapshots.
Interleaved snapshot dedup widens integer primary keys to Int64; narrow cursors back to int4/int2 before keyset chunk reads on SERIAL tables.
Align crate, CLI (`from postgresql-pgoutput`), docs, and checkpoint database_type naming with the wal2json source pattern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces a PostgreSQL CDC source that streams changes through the built-in
pgoutputplugin, with DBLog-style interleaved snapshot and a long-lived replication tail on the same WAL stream, matching the operational model of the MySQL binlog source #105Please see
docs/postgresql-pgoutput-source.mdfor the usage!