Skip to content

feat: add compact payload mode#196

Merged
Emrehzl94 merged 7 commits intoneo4j:mainfrom
Emrehzl94:compact-payload-support
Oct 29, 2024
Merged

feat: add compact payload mode#196
Emrehzl94 merged 7 commits intoneo4j:mainfrom
Emrehzl94:compact-payload-support

Conversation

@Emrehzl94
Copy link
Contributor

This PR adds a configuration property to select the type of payload mode to be used by the source connector for serializing the data that will be published. The configuration property, neo4j.payload-mode, has two options: EXTENDED and COMPACT. By default, the mode is set to EXTENDED.

COMPACT: Produces a simpler, more minimal format with just the essential fields.

Example of COMPACT payload with JsonSchema;

{
  "name": "mary",
  "surname": "doe",
  "timestamp": 1729779296311
}

EXTENDED: Includes additional information to handle schema changes and serialization issues of temporal types, ensuring consistency and preventing schema incompatibility issues.

Example of EXTENDED payload with JsonSchema;

{
  "name": {
    "type": "S",
    "B": null,
    "I64": null,
    "F64": null,
    "S": "mary",
    "BA": null,
    "TLD": null,
    "TLDT": null,
    "TLT": null,
    "TZDT": null,
    "TOT": null,
    "TD": null,
    "SP": null,
    "LB": null,
    "LI64": null,
    "LF64": null,
    "LS": null,
    "LTLD": null,
    "LTLDT": null,
    "LTLT": null,
    "LZDT": null,
    "LTOT": null,
    "LTD": null,
    "LSP": null
  },
  "surname": {
    "type": "S",
    "B": null,
    "I64": null,
    "F64": null,
    "S": "doe",
    "BA": null,
    "TLD": null,
    "TLDT": null,
    "TLT": null,
    "TZDT": null,
    "TOT": null,
    "TD": null,
    "SP": null,
    "LB": null,
    "LI64": null,
    "LF64": null,
    "LS": null,
    "LTLD": null,
    "LTLDT": null,
    "LTLT": null,
    "LZDT": null,
    "LTOT": null,
    "LTD": null,
    "LSP": null
  },
  "timestamp": {
    "type": "I64",
    "B": null,
    "I64": 1729779365447,
    "F64": null,
    "S": null,
    "BA": null,
    "TLD": null,
    "TLDT": null,
    "TLT": null,
    "TZDT": null,
    "TOT": null,
    "TD": null,
    "SP": null,
    "LB": null,
    "LI64": null,
    "LF64": null,
    "LS": null,
    "LTLD": null,
    "LTLDT": null,
    "LTLT": null,
    "LZDT": null,
    "LTOT": null,
    "LTD": null,
    "LSP": null
  }
}

@Emrehzl94 Emrehzl94 marked this pull request as ready for review October 24, 2024 14:35
@Emrehzl94 Emrehzl94 requested a review from a team as a code owner October 24, 2024 14:35
Copy link
Contributor

@fbiville fbiville left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nicely done! just a few minor questions for me

Co-authored-by: Florent Biville <445792+fbiville@users.noreply.github.com>
@neo4j-connectors
Copy link
Collaborator

neo4j-connectors commented Oct 25, 2024


Warnings
⚠️ Commit Message 'Update common/src/main/resources/neo4j-source-configuration.properties

Co-authored-by: Florent Biville 445792+fbiville@users.noreply.github.com': subject may not be empty, type may not be empty

Generated by 🚫 dangerJS against 91f71aa

@fbiville fbiville self-requested a review October 25, 2024 14:47
…kafka-connector into compact-payload-support

merge changes from branch
Copy link
Contributor

@ali-ince ali-ince left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just left a few minor comments.

Co-authored-by: Ali Ince <ali-ince@users.noreply.github.com>
@Emrehzl94 Emrehzl94 merged commit 4b575e7 into neo4j:main Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants