Skip to content

Conversation

@jogrogan
Copy link
Collaborator

@jogrogan jogrogan commented Apr 7, 2025

Supports automatic connector hint passthrough without the need for templates.

Format is expected as <engine-connector-name>.<source|sink>.<connectorConfigName>

See README for more information

Also fixed a bug when applying sqljob yaml for configs where it would complain about ints or booleans because it wasn't surrounded in single quotes.

@ryannedolan
Copy link
Collaborator

<schema>.<source|sink>.<connectorConfigName>

Connector configs depend on the connector, which is hard to predict and impossible to control based on schema. For example, say KAFKA schema is implemented by both kafka and iceberg connectors, or by kafka in streaming mode and iceberg in batch mode. The user specifying KAFKA.sink.foo could not be sure foo will get passed to a Kafka connector -- only that it will get passed to whatever is configured by the KAFKA table/job template(s). In particular, their hint could easily get passed to an iceberg connector instead.

This creates a problem, since Flink connectors will generally throw an error if they see a configuration they don't understand. The kafka connector's property.x.y option is an exception to this rule -- the Kafka connector just passes these configs through to the Kafka client. So this works fine for KAFKA.sink.property.client.id, for example, even though there is no matching connector config. But that same config would break if it were passed to a more typical connector.

An alternative might be to use the connector name rather than the schema name as a prefix. That ensures we only pass iceberg.source.foo.bar to an iceberg source connector, for example. This also feels more "hinty" to me, cuz hints will just get ignored if they aren't relevant.

Does that work?

Copy link
Collaborator

@ryannedolan ryannedolan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

neat!

@jogrogan jogrogan merged commit c7fa3e8 into main Apr 8, 2025
1 check passed
@jogrogan jogrogan deleted the jogrogan/connectorHints branch April 8, 2025 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants