Migrate remaining provider connection UI metadata to YAML #62345
Description
Background
As part of decoupling connection form UI metadata from runtime hook imports,
we are migrating ui-field-behaviour and conn-fields from Python hook
methods (get_ui_field_behaviour() / get_connection_form_widgets()) into
declarative provider.yaml files.
This allows ProvidersManager to load connection form metadata without
importing hook classes, improving startup performance and separating UI
concerns from runtime execution.
Progress so far
- Phase 1 — Google, Amazon, Microsoft Azure, Hashicorp, Databricks, Snowflake (PR: Migrate connection UI metadata to YAML for some providers #62011)
- Phase 2 — git, ssh, sftp, docker, cncf/kubernetes, standard, grpc, opensearch, datadog, influxdb (PR: [Part 2] Migrate connection UI metadata to YAML for more providers #62109)
- Phase 3 — slack, discord, pagerduty, opsgenie, smtp, telegram, apprise (PR: [Part 3] Migrate connection UI metadata to YAML for more providers #62165)
Remaining Providers
Remaining providers
| Provider | # Connection Types | Claimed by | Status |
|---|---|---|---|
| airbyte | 1 | Eason09053360 | Done |
| alibaba | 4 | anishgirianish | Done |
| apache/hive | 3 | anishgirianish | Done |
| apache/iceberg | 1 | wjddn279 | Done |
| apache/kafka | 1 | wjddn279 | Done |
| apache/spark | 4 | anishgirianish | Done |
| arangodb | 1 | yjaw | Done |
| asana | 1 | yjaw | Done |
| cloudant | 1 | yjaw | Done |
| cohere | 1 | sjyangkevin | Done |
| github | 1 | yuseok89 | Done |
| jdbc | 1 | Eason09053360 | Done |
| jenkins | 1 | Eason09053360 | Done |
| mongo | 1 | haseebmalik18 | Done |
| openai | 1 | yuseok89 | Done |
| openfaas | 1 | ahilashsasidharan | Done |
| pinecone | 1 | justinpakzad | Done |
| postgres | 1 | haseebmalik18 | Done |
| qdrant | 1 | fat-catTW | Done |
| redis | 1 | yuseok89 | Done |
| salesforce | 1 | haseebmalik18 | Done |
| samba | 1 | keisuke-kawamura | Done |
| teradata | 1 | Srabasti | Done |
| trino | 1 | sjyangkevin | Done |
| weaviate | 1 | henry3260 | Done |
| yandex | 1 | Prab-27 | Done |
| ydb | 1 | henry3260 | Done |
| zendesk | 1 | henry3260 | Done |
To claim a provider: leave a comment saying you're working on it to avoid duplicate work.
How to contribute?
Follow https://github.com/apache/airflow/blob/main/contributing-docs/23_provider_hook_migration_to_yaml.rst broadly.
- But in short you can do this in an venv with all airflow deps installed as per: https://github.com/apache/airflow/blob/main/contributing-docs/03a_contributors_quick_start_beginners.rst (cd into airflow repo and
uv sync --all-packagesshould do the trick) python scripts/tools/generate_yaml_format_for_hooks.py --provider <name>
Example:
python scripts/tools/generate_yaml_format_for_hooks.py --provider mongo
The script will print ui-field-behaviour and/or conn-fields for each connection type defined in the provider.
- Do not use the
update-yamlflag, manually make edits to the provider.yml to avoid errors - If you want to quickly verify if it works at runtime, try running this snippet:
from airflow.providers.<name>.get_provider_info import get_provider_info
for ct in get_provider_info().get('connection-types', []):
print(
ct.get('connection-type'),
'ui:', 'ui-field-behaviour' in ct,
'conn-fields:', list(ct.get('conn-fields', {}).keys())
)Once all the above instructions are done, open a PR with title: Migrate <name> connection UI metadata to YAML
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.