Open Source Data Platform Engineer specialising in Apache Airflow & dbt orchestration.
Apache Airflow Committer · Astronomer Cosmos Committer · Conference speaker & workshop host on data orchestration and OSS platforms
Helping teams design, scale, and operate reliable data pipelines.
- Design & build production-grade data orchestration platforms
- Contribute to and help maintain the Apache Airflow ecosystem
- Help companies adopt dbt + Airflow correctly (architecture, performance, reliability)
- Advise teams on OSS strategy, integrations, and platform maturity
- Orchestration: Apache Airflow (core + providers)
- Data Engineering: dbt, SQL, analytics engineering
- Platforms: Kubernetes, Docker, Cloud (AWS / GCP / Azure)
- Languages: Python (primary), SQL
- Open Source: Design, reviews, long-term maintenance
I actively contribute to and help maintain core projects in the Apache Airflow ecosystem, with a strong focus on production reliability, performance, and developer experience.
Currently working on the Apache Airflow ecosystem at Astronomer.
-
Apache Airflow
Contributions across operators, providers, security, and performance improvements https://github.com/apache/airflow -
Astronomer Cosmos (dbt ↔ Airflow integration)
Contributor to execution modes, asynchronous patterns, remote caching, and dbt orchestration design https://github.com/astronomer/astronomer-cosmos -
Dag Factory
Improving DAG generation patterns, usability, and maintainability
https://github.com/astronomer/dag-factory -
Astronomer Providers
Enhancements and fixes across Airflow providers for real-world asynchronous production use cases
https://github.com/astronomer/astronomer-providers -
Great Expectations Airflow Provider
Contributions towards CI pipelines, validation workflows, and Airflow integration https://github.com/great-expectations/airflow-provider-great-expectations
-
Boosting dbt-Core Workflow Performance with Airflow’s Deferrable Capabilities
Airflow Summit 2025 — Talk
https://airflowsummit.org/sessions/2025/boosting-dbt-core-workflows-performance-with-airflows-deferrable-capabilities/ -
Productionising dbt-Core in Airflow
Airflow Summit 2025 — Workshop
https://airflowsummit.org/sessions/2025/productionising-dbt-core/
-
Overcoming Performance Hurdles in Integrating dbt With Airflow
Airflow Summit 2024 — Talk
https://airflowsummit.org/sessions/2024/overcoming-performance-hurdles-in-integrating-dbt-with-airflow/ -
Optimizing Airflow Performance: Strategies, Techniques & Best Practices
Airflow Summit 2024 — Talk
https://airflowsummit.org/sessions/2024/optimizing-airflow-performance-strategies-techniques-and-best-practices/
📩 Reach out:
Always happy to discuss architecture, open-source collaboration, and data orchestration challenges.
- LinkedIn: https://www.linkedin.com/in/pankajkoti
- Email: [email protected]