Status of testing of Apache Airflow 2.5.3rc2

[pierrejeambrun]

We have a kind request for all the contributors to the latest Apache Airflow RC 2.5.3rc2.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

• Fix warning in airflow tasks test command regarding absence of data_interval (#27106): @tirkarthi
• Strip markup from app_name if instance_name_has_markup = True (#28894): @IAL32
• fix update_mask in patch variable route (#29711): @PannagaMayya @hussein-awala
  Linked issues:
  • Updating Variables Description via PATCH in Airflow API is Clearing the existing description field of the variable and unable to update description field (#29702)
• Param argument accepts generic dict subtypes (#29782): @jcobb-healx
  Linked issues:
  • Cannot use TypedDict object when defining params (#29358)
• Fixing broken filter in /taskinstance/list view (#29850): @SamWheating
• Including airflow/example_dags/sql/sample.sql in MANIFEST.in (#29883): @SamWheating
• Remove accidentally merged vendor daemon patch code (#29895): @potiuk
  Linked issues:
  • Move help message to the google auth code (#29888)
• Fix tags selection in DAGs UI (#29944): @vemikhaylov
• fix dag docs toggle icon initial angle (#29970): @aaronsatae
• Fix some long known Graph View UI problems (#29971): @yuqian90
  Linked issues:
  • Fix graph zoom scale (#30355) @bbovenzi
  • Restore graph expand/collapse scale (#30360) @bbovenzi
• Avoid considering EmptyOperator in mini scheduler (#29979): @eladkal
• Fix db clean command for mysql db (#29999): @tanvn
• Remove upper bound limit of astroid (#30033): @auvipy
• dag processor manager, add retry_db_transcation to _fetch_callbacks (#30079): @michaelmicheal
• Fix TriggerRuleDep when the mapped tasks count is 0 (#30084): @hussein-awala @badrisrinivas9
  Linked issues:
  • Task group expand fails on empty list at get_relevant_upstream_map_indexes (#30073)
• Fix www/utils.dag_run_link redirection (#30098): @meujin
• Format copy of dag run config in grid detail (#30103): @jakubno
• Fix inconsistent returned value of airflow dags next-execution cli command (#30117): @boushphong @frankbreetz
  Linked issues:
  • CLI command "airflow dags next-execution" give unexpected results with paused DAG and catchup=False (#22474)
• Ensure that dag.partial_subset doesn't mutate task group properties (#30129): @milczar @ashb
  Linked issues:
  • [Graph view] After clearing the task (and its downstream tasks) in a task group the task group becomes disconnected from the dag (#26059)
• Add custom_operator_name to @task.sensor tasks (#30131): @josh-fell
• Move read only property in order to fix Dagrun API docs (#30149): @SamWheating
• Revert fix for on_failure_callback when task receives a SIGTERM (#30165): @ephraimbuddy
• Correct the VARCHAR size to 250. (#30178): @junho85
• Fix XCom.get_one exactly one exception text (#30183): @vemikhaylov
• Fix proper termination of gunicorn when it hangs (#30188): @potiuk @mik-laj
  Linked issues:
  • Improve web server stopping (#11734)
• Revert "Also limit importlib on Python 3.9 (#30069)" (#30209): @potiuk
• Fix DagProcessorJob integration for standalone dag-processor (#30278): @antoniocorralsierra @potiuk @farhan0syakir
  Linked issues:
  • DagProcessor restart constantly when it running as standalone process (#30251)
  • introduce dag processor job fix #27140 (#28799)

Thanks to all who contributed to the release (probably not a complete list!):
@meujin @josh-fell @vemikhaylov @ashb @tirkarthi @farhan0syakir @auvipy @antoniocorralsierra @junho85 @SamWheating @PannagaMayya @mik-laj @IAL32 @eladkal @ephraimbuddy @aaronsatae @potiuk @yuqian90 @frankbreetz @milczar @badrisrinivas9 @jakubno @michaelmicheal @boushphong @hussein-awala @jcobb-healx @tanvn

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[raphaelauv]

the graph view zoom on the first task , it's super weird

if I unzoom it look better

[antoniocorralsierra]

DagProcess running as standalone process is working fine, the restart have disappeared. Congrats!!

[pierrejeambrun]

@raphaelauv I think it comes from #29971 which remove the 1.5 max zoomScale cap. (I would say that this is the indented behavior, I agree, it's really big, I'm not a big fan), but the fact that some nodes can be missing from the first focus is not great.

@potiuk @ephraimbuddy @bbovenzi, Should we fix that and go for rc2 ? (I think yes)

[potiuk]

@raphaelauv I think it comes from #29971 which remove the 1.5 max zoomScale cap. (I would say that this is the indented behavior, I agree, it's really big, I'm not a big fan), but the fact that some nodes can be missing from the first focus is not great.

@potiuk @ephraimbuddy @bbovenzi, Should we fix that and go for rc2 ? (I think yes)

I think it is more of an easthetic thing / UX than technical issue. The question is (I have not looked at it yet) - is it something that will happen with every graph every time or only with some almost-empty-non-connected-and-really-non-realistic graphs like this one above? If the former - yes, we should fix it IMHO, if the latter - I think we do not have to fix it.

[hussein-awala]

I tested:

• fix update_mask in patch variable route #29711
• Fix TriggerRuleDep when the mapped tasks count is 0 #30084

and they both work as expected.

But I agree with @raphaelauv about the graph default zoom level, it's weird and annoying.

The question is (I have not looked at it yet) - is it something that will happen with every graph every time or only with some almost-empty-non-connected-and-really-non-realistic graphs like this one above?

@potiuk it's the former, I have this zoom level on all the graphs for all the dag and dag runs. IMO we should fix it and go for rc2.

[potiuk]

@potiuk it's the former, I have this zoom level on all the graphs for all the dag and dag runs. IMO we should fix it and go for rc2

Then yes.

[ephraimbuddy]

@potiuk it's the former, I have this zoom level on all the graphs for all the dag and dag runs. IMO we should fix it and go for rc2

Then yes.

Agree too

[bbovenzi]

Fix: #30355

[pierrejeambrun]

Due to the graph zoom issue in rc1, rc2 has been made. I think we can safely assume that what was tested for rc1 is still working in rc2. Feel free to do more testing. The issue has been updated to target 2.5.3rc2

[potiuk]

• Remove accidentally merged vendor daemon patch code #29895: fine - the files are not there after installations

• Fix proper termination of gunicorn when it hangs #30188: hard to test that one but basic killing works (and I have not seen a single flaky failure in our CI because of this one) . I am a bit surprised though I do not see Fix process killing with subprocess API (Fix process killing with subprocess API #30207)

• which is actually a fix to my fix - neither on the list of issues here nor in release notes. The change is cherry-picked and technically it's a fix to a fix, so no big deal. Just wonder how it was lost from both release notes and issue status here.

• Revert "Also limit importlib on Python 3.9 (#30069)" #30209 -> correct. METADATA is good now: Requires-Dist: importlib-metadata (<5.0.0,>=1.7) ; python_version < "3.9"