Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling backend] #3042

denik · 2025-06-12T11:22:36Z

Changes

Update default-python template to use pyproject.toml and uv (with hatchling).
Get rid of dynamic version in setup.py and using dynamic_version attribute on artifacts instead (only in dev target and only on non-serverless clusters since serverless has fixed the issue with versions not being updated in the platform). See Add libs/patchwheel: insert dynamic version in whl #2427 Implement dynamic_version attribute in artifacts section #2520

Why

Modern Python is based around pyproject.toml, not setup.py.

Using DAB-provided dynamic_version attribute allows keeping pyproject.toml simple and does not constraint what build backend users can choose.

Note, I've also considered using uv_build backend #3004 but it seems too early at the moment and hatchling is better tested with DABs.

Tests

Existing tests.

Copilot

Pull Request Overview

This PR modernizes the default‐python template by removing setup.py and in‐code versioning in favor of a pyproject.toml‐based hatchling build, switches the bundle to use uv build --wheel, and applies dynamic_version only on non‐serverless dev targets.

Drop __version__ and setup.py.tmpl, introduce pyproject.toml.tmpl with hatchling wheel config.
Update databricks.yml.tmpl to run uv build --wheel and conditionally enable dynamic_version for dev.
Refresh acceptance tests (out.*, check_output.py) and update NEXT_CHANGELOG.md.

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
libs/template/.../init.py	Remove hardcoded `__version__` assignment
libs/template/.../pyproject.toml.tmpl	Add `[project]` metadata and hatchling wheel build config
libs/template/.../databricks.yml.tmpl	Use `uv build --wheel` and conditional `dynamic_version` in dev
acceptance/.../check_output.py	Replace literal upload check with regex-based `UPLOADING_WHL`
NEXT_CHANGELOG.md	Document new default-python template changes

Comments suppressed due to low confidence (2)

NEXT_CHANGELOG.md:6

[nitpick] The backslash in dynamic\_version is only needed to escape Markdown italics. For clarity, wrap it in inline code: dynamic_version.

* Switch default-python template to use pyproject.toml + dynamic\_version in dev target. uv is now required. ([#3042](https://github.com/databricks/cli/pull/3042))

libs/template/templates/default-python/template/{{.project_name}}/databricks.yml.tmpl:8

The conditional artifacts: block under the dev target may be misaligned in indentation and could break YAML parsing. Align the artifacts: and python_artifact: keys with the workspace: block under dev.

{{ if $with_python }}

Copilot · 2025-06-13T08:07:14Z

acceptance/bundle/templates/default-python/combinations/check_output.py

 import subprocess

 CLOUD_ENV = os.environ.get("CLOUD_ENV")
 if CLOUD_ENV and os.environ["SERVERLESS"] == "yes" and not os.environ.get("TEST_METASTORE_ID"):
    sys.exit(f"SKIP_TEST SERVERLESS=yes but TEST_METASTORE_ID is empty in this env {CLOUD_ENV=}")

 BUILDING = "Building python_artifact"
-UPLOADING = "Uploading dist/"
+UPLOADING_WHL = re.compile("Uploading .*whl")


In is_printable_line, using UPLOADING_WHL.match(line) only matches at the start and will miss indented or prefixed upload lines. Consider using search() in both the function and the regex (e.g., re.compile("Uploading .*\.whl")) or anchoring appropriately.

Suggested change

UPLOADING_WHL = re.compile("Uploading .*whl")

UPLOADING_WHL = re.compile(r"Uploading .*\.whl")

acceptance/bundle/templates/default-python/serverless/output/my_default_python/pyproject.toml

pietern

Nice!

libs/template/templates/default-python/template/{{.project_name}}/databricks.yml.tmpl

libs/template/templates/default-python/template/{{.project_name}}/pyproject.toml.tmpl

kanterov · 2025-06-18T14:21:58Z

Do we need an additional level of indirection where we enable artifacts_dynamic_version in some templates but not the other? I think we should have it enabled across the board, or even consider making it a default in dev mode.

We should also enable in jobs-as-code-template because it currently uses a workaround with timestamps.

What about using the setuptools backend? I think it's more adopted because it's the default.

denik · 2025-06-20T05:39:42Z

Do we need an additional level of indirection where we enable artifacts_dynamic_version in some templates but not the other? I think we should have it enabled across the board, or even consider making it a default in dev mode.

It has downsides -with new files created for each deploy you either delete them and potentially break old runs or keep them and potentially accumulate too many. You also accumulate them on cluster. It is still worth it, but it needs to be targeted only to cases where it does help.

Serverless does not have stale version problem, so no need to enable it there.

We should also enable in jobs-as-code-template because it currently uses a workaround with timestamps.
Yes, once this landed we should align both templates. Maybe even make Python an option in default-python rather than separate template.

What about using the setuptools backend? I think it's more adopted because it's the default.

hatchling is default in uv: https://docs.astral.sh/uv/concepts/build-backend/

pietern

Approving to unblock merge.

Before we merge though, we need to ensure this is compatible with DABs in the workspace.

pietern · 2025-06-20T08:57:55Z

libs/template/templates/default-python/template/{{.project_name}}/databricks.yml.tmpl

@@ -18,7 +25,14 @@ targets:
    default: true
    workspace:
      host: {{workspace_host}}
-
+{{ if ($with_classic) }}


You comment:

Serverless does not have stale version problem, so no need to enable it there.

If some cache is enabled again it might break. We can opt to be defensive and include it anyway.

Native solution seem preferable, given the downsides (new wheel name may cause jobs not to find old wheel names OR all wheels are kept, polluting the workspace).

We have a test that monitors serverless behaviour without dynamic_version.

…ot of dependencies, slows down "uv sync" ├── databricks-dlt v0.3.0 (extra: dev) │ └── databricks-connect v15.4.9 │ ├── databricks-sdk v0.57.0 │ │ ├── google-auth v2.40.3 │ │ │ ├── cachetools v5.5.2 │ │ │ ├── pyasn1-modules v0.4.2 │ │ │ │ └── pyasn1 v0.6.1 │ │ │ └── rsa v4.9.1 │ │ │ └── pyasn1 v0.6.1 │ │ └── requests v2.32.4 │ │ ├── certifi v2025.6.15 │ │ ├── charset-normalizer v3.4.2 │ │ ├── idna v3.10 │ │ └── urllib3 v2.4.0 │ ├── googleapis-common-protos v1.70.0 │ │ └── protobuf v6.31.1 │ ├── grpcio v1.73.0 │ ├── grpcio-status v1.73.0 │ │ ├── googleapis-common-protos v1.70.0 (*) │ │ ├── grpcio v1.73.0 │ │ └── protobuf v6.31.1 │ ├── numpy v1.26.4 │ ├── packaging v25.0 │ ├── pandas v2.3.0 │ │ ├── numpy v1.26.4 │ │ ├── python-dateutil v2.9.0.post0 │ │ │ └── six v1.17.0 │ │ ├── pytz v2025.2 │ │ └── tzdata v2025.2 │ ├── py4j v0.10.9.7 │ ├── pyarrow v20.0.0 │ ├── setuptools v80.9.0 │ └── six v1.17.0

denik requested review from pietern, andrewnester, shreyas-goenka and anton-107 as code owners June 12, 2025 11:22

denik temporarily deployed to test-trigger-is June 12, 2025 11:22 — with GitHub Actions Inactive

denik mentioned this pull request Jun 12, 2025

Switch default-python to use pyproject.toml & uv & dynamic_version in dev [uv_build variant] #3004

Closed

denik changed the title ~~Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling]~~ Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling backend] Jun 12, 2025

denik force-pushed the denik/pyproject-hatchling branch from 84838b2 to 739f980 Compare June 12, 2025 11:26

denik temporarily deployed to test-trigger-is June 12, 2025 11:26 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 12, 2025 11:27 — with GitHub Actions Inactive

denik force-pushed the denik/pyproject-hatchling branch from a9fc9e8 to ecd0eae Compare June 12, 2025 11:29

denik temporarily deployed to test-trigger-is June 12, 2025 11:29 — with GitHub Actions Inactive

shreyas-goenka requested a review from Copilot June 13, 2025 08:05

Copilot AI reviewed Jun 13, 2025

View reviewed changes

denik commented Jun 13, 2025

View reviewed changes

acceptance/bundle/templates/default-python/serverless/output/my_default_python/pyproject.toml Show resolved Hide resolved

denik temporarily deployed to test-trigger-is June 13, 2025 09:16 — with GitHub Actions Inactive

denik force-pushed the denik/pyproject-hatchling branch from 63a504d to 039c825 Compare June 13, 2025 09:16

denik temporarily deployed to test-trigger-is June 13, 2025 09:16 — with GitHub Actions Inactive

denik force-pushed the denik/pyproject-hatchling branch from 039c825 to 206342e Compare June 13, 2025 09:16

denik temporarily deployed to test-trigger-is June 13, 2025 09:16 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 13, 2025 09:20 — with GitHub Actions Inactive

denik force-pushed the denik/pyproject-hatchling branch from 77d07fa to 76ac9fb Compare June 16, 2025 07:24

denik temporarily deployed to test-trigger-is June 16, 2025 07:24 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 16, 2025 07:37 — with GitHub Actions Inactive

pietern reviewed Jun 16, 2025

View reviewed changes

denik temporarily deployed to test-trigger-is June 16, 2025 10:00 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 16, 2025 10:07 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 16, 2025 10:49 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 16, 2025 10:51 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 16, 2025 10:52 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is June 18, 2025 12:15 — with GitHub Actions Inactive

denik requested review from pietern, lennartkats-db and kanterov June 18, 2025 12:16

denik temporarily deployed to test-trigger-is June 18, 2025 12:33 — with GitHub Actions Inactive

pietern approved these changes Jun 20, 2025

View reviewed changes

andrewnester approved these changes Jun 20, 2025

View reviewed changes

denik temporarily deployed to test-trigger-is June 23, 2025 07:39 — with GitHub Actions Inactive

denik force-pushed the denik/pyproject-hatchling branch from 27a36eb to 9e8ba1f Compare June 23, 2025 13:49

denik temporarily deployed to test-trigger-is June 23, 2025 13:49 — with GitHub Actions Inactive

denik added 16 commits June 23, 2025 16:27

Switch to pyproject.toml + uv

6e322c5

update NEXT_CHANGELOG.md

1dd427f

set require-python to 3.11+; explicitly specify hatchling

1e8596d

move requirements-dev.txt into pyproject.toml

f7ea370

Add pre-rendered uv.lock

ae5dfe5

restore databricks-dlt install

acca7ba

restore databricks-dlt and add it to lock

8bab87e

update lock with better timestamp regex

1f5f800

rename gentemplates to genuvlock

49ae802

clean up uv.lock

f3417ee

Makefile resolve

4c889d7

add artifacts_dynamic_version

5d0e5ad

update indentation

56082b9

add missing output files

b18f972

update test output

c982940

denik force-pushed the denik/pyproject-hatchling branch from 9e8ba1f to c982940 Compare June 23, 2025 14:28

denik temporarily deployed to test-trigger-is June 23, 2025 14:28 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling backend] #3042

Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling backend] #3042

Uh oh!

denik commented Jun 12, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jun 13, 2025

Uh oh!

Uh oh!

pietern left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kanterov commented Jun 18, 2025

Uh oh!

denik commented Jun 20, 2025

Uh oh!

pietern left a comment

Uh oh!

pietern Jun 20, 2025

Uh oh!

denik Jun 23, 2025

Uh oh!

Uh oh!

	UPLOADING_WHL = re.compile("Uploading .*whl")
	UPLOADING_WHL = re.compile(r"Uploading .*\.whl")

Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling backend] #3042

Are you sure you want to change the base?

Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling backend] #3042

Uh oh!

Conversation

denik commented Jun 12, 2025

Changes

Why

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kanterov commented Jun 18, 2025

Uh oh!

denik commented Jun 20, 2025

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

pietern Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

denik Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!