[Python] Move 'python' outside of experimental #3540

kanterov · 2025-09-02T14:01:06Z

Changes

Move python outside of the experimental section.

The change is backward-compatible and forward-compatible. If CLI is using a newer or older version of databricks-bundles, it will continue to work. It's implemented by copying the content of python into experimental.python. If we don't offer compatibility, there is a risk that previously deployed resources will be erased because we wouldn't be able to load resources.

Why

Python support for Databricks Asset Bundles is graduating from the experimental status.

Tests

Acceptance tests check that previous databricks-bundles versions continue to work.

eng-dev-ecosystem-bot · 2025-09-02T14:29:22Z

Run: 17406397515

	Env	✅pass	🔄flaky	🙈skip
✅	aws linux	308		511
🔄	aws windows	306	3	510

Test Name	aws windows
TestAccept	🔄flaky
TestAccept/bundle/templates/default-python/integration_classic	🔄flaky
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_CLI_DEPLOYMENT=terraform/UV_PYTHON=3.13	🔄flaky

eng-dev-ecosystem-bot · 2025-09-02T14:29:22Z

Run: 18834140817

	Env	❌FAIL	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip
💚	aws linux				1	1	319	583
💚	aws windows				1	1	320	582
💚	aws-ucws linux				1	1	440	478
💚	aws-ucws windows				1	1	441	477
💚	azure linux				1	1	319	582
💚	azure windows				1	1	320	581
❌	azure-ucws linux	1	1			1	437	477
❌	azure-ucws windows	1	1			1	438	476
🔄	gcp linux			11	1	1	307	584
🔄	gcp windows			4	1	1	315	583

18 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚R	💚R	💚R	💚R	💚R	💚R	🟨K	🟨K	💚R	💚R
TestAccept/bundle/deploy/files/no-snapshot-sync	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/destroy/jobs-and-pipeline	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/local_state_staleness	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/local_state_staleness/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/experiments/basic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/synced_database_tables/basic	🙈s	🙈s	✅p	✅p	🙈s	🙈s	❌F	❌F	🙈s	🙈s
TestAccept/bundle/run/app-with-job	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=no/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=no/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct-exp/DLT=no/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f
TestAccept/bundle/templates/default-python/combinations/serverless	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/integration_classic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.10	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p

denik · 2025-09-08T08:12:20Z

acceptance/bundle/python/mutator-ordering/out.test.toml

 [EnvMatrix]
  DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"]
-  UV_ARGS = ["--with databricks-bundles==0.7.3", "--with-requirements requirements-latest.txt --no-cache"]
+  UV_ARGS = ["--with databricks-bundles==0.266.0", "--with-requirements requirements-latest.txt --no-cache"]


Is it intentional to test only 0.266.0? What about the most recent release or main?

We need some version from the past, and it was selected to be 0.266.0. It doesn't have to be the latest release. We just need to select a version that users might pin.

The second case for UV_ARGS is testing against the latest version from main

Makes sense to test past version -- please add a comment that this is intentional.

Does UV_ARGS="--with-requirements requirements-latest.txt --no-cache" results in testing the python package from the current branch or latest release from PyPI. Please add a comment as well, it's not clear.

👍 , I've clarified. It's using the same commit as acceptance tests

denik · 2025-09-08T08:13:05Z

bundle/config/mutator/python/python_mutator.go

-		// don't execute for phases for 'pydabs' section
-		if phase == PythonMutatorPhaseLoadResources || phase == PythonMutatorPhaseApplyMutators {
+	if experimentalPythonEnabled && pythonEnabled {
+		if !reflect.DeepEqual(experimental.Python, b.Config.Python) {


why reflect.Equal? Why not require users specify this once?

I see below that this is to make python package work but IMO we should separate what we require from users and what we send to Python. E.g. we can have validator that requires that only one is present but then duplicate and send both to Python.

The first time we call "getOpts", only 1 property can be specified. The second time, we call "getOpts" on the mutated bundle that contains both options specified after we have called applyBackwardsCompatibilityFixes, so we need to allow this case as well.

Here, we call applyBackwardsCompatibilityFixes before getOpts, so even the first time, we expect both options to be set.

why do we call it twice? cannot we remember the output from the first call?

Our mutators are intended to be stateless, so we can't remember. And we call Python mutator twice in the mutator pipeline

Our mutators are intended to be stateless, so we can't remember. And we call Python mutator twice in the mutator pipeline

You can certainly have a field on *Bundle to track state relevant for you. Especially if it saves work and simplifies code.

Right. I just don't see a problem. If we want the bundle spec to be stable, we should allow specifying both properties during the transition period.

denik · 2025-09-08T08:14:54Z

acceptance/bundle/python/mutator-ordering/databricks.yml

-    mutators:
-      - "mutators:add_task_1"
-      - "mutators:add_task_2"
+python:


Would be good to have acceptance tests for new functionality:

only experimental.python is provided

both are provided and equal

both are provided and not equal

👍 , added in the latest commit

## Changes Support reading both `python` and `experimental.python` configs. If both are set, their value should be equivalent. After we introduce the `python` section into the CLI, it has to keep setting both `python` and `experimental.python` into the payload for backward compatibility. After that, Python code can start erroring out if `experimental/python` is set but not `python` because it means that users have an outdated CLI version. See also #3540 ## Why It's a step toward graduating `python` outside the `experimental` section. ## Tests Unit tests and existing acceptance tests

## Release v0.275.0 ### Notable Changes * Python support for Databricks Asset Bundles is now generally available. ### CLI * Remove `inplace` mode for the `--progress-format` flag ([#3811](#3811)) * Remove `json` mode for the `--progress-format` flag ([#3812](#3812)) * Deprecate the `--progress-format` flag ([#3819](#3819)) ### Bundles * Add support for `--bind` flag in `bundle generate` ([#3782](#3782)) * Add `pydabs` template replacing `experimental-jobs-as-code` template ([#3806](#3806)) * You can now use the top-level `python` section instead of `experimental/python` ([#3540](#3540))

kanterov temporarily deployed to test-trigger-is September 2, 2025 14:01 — with GitHub Actions Inactive

kanterov force-pushed the python-non-experimental branch from fe939e5 to 3c1043c Compare September 2, 2025 14:15

kanterov temporarily deployed to test-trigger-is September 2, 2025 14:15 — with GitHub Actions Inactive

kanterov marked this pull request as ready for review September 2, 2025 14:52

kanterov requested review from andrewnester, anton-107, denik, pietern and shreyas-goenka as code owners September 2, 2025 14:52

kanterov mentioned this pull request Sep 2, 2025

[Python] Support non-experimental conf in Python code #3539

Merged

kanterov added the do-not-merge label Sep 2, 2025

denik reviewed Sep 8, 2025

View reviewed changes

kanterov temporarily deployed to test-trigger-is September 8, 2025 08:43 — with GitHub Actions Inactive

kanterov temporarily deployed to test-trigger-is September 8, 2025 08:48 — with GitHub Actions Inactive

kanterov temporarily deployed to test-trigger-is September 8, 2025 11:23 — with GitHub Actions Inactive

kanterov force-pushed the python-non-experimental branch from d77a490 to f751173 Compare September 8, 2025 13:49

kanterov temporarily deployed to test-trigger-is September 8, 2025 13:49 — with GitHub Actions Inactive

denik approved these changes Sep 15, 2025

View reviewed changes

kanterov force-pushed the python-non-experimental branch from f751173 to 8fe901c Compare October 24, 2025 12:54

kanterov requested a review from lennartkats-db as a code owner October 24, 2025 12:54

kanterov temporarily deployed to test-trigger-is October 24, 2025 12:54 — with GitHub Actions Inactive

kanterov added 6 commits October 24, 2025 15:01

[Python] Move 'python' outside of experimental

7612d1b

Add more acceptenace tests

cf22af4

Clarify UV_ARGS comments

e881b5f

Fix lint

15a20d6

Few more fixes

68e1b7f

Update template

a0b0f66

kanterov added 4 commits October 24, 2025 15:01

Update tests

251596d

Add CHANGELOG

c0db934

Update acc

97cb2e7

Fix tests

32adbe7

kanterov force-pushed the python-non-experimental branch from 8fe901c to 32adbe7 Compare October 24, 2025 13:01

kanterov temporarily deployed to test-trigger-is October 24, 2025 13:01 — with GitHub Actions Inactive

Fix changelog

5848f32

kanterov temporarily deployed to test-trigger-is October 24, 2025 13:02 — with GitHub Actions Inactive

Fix tests

ba1d8c2

kanterov temporarily deployed to test-trigger-is October 24, 2025 13:09 — with GitHub Actions Inactive

Fix minor diffs

084dabf

kanterov temporarily deployed to test-trigger-is October 24, 2025 13:18 — with GitHub Actions Inactive

kanterov removed the do-not-merge label Oct 24, 2025

Merge branch 'main' into python-non-experimental

bec18b7

kanterov temporarily deployed to test-trigger-is October 27, 2025 08:10 — with GitHub Actions Inactive

kanterov enabled auto-merge October 27, 2025 10:06

kanterov disabled auto-merge October 27, 2025 10:29

pietern merged commit 6f8e2b0 into main Oct 27, 2025
18 of 19 checks passed

pietern deleted the python-non-experimental branch October 27, 2025 13:58

[Python] Move 'python' outside of experimental #3540

[Python] Move 'python' outside of experimental #3540

Uh oh!

Conversation

kanterov commented Sep 2, 2025

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Sep 2, 2025

Uh oh!

eng-dev-ecosystem-bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kanterov Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

eng-dev-ecosystem-bot commented Sep 2, 2025 •

edited

Loading

kanterov Sep 8, 2025 •

edited

Loading