Skip to content

Commit 8ffff79

Browse files
denikdenik
authored
When globbing for wheels, only consider most recent version; do not clean up dist/build/__pycache__ (#2982)
## Changes - When glob like *.whl is expanded, if there are multiple versions of the same package, we will only consider the most recent version. - Do not forcibly delete dist, build and `__pycache__` folders on every deploy. ## Why Deleting dist folder is surprising and invasive (e.g. see #2969). In order to avoid doing that but keep old "dist/*.whl" globs working, we're modifying glob behaviour to do additional filtering so that if there are more than 1 wheel from the same package, we'll only upload the one with most recent version. ## Tests New acceptance test simulates situation in #2969 Existing acceptance test show that wheels that were uploaded twice (from .internal and dist folders) are not uploaded once, which is more inline with the expectations.
1 parent b5c872e commit 8ffff79

File tree

19 files changed

+458
-121
lines changed

19 files changed

+458
-121
lines changed

‎NEXT_CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,7 @@
1010

1111
### Bundles
1212
* Fix reading dashboard contents when the sync root is different than the bundle root ([#3006](https://github.com/databricks/cli/pull/3006))
13+
* When glob for wheels is used, like "\*.whl", it will filter out different version of the same package and will only take the most recent version. ([#2982](https://github.com/databricks/cli/pull/2982))
14+
* When building Python artifacts as part of "bundle deploy" we no longer delete `dist`, `build`, `*egg-info` and `__pycache__` directories. ([#2982](https://github.com/databricks/cli/pull/2982))
1315

1416
### API Changes

‎acceptance/bundle/artifacts/whl_change_version/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
build/
2+
*.egg-info
3+
.databricks

‎acceptance/bundle/artifacts/whl_change_version/databricks.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
resources:
2+
jobs:
3+
test_job:
4+
name: "[${bundle.target}] My Wheel Job"
5+
tasks:
6+
- task_key: TestTask
7+
existing_cluster_id: "0717-aaaaa-bbbbbb"
8+
python_wheel_task:
9+
package_name: "my_test_code"
10+
entry_point: "run"
11+
libraries:
12+
- whl: ./dist/*.whl

‎acceptance/bundle/artifacts/whl_change_version/my_test_code/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
__version__ = "0.1.0"
2+
__author__ = "Databricks"

‎acceptance/bundle/artifacts/whl_change_version/my_test_code/__main__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
"""
2+
The entry point of the Python Wheel
3+
"""
4+
5+
import sys
6+
7+
8+
def main():
9+
# This method will print the provided arguments
10+
print("Hello from my func")
11+
print("Got arguments:")
12+
print(sys.argv)
13+
14+
15+
if __name__ == "__main__":
16+
main()

‎acceptance/bundle/artifacts/whl_change_version/output.txt‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
2+
>>> [CLI] bundle deploy
3+
Building python_artifact...
4+
Uploading dist/my_test_code-0.1.0-py3-none-any.whl...
5+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
6+
Deploying resources...
7+
Updating deployment state...
8+
Deployment complete!
9+
10+
>>> find.py --expect 1 whl
11+
dist/my_test_code-0.1.0-py3-none-any.whl
12+
13+
=== Expecting 1 wheel in libraries section in /jobs/create
14+
>>> jq -s .[] | select(.path=="/api/2.2/jobs/create") | .body.tasks out.requests.txt
15+
[
16+
{
17+
"existing_cluster_id": "0717-aaaaa-bbbbbb",
18+
"libraries": [
19+
{
20+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.1.0-py3-none-any.whl"
21+
}
22+
],
23+
"python_wheel_task": {
24+
"entry_point": "run",
25+
"package_name": "my_test_code"
26+
},
27+
"task_key": "TestTask"
28+
}
29+
]
30+
31+
=== Expecting 1 wheel to be uploaded
32+
>>> jq .path out.requests.txt
33+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.1.0-py3-none-any.whl"
34+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/.gitignore"
35+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/databricks.yml"
36+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/dist/my_test_code-0.1.0-py3-none-any.whl"
37+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/my_test_code/__init__.py"
38+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/my_test_code/__main__.py"
39+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/out.requests.txt"
40+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/output.txt"
41+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/repls.json"
42+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/script"
43+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/setup.py"
44+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deploy.lock"
45+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json"
46+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
47+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
48+
49+
>>> update_file.py my_test_code/__init__.py 0.1.0 0.2.0
50+
51+
>>> [CLI] bundle deploy
52+
Building python_artifact...
53+
Uploading dist/my_test_code-0.2.0-py3-none-any.whl...
54+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
55+
Deploying resources...
56+
Updating deployment state...
57+
Deployment complete!
58+
59+
>>> find.py --expect 2 whl
60+
dist/my_test_code-0.1.0-py3-none-any.whl
61+
dist/my_test_code-0.2.0-py3-none-any.whl
62+
63+
=== Expecting 1 wheel in libraries section in /jobs/reset
64+
>>> jq -s .[] | select(.path=="/api/2.2/jobs/reset") | .body.new_settings.tasks out.requests.txt
65+
[
66+
{
67+
"existing_cluster_id": "0717-aaaaa-bbbbbb",
68+
"libraries": [
69+
{
70+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.2.0-py3-none-any.whl"
71+
}
72+
],
73+
"python_wheel_task": {
74+
"entry_point": "run",
75+
"package_name": "my_test_code"
76+
},
77+
"task_key": "TestTask"
78+
}
79+
]
80+
81+
=== Expecting 1 wheel to be uploaded
82+
>>> jq .path out.requests.txt
83+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.2.0-py3-none-any.whl"
84+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/dist/my_test_code-0.2.0-py3-none-any.whl"
85+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/my_test_code/__init__.py"
86+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/out.requests.txt"
87+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/output.txt"
88+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deploy.lock"
89+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json"
90+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
91+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
92+
93+
=== Restore config to target old wheel
94+
>>> update_file.py databricks.yml ./dist/*.whl ./dist/my*0.1.0*.whl
95+
96+
>>> [CLI] bundle deploy
97+
Building python_artifact...
98+
Uploading dist/my_test_code-0.1.0-py3-none-any.whl...
99+
Uploading dist/my_test_code-0.2.0-py3-none-any.whl...
100+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
101+
Deploying resources...
102+
Updating deployment state...
103+
Deployment complete!
104+
105+
>>> find.py --expect 2 whl
106+
dist/my_test_code-0.1.0-py3-none-any.whl
107+
dist/my_test_code-0.2.0-py3-none-any.whl
108+
109+
=== Expecting 1 wheel in libraries section in /jobs/reset
110+
>>> jq -s .[] | select(.path=="/api/2.2/jobs/reset") | .body.new_settings.tasks out.requests.txt
111+
[
112+
{
113+
"existing_cluster_id": "0717-aaaaa-bbbbbb",
114+
"libraries": [
115+
{
116+
"whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.1.0-py3-none-any.whl"
117+
}
118+
],
119+
"python_wheel_task": {
120+
"entry_point": "run",
121+
"package_name": "my_test_code"
122+
},
123+
"task_key": "TestTask"
124+
}
125+
]
126+
127+
=== Expecting 1 wheel to be uploaded
128+
>>> jq .path out.requests.txt
129+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.1.0-py3-none-any.whl"
130+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.2.0-py3-none-any.whl"
131+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/databricks.yml"
132+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/dist/my_test_code-0.2.0-py3-none-any.whl"
133+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/out.requests.txt"
134+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/output.txt"
135+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deploy.lock"
136+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json"
137+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
138+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"

‎acceptance/bundle/artifacts/whl_change_version/script‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
trace $CLI bundle deploy
2+
3+
trace find.py --expect 1 whl
4+
5+
title "Expecting 1 wheel in libraries section in /jobs/create"
6+
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body.tasks' out.requests.txt
7+
8+
title "Expecting 1 wheel to be uploaded"
9+
trace jq .path out.requests.txt | grep import | sort
10+
11+
rm out.requests.txt
12+
13+
14+
trace update_file.py my_test_code/__init__.py 0.1.0 0.2.0
15+
trace $CLI bundle deploy
16+
17+
trace find.py --expect 2 whl # there are now 2 wheels on disk
18+
19+
title "Expecting 1 wheel in libraries section in /jobs/reset"
20+
trace jq -s '.[] | select(.path=="/api/2.2/jobs/reset") | .body.new_settings.tasks' out.requests.txt
21+
22+
title "Expecting 1 wheel to be uploaded"
23+
trace jq .path out.requests.txt | grep import | sort
24+
25+
rm out.requests.txt
26+
27+
28+
title 'Restore config to target old wheel'
29+
trace update_file.py databricks.yml './dist/*.whl' './dist/my*0.1.0*.whl'
30+
trace $CLI bundle deploy
31+
trace find.py --expect 2 whl
32+
33+
title "Expecting 1 wheel in libraries section in /jobs/reset"
34+
trace jq -s '.[] | select(.path=="/api/2.2/jobs/reset") | .body.new_settings.tasks' out.requests.txt
35+
36+
title "Expecting 1 wheel to be uploaded"
37+
trace jq .path out.requests.txt | grep import | sort
38+
39+
rm out.requests.txt

‎acceptance/bundle/artifacts/whl_change_version/setup.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
from setuptools import setup, find_packages
2+
3+
import my_test_code
4+
5+
setup(
6+
name="my_test_code",
7+
version=my_test_code.__version__,
8+
author=my_test_code.__author__,
9+
url="https://databricks.com",
10+
author_email="john.doe@databricks.com",
11+
description="my test wheel",
12+
packages=find_packages(include=["my_test_code"]),
13+
entry_points={"group_1": "run=my_test_code.__main__:main"},
14+
install_requires=["setuptools"],
15+
)

‎bundle/artifacts/build.go‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,6 @@ func (m *build) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
3838
})
3939
}
4040

41-
cleanupPythonDistBuild(ctx, b)
42-
4341
for _, artifactName := range utils.SortedKeys(b.Config.Artifacts) {
4442
a := b.Config.Artifacts[artifactName]
4543

‎bundle/artifacts/cleanup.go‎

Lines changed: 0 additions & 38 deletions
This file was deleted.

0 commit comments

Comments
 (0)