Skip to content

Commit b60579b

Browse files
andrewnesterandrewnester
authored
Added experimental.skip_artifact_cleanup flag (#2980)
## Changes Added `experimental.skip_artifact_cleanup` flag. When set to `true` .internal folder in Databricks workspace which contains built artifacts such as wheels won't be cleaned up on deploy Fixes #2969 #2958 ## Why It can be useful to keep old built wheels around for cases when they are running jobs using the older version of wheels to prevent new deployments from breaking these jobs. ## Tests Added acceptance tests <!-- If your PR needs to be included in the release notes for next release, add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
1 parent ecb6307 commit b60579b

File tree

25 files changed

+208
-7
lines changed

25 files changed

+208
-7
lines changed

‎NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
### CLI
1010

1111
### Bundles
12+
* Added `experimental.skip_artifact_cleanup` flag ([#2980](https://github.com/databricks/cli/pull/2980))
1213
* Add an experimental project template for Lakeflow Declarative Pipelines ([#2959](https://github.com/databricks/cli/pull/2959))
1314

1415
### API Changes

‎acceptance/bundle/artifacts/whl_no_cleanup/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
build/
2+
*.egg-info
3+
.databricks

‎acceptance/bundle/artifacts/whl_no_cleanup/databricks.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
experimental:
2+
skip_artifact_cleanup: true
3+
4+
resources:
5+
jobs:
6+
test_job:
7+
name: "[${bundle.target}] My Wheel Job"
8+
tasks:
9+
- task_key: TestTask
10+
existing_cluster_id: "0717-aaaaa-bbbbbb"
11+
python_wheel_task:
12+
package_name: "my_test_code"
13+
entry_point: "run"
14+
libraries:
15+
- whl: ./dist/*.whl

‎acceptance/bundle/artifacts/whl_no_cleanup/my_test_code/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
__version__ = "0.0.1"
2+
__author__ = "Databricks"

‎acceptance/bundle/artifacts/whl_no_cleanup/my_test_code/__main__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
"""
2+
The entry point of the Python Wheel
3+
"""
4+
5+
import sys
6+
7+
8+
def main():
9+
# This method will print the provided arguments
10+
print("Hello from my func")
11+
print("Got arguments:")
12+
print(sys.argv)
13+
14+
15+
if __name__ == "__main__":
16+
main()

‎acceptance/bundle/artifacts/whl_no_cleanup/output.txt‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
2+
>>> [CLI] bundle deploy
3+
Building python_artifact...
4+
Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
5+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
6+
Deploying resources...
7+
Updating deployment state...
8+
Deployment complete!
9+
10+
>>> find.py --expect 1 whl
11+
dist/my_test_code-0.0.1-py3-none-any.whl
12+
13+
=== Expecting 1 wheels to be uploaded
14+
>>> jq .path
15+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1-py3-none-any.whl"
16+
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/dist/my_test_code-0.0.1-py3-none-any.whl"
17+
18+
>>> [CLI] bundle deploy
19+
Building python_artifact...
20+
Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
21+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
22+
Deploying resources...
23+
Updating deployment state...
24+
Deployment complete!
25+
26+
=== No calls to delete internal folder expected
27+
>>> jq -s .[] | select(.path=="/api/2.0/workspace/delete") | select(.body.path | test(".*/artifacts/.internal")) out.requests.txt
28+
29+
=== Expected 2 calls to create artifacts folder (because 2 deploys were done)
30+
>>> jq -s .[] | select(.path=="/api/2.0/workspace/mkdirs") | select(.body.path | test(".*/artifacts/.internal")) out.requests.txt
31+
{
32+
"method": "POST",
33+
"path": "/api/2.0/workspace/mkdirs",
34+
"body": {
35+
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal"
36+
}
37+
}
38+
{
39+
"method": "POST",
40+
"path": "/api/2.0/workspace/mkdirs",
41+
"body": {
42+
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal"
43+
}
44+
}

‎acceptance/bundle/artifacts/whl_no_cleanup/script‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
trace $CLI bundle deploy
2+
3+
trace find.py --expect 1 whl
4+
5+
title "Expecting 1 wheels to be uploaded"
6+
trace jq .path < out.requests.txt | grep import | grep whl | sort
7+
8+
trace $CLI bundle deploy
9+
title "No calls to delete internal folder expected"
10+
trace jq -s '.[] | select(.path=="/api/2.0/workspace/delete") | select(.body.path | test(".*/artifacts/.internal"))' out.requests.txt
11+
12+
title "Expected 2 calls to create artifacts folder (because 2 deploys were done)"
13+
trace jq -s '.[] | select(.path=="/api/2.0/workspace/mkdirs") | select(.body.path | test(".*/artifacts/.internal"))' out.requests.txt
14+
15+
rm out.requests.txt

‎acceptance/bundle/artifacts/whl_no_cleanup/setup.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
from setuptools import setup, find_packages
2+
3+
import my_test_code
4+
5+
setup(
6+
name="my_test_code",
7+
version=my_test_code.__version__,
8+
author=my_test_code.__author__,
9+
url="https://databricks.com",
10+
author_email="john.doe@databricks.com",
11+
description="my test wheel",
12+
packages=find_packages(include=["my_test_code"]),
13+
entry_points={"group_1": "run=my_test_code.__main__:main"},
14+
install_requires=["setuptools"],
15+
)

‎acceptance/bundle/paths/fallback_metric/output.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ Deployment complete!
1818
"key": "presets_name_prefix_is_set",
1919
"value": false
2020
},
21+
{
22+
"key": "skip_artifact_cleanup",
23+
"value": false
24+
},
2125
{
2226
"key": "python_wheel_wrapper_is_set",
2327
"value": false
@@ -44,6 +48,10 @@ Deployment complete!
4448
"key": "is_job_path_fallback",
4549
"value": true
4650
},
51+
{
52+
"key": "skip_artifact_cleanup",
53+
"value": false
54+
},
4755
{
4856
"key": "python_wheel_wrapper_is_set",
4957
"value": false

‎acceptance/bundle/telemetry/deploy-artifact-path-type/output.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,10 @@ Deployment complete!
3333
"lookup_variable_count": 0,
3434
"target_count": 2,
3535
"bool_values": [
36+
{
37+
"key": "skip_artifact_cleanup",
38+
"value": false
39+
},
3640
{
3741
"key": "python_wheel_wrapper_is_set",
3842
"value": false
@@ -77,6 +81,10 @@ Deployment complete!
7781
"lookup_variable_count": 0,
7882
"target_count": 2,
7983
"bool_values": [
84+
{
85+
"key": "skip_artifact_cleanup",
86+
"value": false
87+
},
8088
{
8189
"key": "python_wheel_wrapper_is_set",
8290
"value": false

0 commit comments

Comments
 (0)