-
Notifications
You must be signed in to change notification settings - Fork 121
Added support for lifecycle prevent_destroy option #3448
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
2ddb691
Added support for lifecycle prevent_deploy option
andrewnester 637a057
added support for prevent_destroy in direct
andrewnester 251284c
Merge branch 'main' into feature/prevent-destroy
andrewnester fd67d51
fixes
andrewnester d8c5ecb
Merge branch 'main' into feature/prevent-destroy
andrewnester cc79894
fixes
andrewnester 9dc81ce
removed unnecessary test
andrewnester 877deb3
Merge branch 'main' into feature/prevent-destroy
andrewnester 04b059d
update schema
andrewnester 7e5b1fb
omitempty
andrewnester db9bafa
fixed schema
andrewnester 71d8981
make python
andrewnester 404192e
fixes
andrewnester 1f67f4f
fixed schema
andrewnester 3181033
whitespaces
andrewnester c066038
Merge branch 'main' into feature/prevent-destroy
andrewnester a32e69a
codegen python
andrewnester 109d453
fixes
andrewnester 1165def
Merge branch 'main' into feature/prevent-destroy
andrewnester db34f10
fixes
andrewnester 9118ceb
regenerate python
andrewnester fe565f5
do not use isDestroy flag and read prevent_destroy directly
andrewnester 65bc5dd
add schemas to test
andrewnester 42faa7b
show all failed resources
andrewnester File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| bundle: | ||
| name: prevent-destroy | ||
|
|
||
| include: | ||
| - resources/*.yml |
67 changes: 67 additions & 0 deletions
67
acceptance/bundle/lifecycle/prevent-destroy/out.direct-exp.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
|
|
||
| >>> musterr [CLI] bundle destroy --auto-approve | ||
| Error: resource my_pipelines has lifecycle.prevent_destroy set, but the plan calls for this resource to be recreated or destroyed. To avoid this error, disable lifecycle.prevent_destroy for pipelines.my_pipelines | ||
| resource my_schema has lifecycle.prevent_destroy set, but the plan calls for this resource to be recreated or destroyed. To avoid this error, disable lifecycle.prevent_destroy for schemas.my_schema | ||
|
|
||
|
|
||
| Exit code (musterr): 1 | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| recreate pipelines.my_pipelines | ||
|
|
||
| >>> musterr [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
| Error: resource my_pipelines has lifecycle.prevent_destroy set, but the plan calls for this resource to be recreated or destroyed. To avoid this error, disable lifecycle.prevent_destroy for pipelines.my_pipelines | ||
|
|
||
|
|
||
| Exit code (musterr): 1 | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| recreate pipelines.my_pipelines | ||
| recreate schemas.my_schema | ||
|
|
||
| >>> musterr [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
| Error: resource my_pipelines has lifecycle.prevent_destroy set, but the plan calls for this resource to be recreated or destroyed. To avoid this error, disable lifecycle.prevent_destroy for pipelines.my_pipelines | ||
| resource my_schema has lifecycle.prevent_destroy set, but the plan calls for this resource to be recreated or destroyed. To avoid this error, disable lifecycle.prevent_destroy for schemas.my_schema | ||
|
|
||
|
|
||
| Exit code (musterr): 1 | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| recreate pipelines.my_pipelines | ||
| recreate schemas.my_schema | ||
|
|
||
| >>> [CLI] bundle deploy --auto-approve | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
|
|
||
| This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost: | ||
| recreate schema my_schema | ||
|
|
||
| This action will result in the deletion or recreation of the following Lakeflow Declarative Pipelines along with the | ||
| Streaming Tables (STs) and Materialized Views (MVs) managed by them. Recreating the pipelines will | ||
| restore the defined STs and MVs through full refresh. Note that recreation is necessary when pipeline | ||
| properties such as the 'catalog' or 'storage' are changed: | ||
| recreate pipeline my_pipelines | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| delete pipelines.my_pipelines | ||
| delete schemas.my_schema | ||
|
|
||
| >>> [CLI] bundle deploy --auto-approve | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
|
|
||
| This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost: | ||
| delete schema my_schema | ||
|
|
||
| This action will result in the deletion or recreation of the following Lakeflow Declarative Pipelines along with the | ||
| Streaming Tables (STs) and Materialized Views (MVs) managed by them. Recreating the pipelines will | ||
| restore the defined STs and MVs through full refresh. Note that recreation is necessary when pipeline | ||
| properties such as the 'catalog' or 'storage' are changed: | ||
| delete pipeline my_pipelines | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
155 changes: 155 additions & 0 deletions
155
acceptance/bundle/lifecycle/prevent-destroy/out.terraform.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
|
|
||
| >>> musterr [CLI] bundle destroy --auto-approve | ||
| Error: exit status 1 | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 15, in resource.databricks_pipeline: | ||
| 15: "my_pipelines": { | ||
|
|
||
| Resource databricks_pipeline.my_pipelines has lifecycle.prevent_destroy set, | ||
| but the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 38, in resource.databricks_schema: | ||
| 38: "my_schema": { | ||
|
|
||
| Resource databricks_schema.my_schema has lifecycle.prevent_destroy set, but | ||
| the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
|
|
||
|
|
||
| Exit code (musterr): 1 | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
andrewnester marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Error: exit status 1 | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 15, in resource.databricks_pipeline: | ||
| 15: "my_pipelines": { | ||
|
|
||
| Resource databricks_pipeline.my_pipelines has lifecycle.prevent_destroy set, | ||
| but the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
|
|
||
|
|
||
| Exit code: 1 | ||
|
|
||
| >>> musterr [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
| Error: exit status 1 | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 15, in resource.databricks_pipeline: | ||
| 15: "my_pipelines": { | ||
|
|
||
| Resource databricks_pipeline.my_pipelines has lifecycle.prevent_destroy set, | ||
andrewnester marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| but the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
|
|
||
|
|
||
| Exit code (musterr): 1 | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| Error: exit status 1 | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 15, in resource.databricks_pipeline: | ||
| 15: "my_pipelines": { | ||
|
|
||
| Resource databricks_pipeline.my_pipelines has lifecycle.prevent_destroy set, | ||
| but the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 38, in resource.databricks_schema: | ||
| 38: "my_schema": { | ||
|
|
||
| Resource databricks_schema.my_schema has lifecycle.prevent_destroy set, but | ||
| the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
|
|
||
|
|
||
| Exit code: 1 | ||
|
|
||
| >>> musterr [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
| Error: exit status 1 | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 15, in resource.databricks_pipeline: | ||
| 15: "my_pipelines": { | ||
|
|
||
| Resource databricks_pipeline.my_pipelines has lifecycle.prevent_destroy set, | ||
| but the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
| Error: Instance cannot be destroyed | ||
|
|
||
| on bundle.tf.json line 38, in resource.databricks_schema: | ||
| 38: "my_schema": { | ||
|
|
||
| Resource databricks_schema.my_schema has lifecycle.prevent_destroy set, but | ||
| the plan calls for this resource to be destroyed. To avoid this error and | ||
| continue with the plan, either disable lifecycle.prevent_destroy or reduce | ||
| the scope of the plan using the -target flag. | ||
|
|
||
|
|
||
|
|
||
| Exit code (musterr): 1 | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| recreate pipelines.my_pipelines | ||
| recreate schemas.my_schema | ||
|
|
||
| >>> [CLI] bundle deploy --auto-approve | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
|
|
||
| This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost: | ||
| recreate schema my_schema | ||
|
|
||
| This action will result in the deletion or recreation of the following Lakeflow Declarative Pipelines along with the | ||
| Streaming Tables (STs) and Materialized Views (MVs) managed by them. Recreating the pipelines will | ||
| restore the defined STs and MVs through full refresh. Note that recreation is necessary when pipeline | ||
| properties such as the 'catalog' or 'storage' are changed: | ||
| recreate pipeline my_pipelines | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| delete pipelines.my_pipelines | ||
| delete schemas.my_schema | ||
|
|
||
| >>> [CLI] bundle deploy --auto-approve | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
|
|
||
| This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost: | ||
| delete schema my_schema | ||
|
|
||
| This action will result in the deletion or recreation of the following Lakeflow Declarative Pipelines along with the | ||
| Streaming Tables (STs) and Materialized Views (MVs) managed by them. Recreating the pipelines will | ||
| restore the defined STs and MVs through full refresh. Note that recreation is necessary when pipeline | ||
| properties such as the 'catalog' or 'storage' are changed: | ||
| delete pipeline my_pipelines | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| Local = true | ||
| Cloud = false | ||
|
|
||
| [EnvMatrix] | ||
| DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
|
|
||
| >>> [CLI] bundle validate | ||
| Name: prevent-destroy | ||
| Target: default | ||
| Workspace: | ||
| User: [USERNAME] | ||
| Path: /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default | ||
|
|
||
| Validation OK! | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/prevent-destroy/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> errcode [CLI] bundle plan |
11 changes: 11 additions & 0 deletions
11
acceptance/bundle/lifecycle/prevent-destroy/resources/pipeline.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| resources: | ||
| pipelines: | ||
| my_pipelines: | ||
| name: "test-pipeline" | ||
| libraries: | ||
| - notebook: | ||
| path: "../test-notebook.py" | ||
| lifecycle: | ||
| prevent_destroy: true | ||
| schema: test-schema | ||
| catalog: main |
7 changes: 7 additions & 0 deletions
7
acceptance/bundle/lifecycle/prevent-destroy/resources/schema.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| resources: | ||
| schemas: | ||
| my_schema: | ||
| catalog_name: "test-catalog" | ||
| name: test-schema | ||
| lifecycle: | ||
| prevent_destroy: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| trace $CLI bundle validate | ||
|
|
||
| trace $CLI bundle deploy | ||
|
|
||
| trace errcode $CLI bundle plan | ||
| trace musterr $CLI bundle destroy --auto-approve >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
|
|
||
| # Changing the catalog name, deploy must fail because pipeline will be recreated | ||
| update_file.py resources/pipeline.yml 'catalog: main' 'catalog: mainnew' | ||
| trace errcode $CLI bundle plan >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
| trace musterr $CLI bundle deploy >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
|
|
||
| # Changing the schema name, deploy must fail because schema will be recreated | ||
| update_file.py resources/schema.yml 'name: test-schema' 'name: test-schema-new' | ||
| trace errcode $CLI bundle plan >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
| trace musterr $CLI bundle deploy >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
|
|
||
| # Removing the prevent_destroy, deploy must succeed | ||
| update_file.py resources/pipeline.yml 'prevent_destroy: true' 'prevent_destroy: false' | ||
| update_file.py resources/schema.yml 'prevent_destroy: true' 'prevent_destroy: false' | ||
| trace errcode $CLI bundle plan >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
| trace $CLI bundle deploy --auto-approve >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
| update_file.py resources/pipeline.yml 'prevent_destroy: false' 'prevent_destroy: true' | ||
| update_file.py resources/schema.yml 'prevent_destroy: false' 'prevent_destroy: true' | ||
|
|
||
|
|
||
| # Removing the pipeline and schema, deploy must succeed | ||
| rm resources/pipeline.yml resources/schema.yml | ||
| trace errcode $CLI bundle plan >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 | ||
| trace $CLI bundle deploy --auto-approve >>out.$DATABRICKS_CLI_DEPLOYMENT.txt 2>&1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # Databricks notebook source | ||
|
|
||
| print("Hello, World!") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| EnvVaryOutput = "DATABRICKS_CLI_DEPLOYMENT" | ||
|
|
||
| Ignore = [ | ||
| ".databricks" | ||
| ] | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/serving-endpoints" | ||
| Response.Body = ''' | ||
| { | ||
| "id": "test-endpoint-6260d50f-e8ff-4905-8f28-812345678903", | ||
| "name": "test-endpoint-6260d50f-e8ff-4905-8f28-812345678903" | ||
| } | ||
| ''' | ||
|
|
||
| [[Server]] | ||
| Pattern = "GET /api/2.0/serving-endpoints/" | ||
|
|
||
|
|
||
| [[Server]] | ||
| Pattern = "GET /api/2.0/serving-endpoints/test-endpoint-6260d50f-e8ff-4905-8f28-812345678903" | ||
| Response.Body = ''' | ||
| { | ||
| "id": "test-endpoint-6260d50f-e8ff-4905-8f28-812345678903", | ||
| "permission_level": "CAN_MANAGE", | ||
| "route_optimized": false, | ||
| "state": { | ||
| "config_update": "NOT_UPDATING", | ||
| "ready": "NOT_READY" | ||
| } | ||
| } | ||
| ''' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Local = true | ||
| Cloud = false |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.