Skip to content

Add warning for when required bundle fields are not set #3044

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shreyas-goenka merged 34 commits into from
Jul 18, 2025
Merged

Add warning for when required bundle fields are not set #3044

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
b7e0bed
Generate required fields for DABs
shreyas-goenka Jun 11, 2025
4c327c5
some cleanup
shreyas-goenka Jun 11, 2025
f612f5a
some cleanup
shreyas-goenka Jun 12, 2025
9199d25
add generate
shreyas-goenka Jun 12, 2025
de8477e
fix and generate
shreyas-goenka Jun 12, 2025
849787f
Merge remote-tracking branch 'origin' into generate-required-v3
shreyas-goenka Jun 12, 2025
427ae3b
minor rename to type
shreyas-goenka Jun 12, 2025
818f0df
Merge remote-tracking branch 'origin' into generate-required-v3
shreyas-goenka Jul 10, 2025
bcb614e
Validate required bundle fields
shreyas-goenka Jun 12, 2025
0c988db
additional validation things
shreyas-goenka Jun 12, 2025
539bd71
again
shreyas-goenka Jun 12, 2025
5c8d934
-
shreyas-goenka Jun 12, 2025
e4124ae
-
shreyas-goenka Jun 12, 2025
6ed0868
deterministic output
shreyas-goenka Jun 12, 2025
7516ec1
deterministic output
shreyas-goenka Jun 12, 2025
126dfb8
-gs
shreyas-goenka Jun 12, 2025
1ab0efb
Add benchmark
shreyas-goenka Jun 13, 2025
3607bb7
add more benchmarks
shreyas-goenka Jun 13, 2025
b112506
lint
shreyas-goenka Jun 13, 2025
5c85284
baseline
shreyas-goenka Jun 13, 2025
b95da1b
refactor
shreyas-goenka Jun 13, 2025
b405af6
more efficient validation
shreyas-goenka Jun 17, 2025
578112a
update the PR with latest changes
shreyas-goenka Jul 10, 2025
7612cf9
lint
shreyas-goenka Jul 10, 2025
fbffb58
merge
shreyas-goenka Jul 11, 2025
159333e
update acc tests
shreyas-goenka Jul 11, 2025
2c68220
revert node test
shreyas-goenka Jul 11, 2025
e12d8f2
fix tests
shreyas-goenka Jul 11, 2025
93ca657
lint
shreyas-goenka Jul 11, 2025
2229963
typo
shreyas-goenka Jul 11, 2025
b4e1596
add empty case
shreyas-goenka Jul 11, 2025
6e9988f
update test and make it better
shreyas-goenka Jul 11, 2025
7031ce7
Merge remote-tracking branch 'origin' into validate-on-required
shreyas-goenka Jul 18, 2025
8a7e423
update golden files
shreyas-goenka Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions acceptance/bundle/artifacts/issue_3109/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
Warning: required field "entry_point" is not set
at resources.jobs.job1.tasks[0].python_wheel_task
in src/job1/databricks_job/resources/job1.yaml:16:17

Building wheels...
Uploading dist/my_default_python-0.0.1-py3-none-any.whl...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/files...
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/debug/direct/out.stderr.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
10:07:59 Debug: Apply pid=12345 mutator=ProcessStaticResources mutator=TranslatePathsDashboards
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(load_resources)
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(apply_mutators)
10:07:59 Debug: Apply pid=12345 mutator=validate:required
10:07:59 Debug: Apply pid=12345 mutator=CheckPermissions
10:07:59 Debug: Apply pid=12345 mutator=TranslatePaths
10:07:59 Debug: Apply pid=12345 mutator=PythonWrapperWarning
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/debug/direct/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Validation OK!
+>>> [CLI] bundle validate --debug
10:07:59 Info: start pid=12345 version=[DEV_VERSION] args="[CLI], bundle, validate, --debug"
10:07:59 Debug: Found bundle root at [TEST_TMP_DIR] (file [TEST_TMP_DIR]/databricks.yml) pid=12345
@@ -60,8 +62,4 @@
@@ -61,8 +63,4 @@
10:07:59 Debug: Apply pid=12345 mutator=metadata.AnnotateJobs
10:07:59 Debug: Apply pid=12345 mutator=metadata.AnnotatePipelines
-10:07:59 Debug: Apply pid=12345 mutator=terraform.Initialize
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/debug/tf/out.stderr.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
10:07:59 Debug: Apply pid=12345 mutator=ProcessStaticResources mutator=TranslatePathsDashboards
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(load_resources)
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(apply_mutators)
10:07:59 Debug: Apply pid=12345 mutator=validate:required
10:07:59 Debug: Apply pid=12345 mutator=CheckPermissions
10:07:59 Debug: Apply pid=12345 mutator=TranslatePaths
10:07:59 Debug: Apply pid=12345 mutator=PythonWrapperWarning
Expand Down
11 changes: 10 additions & 1 deletion acceptance/bundle/paths/fallback/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@

>>> [CLI] bundle validate -t development -o json
Warning: required field "entry_point" is not set
at resources.jobs.my_job.tasks[2].python_wheel_task
in resources/my_job.yml:33:13
override_job.yml:29:17

Warn: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Warn: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
Expand All @@ -10,6 +15,10 @@ Warn: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resourc
Warn: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.

>>> [CLI] bundle validate -t error
Warning: required field "entry_point" is not set
at resources.jobs.my_job.tasks[2].python_wheel_task
in resources/my_job.yml:33:13

Error: notebook "resources/this value is overridden" not found. Local notebook references are expected
to contain one of the following file extensions: [.py, .r, .scala, .sql, .ipynb]

Expand All @@ -19,6 +28,6 @@ Workspace:
User: [USERNAME]
Path: /Workspace/Users/[USERNAME]/.bundle/fallback/error

Found 1 error
Found 1 error and 1 warning

Exit code: 1
47 changes: 46 additions & 1 deletion acceptance/bundle/paths/nominal/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,52 @@

>>> [CLI] bundle validate -t development -o json
Warning: required field "entry_point" is not set
at resources.jobs.my_job.tasks[4].python_wheel_task
in resources/my_job.yml:33:13
override_job.yml:29:17

Warning: required field "inputs" is not set
at resources.jobs.my_job.tasks[1].for_each_task
in resources/my_job.yml:47:13
override_job.yml:45:17

Warning: required field "inputs" is not set
at resources.jobs.my_job.tasks[2].for_each_task
in resources/my_job.yml:53:13
override_job.yml:51:17

Warning: required field "task_key" is not set
at resources.jobs.my_job.tasks[1].for_each_task.task
in resources/my_job.yml:48:15
override_job.yml:46:19

Warning: required field "task_key" is not set
at resources.jobs.my_job.tasks[2].for_each_task.task
in resources/my_job.yml:54:15
override_job.yml:52:19


>>> [CLI] bundle validate -t error
Warning: required field "entry_point" is not set
at resources.jobs.my_job.tasks[4].python_wheel_task
in resources/my_job.yml:33:13

Warning: required field "inputs" is not set
at resources.jobs.my_job.tasks[1].for_each_task
in resources/my_job.yml:47:13

Warning: required field "inputs" is not set
at resources.jobs.my_job.tasks[2].for_each_task
in resources/my_job.yml:53:13

Warning: required field "task_key" is not set
at resources.jobs.my_job.tasks[1].for_each_task.task
in resources/my_job.yml:48:15

Warning: required field "task_key" is not set
at resources.jobs.my_job.tasks[2].for_each_task.task
in resources/my_job.yml:54:15

Error: notebook "resources/this value is overridden" not found. Local notebook references are expected
to contain one of the following file extensions: [.py, .r, .scala, .sql, .ipynb]

Expand All @@ -11,6 +56,6 @@ Workspace:
User: [USERNAME]
Path: /Workspace/Users/[USERNAME]/.bundle/nominal/error

Found 1 error
Found 1 error and 5 warnings

Exit code: 1
8 changes: 8 additions & 0 deletions acceptance/bundle/quality_monitor/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@

>>> [CLI] bundle validate -o json -t development
Warning: required field "quartz_cron_expression" is not set
at resources.quality_monitors.my_monitor.schedule
in databricks.yml:17:9
Copy link
Contributor

@denik denik Jul 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning should not be there, actually. The whole 'schedule' field is cleared by dev mode and it's okay, not user's problem. However, in the current implementation it is set to null. That null should be treated as absence.

I noticed this because my PR improving null handling actually makes schedule disappear and that removes the warnings: https://github.com/databricks/cli/pull/3230/files#diff-18b1807771a343c3d7e7c543232e7ec53637f4a65b1191ab04becc7706b5fcd5L3

However, we should probably look into fixing null handling in required logic specifically. It may also come from other places.

Copy link
Contributor Author

@shreyas-goenka shreyas-goenka Aug 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole 'schedule' field is cleared by dev mode and it's okay

That seems like a mistake if this is a field that is required by the API. Does the API succeed even if this field is missing?


Warning: required field "timezone_id" is not set
at resources.quality_monitors.my_monitor.schedule
in databricks.yml:17:9

{
"mode": "development",
"quality_monitors": {
Expand Down
60 changes: 60 additions & 0 deletions acceptance/bundle/validate/empty_resources/empty_dict/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@
}

=== resources.models.rname ===
Warning: required field "name" is not set
at resources.models.rname
in databricks.yml:6:12

{
"models": {
"rname": {
Expand All @@ -54,27 +58,71 @@
}

=== resources.registered_models.rname ===
Warning: required field "catalog_name" is not set
at resources.registered_models.rname
in databricks.yml:6:12

Warning: required field "name" is not set
at resources.registered_models.rname
in databricks.yml:6:12

Warning: required field "schema_name" is not set
at resources.registered_models.rname
in databricks.yml:6:12

{
"registered_models": {
"rname": {}
}
}

=== resources.quality_monitors.rname ===
Warning: required field "assets_dir" is not set
at resources.quality_monitors.rname
in databricks.yml:6:12

Warning: required field "output_schema_name" is not set
at resources.quality_monitors.rname
in databricks.yml:6:12

Warning: required field "table_name" is not set
at resources.quality_monitors.rname
in databricks.yml:6:12

{
"quality_monitors": {
"rname": {}
}
}

=== resources.schemas.rname ===
Warning: required field "catalog_name" is not set
at resources.schemas.rname
in databricks.yml:6:12

Warning: required field "name" is not set
at resources.schemas.rname
in databricks.yml:6:12

{
"schemas": {
"rname": {}
}
}

=== resources.volumes.rname ===
Warning: required field "catalog_name" is not set
at resources.volumes.rname
in databricks.yml:6:12

Warning: required field "name" is not set
at resources.volumes.rname
in databricks.yml:6:12

Warning: required field "schema_name" is not set
at resources.volumes.rname
in databricks.yml:6:12

{
"volumes": {
"rname": {
Expand Down Expand Up @@ -102,6 +150,14 @@
}

=== resources.apps.rname ===
Warning: required field "name" is not set
at resources.apps.rname
in databricks.yml:6:12

Warning: required field "source_code_path" is not set
at resources.apps.rname
in databricks.yml:6:12

Error: Missing app source code path
in databricks.yml:6:12

Expand Down Expand Up @@ -129,6 +185,10 @@ app resource 'rname' is missing required source_code_path field
}

=== resources.secret_scopes.rname ===
Warning: required field "name" is not set
at resources.secret_scopes.rname
in databricks.yml:6:12

{
"secret_scopes": {
"rname": {
Expand Down
60 changes: 60 additions & 0 deletions acceptance/bundle/validate/empty_resources/with_grants/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ Warning: unknown field: grants
at resources.models.rname
in databricks.yml:7:7

Warning: required field "name" is not set
at resources.models.rname
in databricks.yml:7:7

{
"models": {
"rname": {
Expand All @@ -70,6 +74,18 @@ Warning: unknown field: grants
}

=== resources.registered_models.rname ===
Warning: required field "catalog_name" is not set
at resources.registered_models.rname
in databricks.yml:7:7

Warning: required field "name" is not set
at resources.registered_models.rname
in databricks.yml:7:7

Warning: required field "schema_name" is not set
at resources.registered_models.rname
in databricks.yml:7:7

{
"registered_models": {
"rname": {
Expand All @@ -83,13 +99,33 @@ Warning: unknown field: grants
at resources.quality_monitors.rname
in databricks.yml:7:7

Warning: required field "assets_dir" is not set
at resources.quality_monitors.rname
in databricks.yml:7:7

Warning: required field "output_schema_name" is not set
at resources.quality_monitors.rname
in databricks.yml:7:7

Warning: required field "table_name" is not set
at resources.quality_monitors.rname
in databricks.yml:7:7

{
"quality_monitors": {
"rname": {}
}
}

=== resources.schemas.rname ===
Warning: required field "catalog_name" is not set
at resources.schemas.rname
in databricks.yml:7:7

Warning: required field "name" is not set
at resources.schemas.rname
in databricks.yml:7:7

{
"schemas": {
"rname": {
Expand All @@ -99,6 +135,18 @@ Warning: unknown field: grants
}

=== resources.volumes.rname ===
Warning: required field "catalog_name" is not set
at resources.volumes.rname
in databricks.yml:7:7

Warning: required field "name" is not set
at resources.volumes.rname
in databricks.yml:7:7

Warning: required field "schema_name" is not set
at resources.volumes.rname
in databricks.yml:7:7

{
"volumes": {
"rname": {
Expand Down Expand Up @@ -139,6 +187,14 @@ Warning: unknown field: grants
at resources.apps.rname
in databricks.yml:7:7

Warning: required field "name" is not set
at resources.apps.rname
in databricks.yml:7:7

Warning: required field "source_code_path" is not set
at resources.apps.rname
in databricks.yml:7:7

Error: Missing app source code path
in databricks.yml:7:7

Expand Down Expand Up @@ -174,6 +230,10 @@ Warning: unknown field: grants
at resources.secret_scopes.rname
in databricks.yml:7:7

Warning: required field "name" is not set
at resources.secret_scopes.rname
in databricks.yml:7:7

{
"secret_scopes": {
"rname": {
Expand Down
Loading
Loading