[Python] Add databricks.bundles.core #2469
Conversation
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
kanterov
force-pushed
the
add-databricks-bundles-py-pt1
branch
from
1d609f3 to
fa6e025
Compare
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
kanterov
force-pushed
the
add-databricks-bundles-py-pt1
branch
from
fa6e025 to
fb7d2ba
Compare
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
kanterov
requested review from
andrewnester,
denik,
pietern and
shreyas-goenka
as code owners
pietern
left a comment
pietern
left a comment
There was a problem hiding this comment.
Approving to unblock.
Am I right to see that not all parts of the code here are still relevant? For example, the job mutator decorator, I believe, is no longer recommended in favor of listing the functions to apply in apply_mutators.
| new_bundle = _append_resources(input, resources) | ||
| locations = _relativize_locations(resources._locations) | ||
|
|
||
| return new_bundle, locations, diagnostics.extend(resources.diagnostics) |
There was a problem hiding this comment.
These phases can be pulled out into separate functions.
|
|
||
| if instance and not isinstance(instance, ResourceMutator): | ||
| return None, Diagnostics.create_error( | ||
| f"'{name}' in module '{module_name}' is not instance of ResourceMutator, did you decorate it with @job_mutator?", |
There was a problem hiding this comment.
Is this decorator still used?
The mutators as configured in the bundle are a list of functions already, no?
There was a problem hiding this comment.
We still need decorator to define the type of resources that are handled by this function. It will be relevant when we have more than 1.
| def job_mutator(function: Callable[["Job"], "Job"]) -> ResourceMutator["Job"]: ... | ||
|
|
||
|
|
||
| def job_mutator(function: Callable) -> ResourceMutator["Job"]: |
There was a problem hiding this comment.
With apply_mutators, is this decorator still relevant?
There was a problem hiding this comment.
We still need decorator to define the type of resources that are handled by this function. It will be relevant when we have more than 1.
|
Discussed with @kanterov and agreed to address comments in a follow-up PR. |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces the new databricks.bundles.core package for resource‐independent Python support, adding core functionality for diagnostics, resource loading, transformation, and variable handling. Key changes include:
- Implementation of core modules (e.g. _bundle.py, _diagnostics.py, _location.py, _transform.py, _variable.py) that provide the foundation for resource processing.
- A comprehensive suite of unit tests under experimental/python/databricks_tests to ensure correct behavior.
- Introduction of job mutators and a standardized resources API to support bundle configuration.
Reviewed Changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| experimental/python/databricks_tests/core/test_bundle.py | Unit tests verifying variable resolution and error messages in Bundle. |
| experimental/python/databricks/bundles/core/_resource.py | Definition of the base Resource class. |
| experimental/python/databricks/bundles/core/_diagnostics.py | Implementation of Diagnostics with error and warning creation methods. |
| experimental/python/databricks/bundles/jobs/_models/job.py | Job model using transformation utilities for deserialization. |
| experimental/python/databricks/bundles/core/_resource_mutator.py | Decorator for job mutators with a minor spelling issue in a comment. |
| experimental/python/databricks/bundles/core/_load.py | Modules for scanning and loading resources with error handling. |
| experimental/python/databricks/bundles/core/_transform_to_json.py | Utility to transform Python objects to JSON‐compatible types. |
| experimental/python/databricks/bundles/core/_resources.py | Resources class for aggregating jobs, tracking diagnostics and source locations. |
| experimental/python/databricks_tests/core/test_transform.py | Extensive tests for type transformations and forward reference resolution. |
| experimental/python/databricks/bundles/core/_variable.py | Implementation of bundle variable handling and helper type unions. |
| experimental/python/databricks_tests/core/test_resources.py | Tests validating Resources API, duplicate job handling and diagnostic adds. |
| experimental/python/databricks/bundles/core/_bundle.py | Bundle class implementing variable resolution and list handling via transform utilities. |
| experimental/python/databricks/bundles/core/_transform.py | Complex type transformation logic for dataclasses, collections, and primitives. |
| experimental/python/databricks/bundles/core/init.py | Re-exports core types and functions for public API access. |
Comments suppressed due to low confidence (2)
experimental/python/databricks/bundles/core/_resource_mutator.py:29
- Typo in comment: 'Underling' should be corrected to 'Underlying'.
"Underling function that was decorated. Can be accessed for unit-testing."
experimental/python/databricks/bundles/core/_bundle.py:102
- The implementation of 'resolve_variable_list' assumes that the resolved variable is iterable. Consider adding validation to ensure that the resolved value is a list and provide a clearer error if it is not.
return [self.resolve_variable(item) for item in self.resolve_variable(variable)]
Changes
Add databricks.bundles.core package to experimental/python.
It contains an implementation of resource-independent code for Python support. As a follow-up, we will add Python code generated using bundle schema.
Tests
Unit tests