Skip to content

Commit 5714b49

Browse files
Translate relative paths root_path and glob.include for pipelines (#2931)
## Why Translating these fields is necessary for Lakeflow authoring experience to work well with DABs in the workspace. We expect users to set these fields manually in their bundle config. For: 1. `root_path`: We translate the relative root_path to an absolute path in the workspace. Since we assume that the root_path is within the sync root for the bundle (i.e., it's managed by the bundle), we error if it does not exist or lies outside the bundle root. 2. `glob.include`: We directly translate relative glob patterns to absolute ones. Performing any client-side validation requires keeping the validation logic in sync with the server-side definition, so for now, we'll omit performing any client-side validation for glob patterns specified in `glob.include`. ## Tests Acceptance tests --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
1 parent 48e0120 commit 5714b49

File tree

20 files changed

+119
-6
lines changed

20 files changed

+119
-6
lines changed
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
resources:
2+
pipelines:
3+
abc:
4+
root_path: "foo/bar/doesnotexist"
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2+
>>> [CLI] bundle validate
3+
Error: stat foo/bar/doesnotexist: no such file or directory
4+
5+
Name: test-bundle
6+
Target: default
7+
Workspace:
8+
User: [USERNAME]
9+
Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
10+
11+
Found 1 error
12+
13+
Exit code: 1
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
errcode trace $CLI bundle validate
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Replace error message from windows
2+
[[Repls]]
3+
Old = "CreateFile foo/bar/doesnotexist: The system cannot find the path specified."
4+
New = "stat foo/bar/doesnotexist: no such file or directory"
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
bundle:
2+
name: pipelines_relative_path_translation
3+
4+
include:
5+
- resources/*.yml
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
2+
>>> [CLI] bundle validate -o json
3+
{
4+
"one_libraries": [
5+
{
6+
"glob": {
7+
"include": "/Workspace/Users/[USERNAME]/.bundle/pipelines_relative_path_translation/default/files/src/pipeline_one/a/b/c/**"
8+
}
9+
}
10+
],
11+
"two_libraries": [
12+
{
13+
"glob": {
14+
"include": "/Workspace/Users/me@company.com/a/b/c/**"
15+
}
16+
}
17+
],
18+
"one_root_path": "/Workspace/Users/[USERNAME]/.bundle/pipelines_relative_path_translation/default/files/src/pipeline_one",
19+
"two_root_path": "/Workspace/Users/me@company.com/src"
20+
}
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
resources:
2+
pipelines:
3+
# relative paths should get translated to remote paths
4+
one:
5+
root_path: "../src/pipeline_one"
6+
libraries:
7+
- glob:
8+
include: "../src/pipeline_one/a/b/c/**"
9+
10+
# absolute paths should remain as-is
11+
two:
12+
root_path: "/Workspace/Users/me@company.com/src"
13+
libraries:
14+
- glob:
15+
include: "/Workspace/Users/me@company.com/a/b/c/**"
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
trace $CLI bundle validate -o json | jq '{one_libraries: .resources.pipelines.one.libraries, two_libraries: .resources.pipelines.two.libraries, one_root_path: .resources.pipelines.one.root_path, two_root_path: .resources.pipelines.two.root_path}'

acceptance/bundle/paths/pipelines_glob_include_and_root_path/src/pipeline_one/.gitkeep

Whitespace-only changes.
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
bundle:
2+
name: pipelines_root_path_outside_sync_root
3+
4+
resources:
5+
pipelines:
6+
abc:
7+
root_path: ../../../src

0 commit comments

Comments
 (0)