Skip to content

Commit 5d5ac53

Browse files
alyssa-dbalyssa-db
authored
pipelines: add acc tests for run (#3203)
## Changes Acceptance tests for pipeline run command. ## Tests 4 folders of acceptance tests: - restart for --restart flag - refresh for 4 refresh flags - run-pipeline to test default run command out.test.toml /output.txt is generated Depends on merging of #3178 as needs mocks from testserver and handler Follows #3132
1 parent a7815f1 commit 5d5ac53

File tree

22 files changed

+251
-0
lines changed

22 files changed

+251
-0
lines changed

‎acceptance/pipelines/run/no-wait/databricks.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
bundle:
2+
name: test-pipeline-run-flags
3+
4+
resources:
5+
pipelines:
6+
my_pipeline:
7+
name: test-pipeline-flags
8+
libraries:
9+
- file:
10+
path: pipeline_file.py

‎acceptance/pipelines/run/no-wait/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Local = true
2+
Cloud = false
3+
4+
[EnvMatrix]
5+
DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"]

‎acceptance/pipelines/run/no-wait/output.txt‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
2+
>>> [PIPELINES] deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files...
4+
Deploying resources...
5+
Updating deployment state...
6+
Deployment complete!
7+
8+
=== test --no-wait flag
9+
>>> [PIPELINES] run my_pipeline --no-wait
10+
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
11+
12+
<EOL>

‎acceptance/pipelines/run/no-wait/pipeline_file.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import dlt
2+
from pyspark.sql import SparkSession
3+
4+
5+
@dlt.table
6+
def my_table():
7+
return spark.range(10)

‎acceptance/pipelines/run/no-wait/script‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
trace $PIPELINES deploy
2+
title "test --no-wait flag"
3+
trace $PIPELINES run my_pipeline --no-wait
4+
# This printf is here to fix the whitespace linter error (ensures exactly one newline at end)
5+
printf "<EOL>\n"

‎acceptance/pipelines/run/refresh-flags/databricks.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
bundle:
2+
name: test-pipeline-run-flags
3+
4+
resources:
5+
pipelines:
6+
my_pipeline:
7+
name: test-pipeline-flags
8+
libraries:
9+
- file:
10+
path: pipeline_file.py

‎acceptance/pipelines/run/refresh-flags/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Local = true
2+
Cloud = false
3+
4+
[EnvMatrix]
5+
DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"]

‎acceptance/pipelines/run/refresh-flags/output.txt‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
2+
>>> [PIPELINES] deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/default/files...
4+
Deploying resources...
5+
Updating deployment state...
6+
Deployment complete!
7+
8+
=== Running pipeline with --refresh-all flag
9+
>>> [PIPELINES] run my_pipeline --refresh-all
10+
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
11+
12+
13+
>>> print_requests
14+
{
15+
"body": {},
16+
"method": "POST",
17+
"path": "/api/2.0/pipelines/[UUID]/updates"
18+
}
19+
20+
=== Running pipeline with --refresh flag and specific tables
21+
>>> [PIPELINES] run my_pipeline --refresh table1,table2
22+
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
23+
24+
25+
>>> print_requests
26+
{
27+
"body": {
28+
"refresh_selection": [
29+
"table1",
30+
"table2"
31+
]
32+
},
33+
"method": "POST",
34+
"path": "/api/2.0/pipelines/[UUID]/updates"
35+
}
36+
37+
=== Running pipeline with --full-refresh-all flag
38+
>>> [PIPELINES] run my_pipeline --full-refresh-all
39+
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
40+
41+
42+
>>> print_requests
43+
{
44+
"body": {
45+
"full_refresh": true
46+
},
47+
"method": "POST",
48+
"path": "/api/2.0/pipelines/[UUID]/updates"
49+
}
50+
51+
=== Running pipeline with --full-refresh flag and specific tables
52+
>>> [PIPELINES] run my_pipeline --full-refresh table1,table2
53+
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
54+
55+
56+
>>> print_requests
57+
{
58+
"body": {
59+
"full_refresh_selection": [
60+
"table1",
61+
"table2"
62+
]
63+
},
64+
"method": "POST",
65+
"path": "/api/2.0/pipelines/[UUID]/updates"
66+
}

‎acceptance/pipelines/run/refresh-flags/pipeline_file.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import dlt
2+
from pyspark.sql import SparkSession
3+
4+
5+
@dlt.table
6+
def my_table():
7+
return spark.range(10)

‎acceptance/pipelines/run/refresh-flags/script‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
print_requests() {
2+
jq --sort-keys 'select(.method != "GET" and (.path | contains("/pipelines")))' < out.requests.txt
3+
rm out.requests.txt
4+
}
5+
6+
trace $PIPELINES deploy
7+
rm out.requests.txt
8+
9+
title "Running pipeline with --refresh-all flag"
10+
# Should perform a full graph update (empty refresh_selection and full_refresh_selection implies refresh_all)
11+
trace $PIPELINES run my_pipeline --refresh-all
12+
trace print_requests
13+
14+
title "Running pipeline with --refresh flag and specific tables"
15+
# Should refresh only the specified tables
16+
trace $PIPELINES run my_pipeline --refresh table1,table2
17+
trace print_requests
18+
19+
title "Running pipeline with --full-refresh-all flag"
20+
# Should have full refresh set to true
21+
trace $PIPELINES run my_pipeline --full-refresh-all
22+
trace print_requests
23+
24+
title "Running pipeline with --full-refresh flag and specific tables"
25+
# Should reset and recompute only the specified tables
26+
trace $PIPELINES run my_pipeline --full-refresh table1,table2
27+
trace print_requests

0 commit comments

Comments
 (0)