ci: run TPC-H benchmarks on a Kind Kubernetes cluster by Shekharrajak · Pull Request #3549 · apache/datafusion-comet

Shekharrajak · 2026-02-19T07:19:42Z

Which issue does this PR close?

Add GitHub CI workflow to run TPC-H benchmarks on a Kind Kubernetes cluster, validating Comet performance achieves ≥1.1x speedup over Spark baseline.

Closes #3537

Rationale for this change

Run Spark baseline benchmark
Run Comet benchmark
Validate speedup ≥ 1.1x (10% improvement)

What changes are included in this PR?

The workflow triggers on PRs modifying:

native/**/*.rs
spark/**/*.scala
spark/**/*.java

How are these changes tested?

Local Testing

# Setup cluster
./hack/k8s-benchmark-setup.sh
 
# Run benchmarks
./benchmarks/scripts/run-k8s-benchmark.sh spark q1
./benchmarks/scripts/run-k8s-benchmark.sh comet q1
 
# Compare results
python3 benchmarks/scripts/compare-results.py \
    --spark /tmp/comet-bench-results/spark_q1_result.json \
    --comet /tmp/comet-bench-results/comet_q1_result.json \
    --min-speedup 1.1
 
# Cleanup
./hack/k8s-benchmark-setup.sh --delete

Shekharrajak · 2026-02-19T10:14:21Z

.github/workflows/k8s_benchmark.yml

+on:
+  pull_request:
+    paths:
+      - "native/**/*.rs"


only code changes will trigger this .

Shekharrajak · 2026-02-19T10:14:50Z

.github/workflows/k8s_benchmark.yml

+env:
+  RUST_VERSION: stable
+  K8S_VERSION: "1.32.0"
+  SPARK_VERSION: "3.5"


we can expand it for different spark versions

Shekharrajak · 2026-02-19T10:15:14Z

.github/workflows/k8s_benchmark.yml

+
+      - name: Install K8s tools
+        run: |
+          curl -Lo ./kind "https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-amd64"


Using stable kind version to create k8s cluster

Shekharrajak · 2026-02-19T10:19:36Z

benchmarks/scripts/compare-results.py

+    print(f"Required: {min_speedup:.2f}x")
+    print("-" * 50)
+
+    if speedup >= min_speedup:


This will make sure that we are not degrading the performance - we can have bunch of queries, joins, read, write.

Shekharrajak · 2026-02-19T10:21:42Z

benchmarks/Dockerfile.k8s

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM apache/spark:3.5.8 AS builder


We will extend for multiple spark versions

Shekharrajak · 2026-02-19T10:22:35Z

hack/k8s-benchmark-rbac.yaml

+
+---
+apiVersion: v1
+kind: ServiceAccount


service account and RBAC is needed for driver/executors to have k8s API access.

Shekharrajak · 2026-02-19T10:22:58Z

hack/k8s-benchmark-setup.sh

+    helm repo add spark-operator https://kubeflow.github.io/spark-operator 2>/dev/null || true
+    helm repo update
+
+    if helm list -n spark-operator 2>/dev/null | grep -q spark-operator; then


We are using kubeflow/spark-operator

we can extend it for apache-spark-operator as well but under the hood both have similar way of running spark components.

Shekharrajak · 2026-02-19T10:24:02Z

hack/kind-benchmark-config.yaml

+
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:


Kind k8s cluter worker nodes configs

Shekharrajak · 2026-02-19T10:24:51Z

@andygrove , can I get access to trigger the workflow - this will help me to validate quicker.

Shekharrajak · 2026-02-19T12:15:52Z

.github/workflows/k8s_benchmark.yml

+
+on:
+  pull_request:
+    # paths:


Commenting this so that CI check get triggered.

Shekharrajak · 2026-02-19T17:41:35Z

benchmarks/Dockerfile.k8s


-COPY --from=builder /comet/spark/target/comet-spark-spark${SPARK_VERSION}_${SCALA_VERSION}-*.jar $SPARK_HOME/jars/
+ARG COMET_JAR
+COPY ${COMET_JAR} $SPARK_HOME/jars/


building from source in dockerfile takes time.

andygrove · 2026-02-19T19:28:29Z

@andygrove , can I get access to trigger the workflow - this will help me to validate quicker.

I don't have a way to do that, I'm afraid.

andygrove · 2026-02-19T19:32:00Z

@Shekharrajak Thanks for looking at this, but I have some concerns about this approach. CI takes way too long already and adding this workflow seems like it would add even more overhead. I am not sure that GitHub runners will provide consistent performance for benchmarking, and we really need to be testing with large data sets for meaningful results.

Committers already have the ability to trigger TPC-H benchmarks @ 100GB by commenting on PRs. This is quite new and experimental, so it hasn't been documented yet. These benchmarks run on dedicated hardware.

Shekharrajak · 2026-02-19T20:24:45Z

@Shekharrajak Thanks for looking at this, but I have some concerns about this approach. CI takes way too long already and adding this workflow seems like it would add even more overhead. I am not sure that GitHub runners will provide consistent performance for benchmarking, and we really need to be testing with large data sets for meaningful results.

Committers already have the ability to trigger TPC-H benchmarks @ 100GB by commenting on PRs. This is quite new and experimental, so it hasn't been documented yet. These benchmarks run on dedicated hardware.

So do we expect some kind of script that can help running benchmarks in k8s cluster in local?

andygrove · 2026-02-19T20:30:04Z

@Shekharrajak Thanks for looking at this, but I have some concerns about this approach. CI takes way too long already and adding this workflow seems like it would add even more overhead. I am not sure that GitHub runners will provide consistent performance for benchmarking, and we really need to be testing with large data sets for meaningful results.
Committers already have the ability to trigger TPC-H benchmarks @ 100GB by commenting on PRs. This is quite new and experimental, so it hasn't been documented yet. These benchmarks run on dedicated hardware.

So do we expect some kind of script that can help running benchmarks in k8s cluster in local?

The benchmarks do run in k8s already, but using local mode rather than truly distributed. I am planning on making that change, and I also need to align this with the benchmark scripts currently in the repo, which I am working on refactoring in #3538 and #3539. I planned on supporting k8s as a future step after the docker-compose one gets merged.

Shekharrajak · 2026-02-20T11:30:25Z

The benchmarks do run in k8s already, but using local mode rather than truly distributed. I am planning on making that change, and I also need to align this with the benchmark scripts currently in the repo, which I am working on refactoring in #3538 and #3539. I planned on supporting k8s as a future step after the docker-compose one gets merged.

Thanks for sharing. Then I think no more work required as part of #3537 until those PRs are merged. We can close this PR for now.

Shekharrajak force-pushed the feature/3537-k8s-benchmark-ci branch from 9f78801 to 0b44de0 Compare February 19, 2026 08:14

Shekharrajak commented Feb 19, 2026

View reviewed changes

Shekharrajak force-pushed the feature/3537-k8s-benchmark-ci branch 2 times, most recently from 945c8c4 to 348ea4a Compare February 19, 2026 12:15

Shekharrajak commented Feb 19, 2026

View reviewed changes

.github/workflows/k8s_benchmark.yml

on:

pull_request:

# paths:

Copy link

Contributor Author

Shekharrajak Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commenting this so that CI check get triggered.

Shekharrajak commented Feb 19, 2026

View reviewed changes

Shekharrajak added 7 commits February 19, 2026 23:12

Add Kind cluster setup scripts for K8s benchmarks

9cde940

Add Dockerfile for K8s benchmark image

7c70e52

Add TPC-H benchmark runner scripts and config

473a563

Add GitHub CI workflow for K8s benchmark validation

c9429a7

Add GitHub CI workflow for K8s benchmark validation

d41ce74

Fix setup-builder action to use sudo for apt-get

c81e153

Fix Docker build timeout by using pre-built JAR

d65fffd

Shekharrajak force-pushed the feature/3537-k8s-benchmark-ci branch from ac4eed3 to d65fffd Compare February 19, 2026 17:42

mbutrovich closed this Feb 20, 2026

Comments

Conversation

Shekharrajak commented Feb 19, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Local Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shekharrajak commented Feb 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented Feb 19, 2026

Uh oh!

andygrove commented Feb 19, 2026

Uh oh!

Shekharrajak commented Feb 19, 2026

Uh oh!

andygrove commented Feb 19, 2026

Uh oh!

Shekharrajak commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants