feat: add benchmark automation bot [WIP]#3557
Draft
andygrove wants to merge 2 commits intoapache:mainfrom
Draft
Conversation
andygrove
changed the title
andygrove
force-pushed
the
benchmark-bot
branch
2 times, most recently
from
821b893 to
360a448
Compare
…rks on PRs Adds a GitHub bot that monitors PR comments for slash commands (/run tpch, /run micro, /help) and automatically runs benchmarks in Kubernetes, posting results back as PR comments. Includes CLI for manual benchmark runs, Docker image build/push, K8s job management, and deployment tooling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
andygrove
force-pushed
the
benchmark-bot
branch
2 times, most recently
from
9c00b57 to
d43d71c
Compare
andygrove
force-pushed
the
benchmark-bot
branch
from
d43d71c to
51b5008
Compare
Member
Author
|
@Shekharrajak fyi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3556
Summary
I have been testing a version of this code for several weeks and it seems to work fairly well now, so I would like to get the code into OSS for transparency, and allow others to help make improvements.
It is just a first step. The benchmarks do run in k8s in a constrained environment, which is good, but the tests run in Spark local mode. It would be better to deploy as a real cluster in k8s later on.
There is currently an assumption that TPC-H 100GB data already exists on the k8s nodes. It would be better to generate the data using
tpchgen-clidirectly in the containers. It would also be nice to support different scale factors.There are likely many other improvements that can be made in the future.
Changes
cometbot) that monitors PR comments for slash commands (/run tpch,/run micro,/help) and automatically runs benchmarks in Kubernetes, posting results back as PR commentsDetails
The bot lives in
dev/benchmarking-bot/and includes:src/cometbot/bot.py): Polls GitHub for slash commands on open Comet PRssrc/cometbot/k8s.py): Builds Docker images, creates/manages Kubernetes jobssrc/cometbot/cli.py): Click-based CLI for manual benchmark runs and bot managementk8s/): Job and deployment manifestsdeploy/deploy.sh): Automated deployment to a remote host via SSHAll configuration is via
COMETBOT_*environment variables (registry, GitHub token, deploy host, etc.).