Skip to content

Conversation

@ddelnano
Copy link
Member

@ddelnano ddelnano commented Dec 10, 2025

Summary: Increase ASAN short test timeout to address flaky tests

The following tests are frequently hitting the 2-minute Bazel timeout on ASAN builds:

  • //src/vizier/services/agent/shared/manager:heartbeat_test
  • //src/vizier/services/agent/shared/manager:registration_test
  • //src/carnot/builtins:collections_test

BuildBuddy history from the main branch shows these tests are running up against the timeout threshold (see screenshot below). I believe BuildBuddy is under reporting the issues seen since builds are also seeing BEP API timeouts.

Screenshot 2025-12-10 at 12 05 55 PM
//src/vizier/services/agent/shared/manager:heartbeat_test               TIMEOUT in 120.5s
  /github/home/.cache/bazel/_bazel_root/56ec069a32c4abebc78228236a835895/execroot/px/bazel-out/k8-dbg/testlogs/src/vizier/services/agent/shared/manager/heartbeat_test/test.log
//src/vizier/services/agent/shared/manager:registration_test            TIMEOUT in 120.5s
  /github/home/.cache/bazel/_bazel_root/56ec069a32c4abebc78228236a835895/execroot/px/bazel-out/k8-dbg/testlogs/src/vizier/services/agent/shared/manager/registration_test/test.log

[ ... ]
ERROR: The Build Event Protocol upload timed out. com.google.common.util.concurrent.TimeoutFuture$TimeoutFutureException: Timed out: NonCancellationPropagatingFuture@6ce6bba6[status=PENDING, info=[delegate=[SettableFuture@29e4285e[status=PENDING]]]]
Bazel returned code 38, ignoring...

This PR increases the short test timeout to unblock ongoing Bazel 7 upgrade work and prevent unrelated PRs from failing due to these timeouts.

Relevant Issues: #2295

Type of change: /kind bugfix

Test Plan: Build succeeds

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano ddelnano changed the title Increase timeout for tests that consistently timeout Increase timeouts for flaky agent manager tests Dec 10, 2025
Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano ddelnano marked this pull request as ready for review December 10, 2025 22:25
@ddelnano ddelnano requested a review from a team as a code owner December 10, 2025 22:25
… test increases

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
@ddelnano ddelnano requested a review from a team as a code owner December 12, 2025 03:58
@ddelnano ddelnano changed the title Increase timeouts for flaky agent manager tests Increase ASAN short test timeout to address flaky tests Dec 12, 2025
@ddelnano ddelnano merged commit f4e30ef into pixie-io:main Dec 12, 2025
28 checks passed
@ddelnano ddelnano deleted the ddelnano/increase-timeout-for-flaky-test branch December 12, 2025 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants