HAProxy Configuration: network.loadbalancer.haproxy.idle.timeout

[bradh352]

Description

This PR adds a new configuration parameter of network.loadbalancer.haproxy.idle.timeout which controls the HAProxy timeout server and timeout client configuration parameters under the defaults section.

Fixes #12574

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

• Embedded unit tests to ensure generated configuration file matches.
• Manual testing

How did you try to break this feature and the system with this change?

It has been observed that this will not update dynamically unless some other load balancer configuration is changed (or the VR is restarted). It is not clear if there is any infrastructure to 'trigger' a push to all existing load balancers to regenerate their configuration since this is global.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.70%. Comparing base (7324ef4) to head (5203bd5).
⚠️ Report is 9 commits behind head on 4.22.

❗ There is a different number of reports uploaded between BASE (7324ef4) and HEAD (5203bd5). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (7324ef4)	HEAD (5203bd5)
unittests	1	0

Additional details and impacted files

@@              Coverage Diff              @@
##               4.22   #12586       +/-   ##
=============================================
- Coverage     17.62%    3.70%   -13.92%     
=============================================
  Files          5917      448     -5469     
  Lines        531255    38018   -493237     
  Branches      64951     7035    -57916     
=============================================
- Hits          93639     1409    -92230     
+ Misses       427077    36422   -390655     
+ Partials      10539      187    -10352     

Flag	Coverage Δ
uitests	3.70% <ø> (ø)
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adds a new global configuration parameter network.loadbalancer.haproxy.idle.timeout to control HAProxy's timeout client and timeout server directives in the defaults section. The default value is 50000 milliseconds (matching the current hardcoded value), and setting it to 0 removes the timeout directives entirely for infinite timeout. This addresses issue #12574 where the aggressive 50-second timeout was problematic for long-running database connections through load balancers.

Changes:

• Added NETWORK_LB_HAPROXY_IDLE_TIMEOUT configuration key with default value of 50000ms
• Updated LoadBalancerConfigCommand to include idleTimeout parameter
• Modified HAProxyConfigurator to generate timeout directives based on the configuration value
• Added health check validation for the idle timeout configuration
• Updated all LoadBalancerConfigCommand instantiations across VR, internal LB, and elastic LB implementations
• Added comprehensive unit tests for timeout configuration generation

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
engine/api/src/main/java/org/apache/cloudstack/engine/orchestration/service/NetworkOrchestrationService.java	Defines the new configuration key for HAProxy idle timeout
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java	Registers the new configuration key in the configurable keys array
core/src/main/java/com/cloud/agent/api/routing/LoadBalancerConfigCommand.java	Adds idleTimeout field and updates constructor signature
core/src/main/java/com/cloud/network/HAProxyConfigurator.java	Implements logic to set or remove timeout directives based on idleTimeout value
server/src/main/java/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java	Passes idle timeout value to load balancing data for virtual routers
server/src/main/java/com/cloud/network/router/CommandSetupHelper.java	Updates LoadBalancerConfigCommand instantiation with idle timeout value
plugins/network-elements/internal-loadbalancer/src/main/java/org/apache/cloudstack/network/lb/InternalLoadBalancerVMManagerImpl.java	Updates LoadBalancerConfigCommand instantiation for internal load balancers
plugins/network-elements/elastic-loadbalancer/src/main/java/com/cloud/network/lb/ElasticLoadBalancerManagerImpl.java	Updates LoadBalancerConfigCommand instantiation for elastic load balancers
systemvm/debian/root/health_checks/haproxy_check.py	Adds health check validation for idle timeout configuration (contains bugs)
core/src/test/java/com/cloud/network/HAProxyConfiguratorTest.java	Adds tests for timeout configuration with 0 and non-zero values
core/src/test/java/com/cloud/agent/resource/virtualnetwork/VirtualRoutingResourceTest.java	Updates test fixtures with idle timeout parameter
core/src/test/java/com/cloud/agent/resource/virtualnetwork/ConfigHelperTest.java	Updates test fixtures with idle timeout parameter

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16705

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-15382)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 58689 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12586-t15382-kvm-ol8.zip
Smoke tests completed. 146 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestListIdsParams>:teardown	Error	1.13	test_list_ids_parameter.py
test_router_dhcphosts	Failure	15.87	test_router_dhcphosts.py
test_01_snapshot_root_disk	Error	4.93	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	48.66	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	Error	48.66	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown	Error	27.36	test_snapshots.py
test_01_snapshot_usage	Error	34.86	test_usage.py
test_01_vpn_usage	Error	1.08	test_usage.py

[bhouse-nexthop]

hrm, pretty sure those failures can't be related to this PR

[DaanHoogland]

hrm, pretty sure those failures can't be related to this PR

nah

[bhouse-nexthop]

Should I rebase this for v4.22 so it gets included in the next patch? It would be really useful to us if we could start using the official releases. I think the other PRs I opened that we're running patches for have merged and will be in the next release.

[DaanHoogland]

@bhouse-nexthop , you can

rebase this for v4.22

or even to 4.20 if you wish, We will merge forwards ad call you in if that gives any issues.

[bradh352]

Ok, its moved to 4.22 now...

[DaanHoogland]

lgtm generally, one remark and needs 3rd party testing

[bradh352]

I should mention I built on ubuntu 22.04, and tested on ubuntu 24.04 this using this build sequence:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash - && \
sudo apt-get update && \
sudo apt-get install -y build-essential git devscripts debhelper openjdk-17-jdk libws-commons-util-java genisoimage libcommons-codec-java libcommons-httpclient-java liblog4j1.2-java maven nodejs python3-mysql.connector python2 python-setuptools jq && \
git clone -b 4.22.0.0 https://github.com/apache/cloudstack && \
wget https://patch-diff.githubusercontent.com/raw/apache/cloudstack/pull/11932.patch && \
wget https://patch-diff.githubusercontent.com/raw/apache/cloudstack/pull/11967.patch && \
wget https://patch-diff.githubusercontent.com/raw/apache/cloudstack/pull/12586.patch && \
cd cloudstack && \
git switch -c dev-build && \
git config user.email "someone@example.com" && \
git am < ../11932.patch && \
git am < ../11967.patch && \
git am < ../12586.patch && \
./tools/build/setnextversion.sh -v 4.22.0.3 -n -b dev-build -s $(pwd) && \
cd packaging && \
NODE_OPTIONS=--openssl-legacy-provider ACS_BUILD_OPTS="-DskipTests -T `nproc`" ./build-deb.sh

(A couple other patches were pulled in that our environment needs)