Add circuit breaker pattern to vMCP health monitoring#3528
Conversation
There was a problem hiding this comment.
Large PR Detected
This PR exceeds 1000 lines of changes and requires justification before it can be reviewed.
How to unblock this PR:
Add a section to your PR description with the following format:
## Large PR Justification
[Explain why this PR must be large, such as:]
- Generated code that cannot be split
- Large refactoring that must be atomic
- Multiple related changes that would break if separated
- Migration or data transformationAlternative:
Consider splitting this PR into smaller, focused changes (< 1000 lines each) for easier review and reduced risk.
See our Contributing Guidelines for more details.
This review will be automatically dismissed once you add the justification section.
There was a problem hiding this comment.
Pull request overview
This PR implements a circuit breaker pattern for backend health monitoring in the Virtual MCP Server to prevent cascading failures. The circuit breaker tracks backend failures and transitions through three states (Closed, Open, HalfOpen) to protect the system when backends are experiencing issues.
Changes:
- Adds CircuitBreaker implementation with thread-safe state machine for tracking failures and managing state transitions
- Integrates circuit breaker with StatusTracker and Monitor for health check gating
- Includes comprehensive test coverage for circuit breaker logic and integration scenarios
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/vmcp/health/circuit_breaker.go | New implementation of CircuitBreaker with state machine logic and thread-safe operations |
| pkg/vmcp/health/circuit_breaker_test.go | Comprehensive unit tests for circuit breaker state transitions, concurrency, and edge cases |
| pkg/vmcp/health/status.go | Integration of circuit breaker into status tracker with initialization, state querying, and health check gating |
| pkg/vmcp/health/status_test.go | Integration tests for circuit breaker with status tracker including disabled mode scenarios |
| pkg/vmcp/health/monitor.go | Circuit breaker configuration, initialization, and integration with health check loop |
| pkg/vmcp/health/monitor_test.go | End-to-end tests for circuit breaker behavior including disabled, enabled, recovery, and status reporting scenarios |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Large PR justification has been provided. Thank you!
|
✅ Large PR justification has been provided. The size review has been dismissed and this PR can now proceed with normal review. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3528 +/- ##
==========================================
+ Coverage 65.47% 65.68% +0.21%
==========================================
Files 404 407 +3
Lines 39781 40195 +414
==========================================
+ Hits 26046 26403 +357
- Misses 11721 11754 +33
- Partials 2014 2038 +24 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Implement circuit breaker state machine for backend health monitoring in Virtual MCP Server. The circuit breaker prevents cascading failures by tracking backend failures and transitioning through three states: Closed (normal operation), Open (failing), and HalfOpen (recovery testing).
Key changes:
The circuit breaker is disabled by default for backwards compatibility. Configuration wiring from YAML/CRD to health monitor is intentionally left for a follow-up PR to keep this change focused on the core implementation.
Large PR Justification
This is an atomic PR that exposes the circuit breaker capability. It includes comprehensive tests to validate it. Cannot be split as it's an isolated functionality.
Related-to: #3036