[BCDR] freno - Testing Required

[github-internal-statuspage-new]

BCDR Testing Required

The service freno requires completed BCDR (Business Continuity and Disaster Recovery) tests.

Due Date

📅 Due: May 16, 2026

Why Testing Is Required

To prove our ability to recover from a real disaster, we must demonstrate that we can execute the processes documented in our playbooks. As part of legal requirements for customers, all playbooks must be tested at least once every 12 months.

Current Status

BCDR testing certification expires on 2026-05-16

Test Levels

Tests can be performed at different levels of rigor:

1. Tabletop (Level 1) - Talk through the scenario without executing commands
2. Simulated (Level 2) - Execute recovery in a non-production environment
3. Production (Level 3) - Execute recovery in production with controlled risk
4. Automated (Level 4) - Automated testing integrated into regular operations

Start with the highest level possible without risking production. Even a tabletop exercise is valuable as a stepping stone.

Test Goals

When running a disaster recovery test, aim to:

• Verify automated systems correctly handle the disaster (alerts fire, mitigations trigger, failovers happen)
• Ensure First Responders have all required access to needed systems
• Verify playbooks are easy to follow and provide necessary context
• Calculate actual RTC (Recovery Time Capability) and RPC (Recovery Point Capability)
• Identify gaps and file issues for follow-up

Required Actions

• Complete BCDR Testing
  • Service freno BCDR testing certification expires/expired on 2026-05-16 and needs to be renewed.
  • View details

Before the Test

Ensure you have:

• A playbook for the disaster scenario
• A first responder rotation trained and ready
• A BCDR event created to track the test

During the Test

• Have someone taking notes to document what happened
• First responder should follow the playbook exactly
• Verify first responder can log into all required systems
• Use the same communication channels as during a real disaster
• Do not update the playbook during the test - take notes and file issues afterward

Resources

• Disaster Recovery Exercise Guide
• Required Scenarios
• RTO/RPO Glossary
• ISP Service Page

Need Help?

Reach out in the #bcdr Slack channel where the Incident Lifecycle team can assist.

---

This issue was automatically created by the BCDR Scorecard system. It will be automatically closed when the testing requirement is met.