diff --git a/INCIDENT_REPORT_TEMPLATE.md b/INCIDENT_REPORT_TEMPLATE.md new file mode 100644 index 0000000..d4884a4 --- /dev/null +++ b/INCIDENT_REPORT_TEMPLATE.md @@ -0,0 +1,28 @@ +# YYYY-MM-DD Incident Report + + + +- Incident Commander: +- Severity Level: + + + +## Timeline + + + +## Impact + + + +## Root Cause + + + +## Fix + + + +## Follow-up Work + + diff --git a/incidents/2025-10-20.md b/incidents/2025-10-20.md new file mode 100644 index 0000000..50e89f5 --- /dev/null +++ b/incidents/2025-10-20.md @@ -0,0 +1,44 @@ +# 2025-10-20 Incident Report + +- Incident Commander: @bmuenzenmeyer +- Severity Level: P2 + +For a brief period of time, the downloads page was blank due to a false assumption on their always being an active LTS version of Node. + +## Timeline + +- **2025-10-20 11:58 PM UTC**: Start of impact. v22.21.0 was promoted, updating [index.json](https://nodejs.org/index.json). + +- **2025-10-21 12:33 AM UTC**: Initial report of incident [nodejs/nodejs.org#8248](https://github.com/nodejs/nodejs.org/issues/8248) created. + +- **2025-10-21 12:55 AM UTC**: Initial report acknowledged in [OpenJS Slack](https://openjs-foundation.slack.com/archives/CVAMEJ4UV/p1761008123880979), investigation began. + +- **2025-10-21 01:09 AM UTC**: Root cause identified. + +- **2025-10-21 01:21 AM UTC**: [nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251) opened to resolve issue. + +- **2025-10-21 01:30 AM UTC**: [nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251) merged. Impact stopped shortly after. + +## Impact + +Users navigating to `https://nodejs.org/en/download` would see a blank page instead of the proper download instructions. + +`https://nodejs.org/en/download/current` was unaffected by this issue. + +## Root Cause + +It was assumed that there would always be an active LTS version of Node.js. +This was not the case with the release of v22.21.0, since v22 moved into Maintenance LTS while v24 still had a week until it became Active LTS. + +Since there was no active LTS release, the downloads page rendered nothing. + +## Fix + +The fix ([nodejs/nodejs.org#8251](https://github.com/nodejs/nodejs.org/pull/8251)) was to make the downloads page look for the Maintenance LTS version if the Active LTS version does not exist. + +## Follow-up Work + +- Better in-the-moment contemplation of use of status page +- Better Playwright tests are needed + - We need to test the actual content of the pages instead of just testing for a successful status code. + - [nodejs/nodejs.org#8249](https://github.com/nodejs/nodejs.org/issues/8249)