When upgrade fails, database can get "disabled" and be in an unrecoverable state

[ramonsmits]

Describe the bug

Description

Upgrade failed for a customer from 5.11.3 to 6.1.2 with this message:

The log files shows the following at start and fails to complete:

2025-01-23 17:52:58.2379|1|Error|Program|Unhandled exception was caught.|Raven.Client.Exceptions.Database.DatabaseDisabledException: Raven.Client.Exceptions.Database.DatabaseDisabledException: primary has been disabled

Was not able to launch instance in maintenance mode either.

Versions

6.1.2

Steps to reproduce

Upgrade from 5.11.3 to 6.1.2 and be lucky

Additional Information

The environment of the customer has virusscanning enabled which might have been the cause for the timeout during upgrade.

[mauroservienti]

Additional requirements to try to reproduce:

• ServiceControl is installed on Windows
• Upgrade is performed using SCMU

[mauroservienti]

We have started working on this issue. We raised and merged a set of pull requests aiming at mitigating the situation:

• Allow JIT killing of Raven.Embedded.EmbeddedServer by tearing it down as part of stop sequence #4824
• Logging events to diagnose stop sequence #4823
• Allow setting shutdown timeout for all instances #4822
• Fix watchdog re-creation of TransportInfrastructure and cancellation improvements #4818
• Analyze RavenDB child process #4786
• Support cancellation of web api and audit storage operations #4782

[ramonsmits]

The following releases include the mentioned improvements:

• https://github.com/Particular/ServiceControl/releases/tag/6.5.0
• https://github.com/Particular/ServiceControl/releases/tag/6.5.2

[udidahan]

Could we make the "contact support" part of the error message be a link to our create support case page, possibly prepopulating it with the relevant information?

[mauroservienti]

Raised:

• Allow raising a support case directly from SCMU error messages #4928