RavenDB Server child process remains running

[ramonsmits]

Describe the bug

Description

RavenDB Server child process remains running after SC instance is stopped. This causes issues when the instance is (re)started.

Restarting or shutting down the server results in the RavenDB instance to be terminated which might cause the database to do lots of IO when starts as it detects its data might be incorrect.

This is already a problem but even more when performing an upgrade as this can result in the upgrade to fail (database unavailable or just not stopping).

https://www.youtube.com/watch?v=nkRYMQXZDQE

Expected behavior

RavenDB Server Child process to gracefully exit before service control

Actual behavior

Versions

6.3.0 and prior 6.x and 5.x versions that use RavenDB Server as child processes.

Steps to reproduce

1. host instance
2. load test instance

Ensure that the RavenDB studio has plenty of index backlog

3. stop instance

Observe that the instance process eventually stops but that the RavenDB Server process to remain running

Relevant log output

Additional Information

Workarounds

1. Run RavenDB Server as a local Windows or remote service
2. Manually stop and use Task Manager, resource Manager and/or SysInternals ProcExp to monitor if all processes eventually stop

Possible solutions

1. See workaround
2. Kill child process after stop timeout exceeds (Stop cancellation token) at the risk of flagging the database as unhealthy

[ramonsmits]

Release https://github.com/Particular/ServiceControl/releases/tag/6.5.0 made significant stability improvements.

Specifically the following should have improved the shutdown experience of the RavenDB child process:

• Allow setting shutdown timeout for all instances #4822

However, analysis shows that even if the process is killed that the process still be present in the OS and prevent launching a new instance because the OS could be busy for a long time flush dirty memory mapped pages to disk.

TL;DR: Its fixed by 6.5.0, but only partially due to OS design.