From 20695b60efc39cb1c6254d1d9209bb2b0b0fa26a Mon Sep 17 00:00:00 2001 From: Mauro Servienti Date: Wed, 26 Feb 2025 11:23:01 +0100 Subject: [PATCH 1/5] Document corrupted indexes or corrupted database after a service shutdown --- servicecontrol/troubleshooting.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/servicecontrol/troubleshooting.md b/servicecontrol/troubleshooting.md index a27cb0abc72..f8e1afd3e11 100644 --- a/servicecontrol/troubleshooting.md +++ b/servicecontrol/troubleshooting.md @@ -384,3 +384,34 @@ To mitigate growth or not having enough storage: 7. Scale out audit storage over multiple disks and/or machines: - [ServiceControl remote instances Sharding audit messages with split audit queues](/servicecontrol/servicecontrol-instances/remotes.md#overview-sharding-audit-messages-with-split-audit-queues) + +## Audit instances: Corrupted indexes or corrupted database after a service shutdown + +When the following conditions are met: + +- ServiceControl Audit instances are installed on Windows as a service +- The audit database size is massive (> 500Gb) +- There is a constant load on the database due to: + - Continuously ingesting messages from the audit queue + - Message expiration kicking in to delete expired audit messages +- Database indexes use the Corax indexing engine + +There is a chance that, at service shutdown, ServiceControl takes a long time to shut down and, in most cases, doesn't shut down gracefully because the RavenDB database is busy updating indexes due to ingestion and cleaning up tombstones due to retention. + +To mitigate this situation, migrating indexes from the Corax to the Lucene indexing engine can solve the issue. It might be sufficient to migrate to Lucene the `MessagesViewIndex` (regardless of the fact that full-text search is enabled or not), which is the one with the highest load. + +To migrate indexes from the Corax to the Lucene indexing engine, perform the following steps: + +1. Start the ServiceControl Audit instance in maintenance mode +2. Access the RavenDB studio +3. Edit the index that needs to be changed +4. From the edit index Configuration tab +5. Change the indexing engine from Corax or Corax (inherited) to Lucene +6. Click save + +At this point, there will be two indexes, the original one and the new one with the Lucene indexing engine. The RavenDB studio will offer the option to swap them. The swap operation will: + +- Make the Lucene index the default +- Delete the Corax index + +After the swap operation, the new Lucene-based index must be rebuilt. Depending on the index size, the operation might take a long time. From 9963390a88d2ab218834d8d14ffd8567e87f74cc Mon Sep 17 00:00:00 2001 From: Mauro Servienti Date: Thu, 27 Feb 2025 08:10:11 +0100 Subject: [PATCH 2/5] add link Co-authored-by: Jo Palac --- servicecontrol/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/servicecontrol/troubleshooting.md b/servicecontrol/troubleshooting.md index f8e1afd3e11..20f226be282 100644 --- a/servicecontrol/troubleshooting.md +++ b/servicecontrol/troubleshooting.md @@ -402,7 +402,7 @@ To mitigate this situation, migrating indexes from the Corax to the Lucene index To migrate indexes from the Corax to the Lucene indexing engine, perform the following steps: -1. Start the ServiceControl Audit instance in maintenance mode +1. Start the ServiceControl Audit instance in [maintenance mode](/servicecontrol/ravendb/accessing-database#windows-deployment-maintenance-mode) 2. Access the RavenDB studio 3. Edit the index that needs to be changed 4. From the edit index Configuration tab From 3ee6dc2466f6e38a2af346c126851ddc28dd0568 Mon Sep 17 00:00:00 2001 From: Mauro Servienti Date: Thu, 27 Feb 2025 09:39:29 +0100 Subject: [PATCH 3/5] Fix link --- servicecontrol/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/servicecontrol/troubleshooting.md b/servicecontrol/troubleshooting.md index 20f226be282..58a073ea5ab 100644 --- a/servicecontrol/troubleshooting.md +++ b/servicecontrol/troubleshooting.md @@ -402,7 +402,7 @@ To mitigate this situation, migrating indexes from the Corax to the Lucene index To migrate indexes from the Corax to the Lucene indexing engine, perform the following steps: -1. Start the ServiceControl Audit instance in [maintenance mode](/servicecontrol/ravendb/accessing-database#windows-deployment-maintenance-mode) +1. Start the ServiceControl Audit instance in [maintenance mode](/servicecontrol/ravendb/accessing-database.md#windows-deployment-maintenance-mode) 2. Access the RavenDB studio 3. Edit the index that needs to be changed 4. From the edit index Configuration tab From 52d2c25158dc904359753431844fb9107e4a5ec4 Mon Sep 17 00:00:00 2001 From: Mauro Servienti Date: Thu, 27 Feb 2025 09:56:58 +0100 Subject: [PATCH 4/5] Update servicecontrol/troubleshooting.md --- servicecontrol/troubleshooting.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/servicecontrol/troubleshooting.md b/servicecontrol/troubleshooting.md index 58a073ea5ab..d434f33f956 100644 --- a/servicecontrol/troubleshooting.md +++ b/servicecontrol/troubleshooting.md @@ -415,3 +415,5 @@ At this point, there will be two indexes, the original one and the new one with - Delete the Corax index After the swap operation, the new Lucene-based index must be rebuilt. Depending on the index size, the operation might take a long time. + +When ServiceControl is restarted, the Corax-based index may get recreated. One way to mitigate this behavior is to let the ServiceControl instance recreate the index once and then pause the indexing operations for the Corax-base index. To pause the index from the RavenDB studio while ServiceControl is still in maintenance mode, look for the Corax duplicate index and click the pause indexing button. From be7150636bd528d86cfaaf7500b3fbb31a4a1330 Mon Sep 17 00:00:00 2001 From: Mauro Servienti Date: Mon, 3 Mar 2025 12:43:38 +0100 Subject: [PATCH 5/5] Update servicecontrol/troubleshooting.md --- servicecontrol/troubleshooting.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/servicecontrol/troubleshooting.md b/servicecontrol/troubleshooting.md index d434f33f956..13673eea725 100644 --- a/servicecontrol/troubleshooting.md +++ b/servicecontrol/troubleshooting.md @@ -416,4 +416,6 @@ At this point, there will be two indexes, the original one and the new one with After the swap operation, the new Lucene-based index must be rebuilt. Depending on the index size, the operation might take a long time. -When ServiceControl is restarted, the Corax-based index may get recreated. One way to mitigate this behavior is to let the ServiceControl instance recreate the index once and then pause the indexing operations for the Corax-base index. To pause the index from the RavenDB studio while ServiceControl is still in maintenance mode, look for the Corax duplicate index and click the pause indexing button. +When ServiceControl is restarted, the Corax-based index may get recreated. To prevent the ServiceControl instance from recreating the index, the index can be locked. + +To lock an index, from the RavenDB studio, while ServiceControl is still in maintenance mode, look for the index that was set to use Lucene and click the `🔓 Unlocked` button. Change the setting to `🔒 Locked` ([Locked Ignore](https://ravendb.net/docs/article-page/7.0/csharp/client-api/operations/maintenance/indexes/set-index-lock#lock-modes)). The RavenDB studio will notify the operation completion with the message: _Lock mode was set to: Locked (ignore)_.