ServiceControl Audit Ingestion throwing NREs with audit forwarding enabled

[gbiellem]

Symptoms

Entries in the audit instance log file that indicate forwarding messages failed:

Warn|ServiceControl.Audit.Auditing.AuditIngestor|Forwarding messages failed|System.NullReferenceException: Object reference not set to an instance of an object.

Who's affected

Anyone sending SagaAudit plugin data to an auditing instance that has audit forwarding enabled.

Root cause

The AuditIngestor component was not properly accounting for saga audit data when determining how many messages needed to be forwarded, resulting in an array being sized incorrectly.

Backported to

• 6.5.1
• 5.11.6

Original bug report

Describe the bug

With audit forwarding off ServiceControl is working normally,
But with audit forwarding turned on we see a high proportion of our audit messages being sent to the FailedAuditImports and not being forwarded, these even include message from the ServiceControl SagaState plugin

Expected behavior

With audit forwarding on, expect messages to be ingested and forwarded

Actual behavior

With audit forwarding on a high proportion of messages are being captured in the FailedAuditImports but some are forwarded.

Versions

ServiceControl Audit 6.5
SQL Transport

Steps to reproduce

The log I've uploaded is from an audit instance with the audit forward off initially and run for 5 minutes and then with audit forwarding on. For both configurations we repeated a series of action in one of our apps that generated the same messages, As you can we with audit forwarding on it's throwing object not found exceptions.

logfile.2025-03-20.txt

Failed Imporrt Message From RavenDB.txt

Particular.ServiceControl.Audit-configuration.txt

Relevant log output

Exception:
Object reference not set to an instance of an object.
StackTrace:
   at NServiceBus.Transport.TransportOperations..ctor(TransportOperation[] transportOperations) in /_/src/NServiceBus.Core/Transports/TransportOperations.cs:line 22
   at ServiceControl.Audit.Auditing.AuditIngestor.Forward(IReadOnlyCollection`1 messageContexts, String forwardingAddress, CancellationToken cancellationToken) in /_/src/ServiceControl.Audit/Auditing/AuditIngestor.cs:line 97
   at ServiceControl.Audit.Auditing.AuditIngestor.Ingest(List`1 contexts, CancellationToken cancellationToken) in /_/src/ServiceControl.Audit/Auditing/AuditIngestor.cs:line 54
   at ServiceControl.Audit.Auditing.AuditIngestion.ExecuteAsync(CancellationToken stoppingToken) in /_/src/ServiceControl.Audit/Auditing/AuditIngestion.cs:line 243
   at ServiceControl.Audit.Auditing.AuditIngestion.OnMessage(MessageContext messageContext, CancellationToken cancellationToken) in /_/src/ServiceControl.Audit/Auditing/AuditIngestion.cs:line 213
   at NServiceBus.Transport.Sql.Shared.ProcessStrategy.TryHandleMessage(Message message, TransportTransaction transportTransaction, ContextBag context, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.Sql.Shared/Receiving/ProcessStrategy.cs:line 46
   at NServiceBus.Transport.Sql.Shared.ProcessWithNativeTransaction.TryProcess(Message message, TransportTransaction transportTransaction, ContextBag context, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.Sql.Shared/Receiving/ProcessWithNativeTransaction.cs:line 96
   at NServiceBus.Transport.Sql.Shared.ProcessStrategy.TryHandleMessage(Message message, TransportTransaction transportTransaction, ContextBag context, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.Sql.Shared/Receiving/ProcessStrategy.cs:line 46
   at NServiceBus.Transport.Sql.Shared.ProcessWithNativeTransaction.TryProcess(Message message, TransportTransaction transportTransaction, ContextBag context, CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.Sql.Shared/Receiving/ProcessWithNativeTransaction.cs:line 96
Source:
NServiceBus.Core
TargetSite:
Void .ctor(NServiceBus.Transport.TransportOperation[])

Additional Information

Workarounds

Possible solutions

Additional information