Support cancellation of web api and audit storage operations

[ramonsmits]

When ingestion hangs and the service looks unresponse a user might want to stop the instance. Especially for the batch ingestion is makes sense to support cancellation as this is a long running operations.

This is a port and continuation of my investigation PR:

• Investigation hanging audit instance #4758

[ramonsmits]

@danielmarbach @andreasohlund There are multiple commits, one for audit, one for error, and one for domain events. Issue it that seems like a very deep rabbit hole as many handlers invoke storage and all of these do not support cancellation.

As cancellation itself isn't really a bug but an improvement.

As audit is the most affected I think it makes sense to support cancellation there and for the remaning parts to create cruft issues as it is a lot of effort to add cancellation to all those operations.

What is your opinion?

[danielmarbach]

for the remaning parts to create cruft issues as it is a lot of effort to add cancellation to all those operations.

FYI During the modernization efforts we deliberately had to stop at some point. It was just too much effort and also not in all cases glaringly obvious what the effects of cancellation might have on the stability of the code paths calling it so we did cut off somewhere.