Investigate deadlock, hanging of tests in CI, whether it is CI, F# tasks, TaskSeq CE or XUnit, see also #25

[abelbraaksma]

(see at bottom for original issue description and research)

xUnit bug, see this report

It turns out that if the following is true, xUnit ends up in a deadlock in CI, but not locally:

• You have two or more Task<unit>-returning tests
• These tests live in separate types or modules
• At least two of the tasks yield.

The example in xunit/xunit#2587 is given as two tasks that do a Task.Delay(1), which, as we all know, will yield and pause for the length of at least one timer event (which is 15.6ms on most systems). Even though there are no shared variables, it leads to a deadlock.

A good analysis is in this comment: dotnet/BenchmarkDotNet#2114 (comment).
Possibly related: dotnet/corefx#559 (i.e., that the diff between local and CI is that on CI there are simply not enough threads in the thread pool and thread starvation causes the deadlock).

Solution (for now)

There are only two possible solutions that I can think of, given that the issue isn't likely to be solved any time soon (this is not a critique, it is just a nature of this kind of problems):

• Move to NUnit (neither NUnit and MSUnit have this problem, see Rework ReturnValueValidator and ExecutionValidator dotnet/BenchmarkDotNet#2114 (comment))
• Disable parallelism, though this is unfavorable as we are writing an async library for IAsyncEnumerable<_> and we really want to catch any potential parallelism issues.

---

Original text

Trial and error approach:

• In #23: we already tried removing specific tests, but this didn't appear to help
• In #23: we enabled a global timeout through dotnet test with --blame-hang-timeout 15000ms on the commandline in CI
• In #23: observed that with [<assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)>] the tests run fine
• Observed that (so far) local repro does not exhibit this issue
  Follow up: running in "Crunch mode" (using NCrunch) for several hours using as many parallel threads as possible gives no issues.
• Re-enabled parallel testing (this issue) to be able to see if the behavior persists
  Follow-up: the green test runs were either non parallel, or self-parallelized, still unclear. Edit: see above.
• Run several times in CI to ensure it is stable
  Follow up: it is not. It is an xUnit bug for their CI runner.
• Run the tests on CI using a reflection-based "grab all tests and run in parallel" approach and crunch multiple times.
  Follow up: CI or Windows or dotnet version on CI has nothing to do with this issue, these tests run fine.
• Attempt different dotnet test configurations

[abelbraaksma]

Closing this, we run "in parallel" locally just fine, both in NCrunch, from commandline and in VS 2022, so this only affects CI, where we disabled "in parallel".